"Our field was removed"



  • I returned to work from the Annual Frist Convention (FristCon) to find an email (with the various normal senior people CC'd) in my inbox from a company we outsource to stating that their current round of software issues are caused by a field being removed from our software.

    So to the background: We customise a 3rd-party CRM for InsuraCorp. They have some call-centre requirements, which we have outsourced to a company called SailorDemon who customises their vaguely suitable product to the specific requirements of InsuraCorp. Their product, for these purposes is called MethodCreator. One of the requirements is that they require access to the database of our CRM, so that they can import the data into MethodCreator, and they need to log in using an admin account to avoid using up licenses. If you log into the CRM as an admin, you may add modules and fields, which adds tables and fields to the database.

    At this point, you just about have enough information to guess what happens next. It's ok, I won't make you guess.

    Spoilers: Having received the email and investigated the issue, it turns out they added a field to the production environment without telling me. Surprise, when I deployed some relevant code files, the fields I didn't know existed and as such were not in the code repo, the test system or my dev system were overwritten and disappeared.

    If it's not clear, I do not see myself as accountable for this as I do not expect to have to check the state of the production code every time I want to deploy, and if someone is going to make changes to OUR system they should tell us they're doing so.

    While silly, this circumstance is for me an open/shut case: ask them about the field, restore it, resolved. A minor annoyance. Obviously I asked them to let me know when a new field is being created in the future. However, apart from great latency, there is an oddity in the way the mouth of SailorDemon conveys information, resulting in earlier conversations such as this:

    • Me: The import option does not exist, so I can't import.
    • Mouth of SailorDemon: Yes it does.
    • Me: *Checking a second time to make sure* Here is a screenshot of the import option not existing.
    • Mouth of SailorDemon: Yes it does. Just click the import option.
    • Me: ...
    • Mouth of SailorDemon: It turns out your permissions were not set up for you to see the import option.
    • Me: ... Right.

    In my opinion, failing to even acknowledge a screenshot implied that he is using stalling tactics. Presumably they just take so long to get back to us because they're incompetent and trying (failing) desperately to hide it.

    They did eventually send me the details of the fields which needed restoration, but the next few messages from him (CC'ing other parties) included paraphrases of "our field was removed", forcing me to respond with "tell me when you want to add new fields". I could be using that time to be productive instead of writing defensive messages as diplomatically as possible.



  •  SugarCRM by any chance?



  • @KrakenLover said:

     SugarCRM by any chance?

    Yup. :)

    Interestingly, I got a call from my boss today along the lines of "we're probably starting from scratch without SugarCRM".



  • @Shoreline said:

    it turns out they added a field to the production environment without telling me

    Wow -- you let ANOTHER COMPANY muck around in your PRODUCTION database??? You're lucky all they did was add a new column. This is so wrong in so many ways. I'd let them muck about in an sandboxed database as much as they want, and then ask you to push the changes when they're happy with them. That way you control the complete process including testing and promoting the changes through qa, uat, and finally prod.

    [Would you let me at your database too? I have a few cool things I'm hoping to try out, and I don't want to screw up my database.... ]



  • @DrPepper said:

    @Shoreline said:
    it turns out they added a field to the production environment without telling me

    Wow -- you let ANOTHER COMPANY muck around in your PRODUCTION database??? You're lucky all they did was add a new column. This is so wrong in so many ways. I'd let them muck about in an sandboxed database as much as they want, and then ask you to push the changes when they're happy with them. That way you control the complete process including testing and promoting the changes through qa, uat, and finally prod.

    [Would you let me at your database too? I have a few cool things I'm hoping to try out, and I don't want to screw up my database.... ]

    Something to understand about SugarCRM is that ANY "admin" user can alter the database right through Sugar's admin interface. You can do all sorts of fun things.

    The whole idea being that any salesperson without special training can go into Sugar, add the fields they need, and with the click of a couple buttons the database is updated and the schema modified.

    SugarCRM is not very developer friendly when it comes to this sort of thing.  You can't have different types of admin users, with only some being able to modify the schema, and others having permissions to change layouts.  It's all or nothing.  This makes it really easy for people (especially third-party companies) to do some very stupid things.  And because of SugarCRM's licensing terms, you can't just easily create a "sandbox" instance.  You would have to pay for all the additional user fees if you're using On Demand.

    This is the root reason the OP's problem happened in the first place.  If Sugar had proper ACLs for administrative accounts, then this never would have happened.  As it stands, if you have an admin account in Sugar, you have the keys to the kingdom.

    @Shoreline said:

    @KrakenLover said:

     SugarCRM by any chance?

    Yup. :)

    Interestingly, I got a call from my boss today along the lines of "we're probably starting from scratch without SugarCRM".

    Yeah, I knew it was Sugar right away.

    SugarCRM has a lot of flaws.  However, it can be very effective when used and modified correctly.  SugarCRM lives in its own little world, and if a developer doesn't understand the culture and insanity that is unique to Sugar, then the results are going to be really poor.  But, if you do things just right, then Sugar is as good as any other CRM product out there (and better in some key areas, especially compared to SalesForce in terms of cost and flexibility).

    I think TRWTF is this case was lack of understanding of Sugar on the part of the business, willingness to give a third-party unsupervised control over Sugar, and bad business processes with a poor relationship between the involved parties.

    I've been in the very same boat many times, being as I'm a professional SugarCRM developer.  No matter what Sugar (the company) may say in their marketing materials, not just anyone can develop for Sugar straight off the bat.  There are layers upon layers of WTF in Sugar, and 90% of the Sugar developers and consultants out there are clueless about how to develop for Sugar properly.  You can't just treat it like some generic piece of software and march in blind, or you will get burned.  Badly.

    A lot of my work is cleaning up the messes other developers made with SugarCRM, which were often made at the behest of consulting partner who had fuck-all knowledge of Sugar and how it should be used and managed.

    A CRM system is critical to the workflow of a business, and when you hire consultants who don't understand your chosen CRM, you have no one to blame but yourself when the implementation turns into a disaster.

    Okay, so this turned into a rant...  Not my intention, but Sugar has a way of making you want to scream and rend your garments. There is a reason I keep a little statue of Cthulhu on my desk.

    That said, if you ever want to do SugarCRM development/customization, you might want to talk to someone who actually does SugarCRM development.



  • I once filed a bug to Sugar which amounted to surrounding a block of code (which belonged in an extracted function, really) with an if statement.

    In another instance, I filed a bug saying "this call right here: these two arguments are in the wrong order. They're not even the same type. If it wasn't PHP it wouldn't compile. And if surrounding code didn't skip the call 99.99999% of the times, it would break badly. Proposed code fix: put the arguments in the same order as they are declared."

    Both times their answer was "please fill in, sign and fax us this waiver disclaiming all ownership of your code".
    I admit I'm guilty of not having bothered to push the issue any further in both accounts.

    The first bug was in class SugarBean, the second in ListView. It's not like they're used anywhere.



  • @Zecc said:

    Both times their answer was "please fill in, sign and fax us this waiver disclaiming all ownership of your code".
     

    "No. Please fix YOUR code before I provide full disclosure to many sites that lambast poor programming principles. You have 30 days to confirm this fix has been implemented and tested bug-free for this particular defect."


  • Considered Harmful

    @Zecc said:

    I once filed a bug to Sugar which amounted to surrounding a block of code (which belonged in an extracted function, really) with an if statement.

    In another instance, I filed a bug saying "this call right here: these two arguments are in the wrong order. They're not even the same type. If it wasn't PHP it wouldn't compile. And if surrounding code didn't skip the call 99.99999% of the times, it would break badly. Proposed code fix: put the arguments in the same order as they are declared."

    Both times their answer was "please fill in, sign and fax us this waiver disclaiming all ownership of your code".
    I admit I'm guilty of not having bothered to push the issue any further in both accounts.

    The first bug was in class SugarBean, the second in ListView. It's not like they're used anywhere.

    I think they just want to make sure they don't implement your fix and immediately hit them with some lawsuit saying they used your patch without licensing it from you. I imagine something to the effect of "I release the following code, of which I am the original author, into the public domain" would have sufficed.



  • @Zecc said:

    I once filed a bug to Sugar which amounted to surrounding a block of code (which belonged in an extracted function, really) with an if statement.

    In another instance, I filed a bug saying "this call right here: these two arguments are in the wrong order. They're not even the same type. If it wasn't PHP it wouldn't compile. And if surrounding code didn't skip the call 99.99999% of the times, it would break badly. Proposed code fix: put the arguments in the same order as they are declared."

    Both times their answer was "please fill in, sign and fax us this waiver disclaiming all ownership of your code".
    I admit I'm guilty of not having bothered to push the issue any further in both accounts.

    The first bug was in class SugarBean, the second in ListView. It's not like they're used anywhere.
    I have the benefit of being able to talk to Sugar's developers directly, which helps in getting certain issues addressed.  But, yeah, otherwise you're pretty much screwed if you have to go to sugar support or file a bug.  Unless you are a large enough client.

    There are hundreds of bugs in Sugar which have been around for years.

    The Quotes module itself is a giant, steaming, pile of WTF.  The JavaScript, the DB schema.  It was quite clearly designed by PHP devs who had no experience with JavaScript or good DB design.

    And the AjaxUI.

    And the Product Catalog/Templates.

    And... well, all of Sugar really.

    But it's still pretty good for its target market, and if you understand Sugar's insanity, you can smooth down the rough edges decently enough.



  • @KrakenLover said:

    If you ever want to do SugarCRM development/customization, you might want to talk to someone who actually does SugarCRM development.

    I think you're right. At my current job it's merely difficult or impossible to do the exciting things we want to do. At my previous job, somebody before our time had performed experimental surgery on the instance of SugarCRM's guts. The result was that it wouldn't even do fundamental things it was supposed to do (e.g. reliable custom modules created through the studio). The IT team obviously looked bad when it failed, which it did all the time, so we were at the mercy of SugarCRM or it's dodgy, modified, non-upgrade-safe guts.

    In one customisation, I found fields were being auto-calculated in javascript with timeout events, rather than keyboard events. That alone put me four hours behind schedule trying to find why my code wouldn't work sometimes, and why I could only reproduce the issue when I ran alert boxes.

    I always thought the Lovecraft table-top RPG had a lame sanity system where if you failed at "You see some intestines, roll your sanity." you lost sanity, making it harder to pass the next one. Having used SugarCRM, I began to believe.



  • @Shoreline said:

    At my previous job, somebody before our time had performed experimental surgery on the instance of SugarCRM's guts. The result was that it wouldn't even do fundamental things it was supposed to do (e.g. reliable custom modules created through the studio). The IT team obviously looked bad when it failed, which it did all the time, so we were at the mercy of SugarCRM or it's dodgy, modified, non-upgrade-safe guts.
    Yeah, doing non-upgrade safe things in SugarCRM is a really, really, bad idea.  I understand why people do it - it's seemingly so much easier than doing things the upgrade safe way with custom installable packages.  But it will bite you in the ass (if you're lucky, that's all it will do down there).

    Why is it so much easier to just make modifications to the core of Sugar rather than doing things the upgrade safe way?  Because there is little to no documentation about creating custom installable packages, and the syntax and expected values of variables are sometimes quite nonsensical.  If you want to learn how to do it, you need to read a billion blog posts, trawl through the Sugar source code, and experiment - because there is nothing out there which tells you how to do anything truly useful or significant in simple 1, 2, 3 steps with logical explanation behind it.

    @Shoreline said:

    In one customisation, I found fields were being auto-calculated in javascript with timeout events, rather than keyboard events. That alone put me four hours behind schedule trying to find why my code wouldn't work sometimes, and why I could only reproduce the issue when I ran alert boxes.
    SugarCRM really molests JavaScript too.  Segments of Sugar's JS were written by PHP developers who had no experience with JavaScript development, and a poor understanding of the DOM (Quotes module, I'm looking at you).

    This means that in some places, it is really hard to do what you really need to do using standard JS development practices/coding techniques.  If you want to develop custom JS for SugarCRM, you have to have knowledge of SugarCRM's JS objects, which have custom utility functions that allow you to do certain things that you would have no alternative for using regular JS.  Programming timeouts in custom JS for Sugar is commonly used when people don't know about the SUGAR.util.doWhen method.

    Developing JS for Sugar is a balancing act between knowing what you can code around and ignore, and what you have to use Sugar's JS objects for. And there isn't documentation for this either.

    @Shoreline said:

    I always thought the Lovecraft table-top RPG had a lame sanity system where if you failed at "You see some intestines, roll your sanity." you lost sanity, making it harder to pass the next one. Having used SugarCRM, I began to believe.
    That old, cliche, saying "You don't have to be crazy to work here, but it sure helps" is pretty much literally true when it comes to working with Sugar professionally.

    We got a new developer not long ago, and he told me he was having a lot of trouble getting SugarCRM to make sense at first.  He said he's doing a lot better now that he realizes SugarCRM doesn't make sense.

     



  • @KrakenLover said:

    Why is it so much easier to just make modifications to the core of Sugar rather than doing things the upgrade safe way?
    I've had to hack the core when certain assumptions hardcoded in central logic were against what I was trying to achieve, and when I needed to modify the cardinality of relationships between core entities which did not make sense as they were on certain business cases. (I'm a little fuzzy on the details since it's been years, sorry) Some logic was simply too ingrained.

    One of the most painful things I've had to do was allow accounts, contacts and other stuff to come from different database instances and from other datasources. Every bean had a $db property and there was a $db global variable too, and they were used interchangeably according to the latest mood of whoever wrote the code, and SQL was everywhere.



  • @Zecc said:

    I've had to hack the core when certain assumptions hardcoded in central logic were against what I was trying to achieve, and when I needed to modify the cardinality of relationships between core entities which did not make sense as they were on certain business cases. (I'm a little fuzzy on the details since it's been years, sorry) Some logic was simply too ingrained.

    One of the most painful things I've had to do was allow accounts, contacts and other stuff to come from different database instances and from other datasources. Every bean had a $db property and there was a $db global variable too, and they were used interchangeably according to the latest mood of whoever wrote the code, and SQL was everywhere.

    There are a lot of aspect of Sugar that make assumptions about the business logic.  But as a hard and fast rule with Sugar, you should explore every possible alternative before doing core modifications.  You can use Sugar's custom directory to overwrite default behaviors and relationships, and also use JavaScript injected into various pages to alter layouts and functionality without making direct modifications to the core layout files.  Logic-hooks can also be used to implement custom business logic in a way that is upgrade-safe. There is almost always some upgrade-safe way to implement a customization in SugarCRM - but it requires that you really know Sugar very well, working with it heavily.

    If the client won't change their (often arbitrary) requirements, I will turn down work if the only way to do something is to modify the Sugar core.  Of course, not everyone has the luxury of being able to refuse work like that.  But the end result of modifying the core is, in my personal experience, never worth the cost that comes later on when it's time to upgrade or make further customizations, and the client is better off in the end for not having to pay another consultant to redo potentially hundreds of hours of work.



  • Logic hooks was one of the nice things that came with version 5.0 that I was referring to. One of the core hacks I've done was to make it possible for a hook to signal the core "I got this, don't do your thing because I've already handled it"; event.preventDefault(), in other words.



  • @Zecc said:

    Logic hooks was one of the nice things that came with version 5.0 that I was referring to. One of the core hacks I've done was to make it possible for a hook to signal the core "I got this, don't do your thing because I've already handled it"; event.preventDefault(), in other words.
    Ah, pre-version-5.  I have heard stories about that.  Thankfully I started Sugar development long after 5 was already released.

    A lot of my development work is for SugarCRM On Demand, so I can't make core modifications even if I wanted to.   Especially now that Sugar requires modules to be certified as being acceptable in the On Demand environment.

    Sugar is moving more and more toward a closed-source, Salesforce-esque, system.


Log in to reply