Email Blasted


  • Trolleybus Mechanic

    So one of our systems just sent out a mass email to thousands of customers informing them of an impending fire drill.

    Except that there is no fire drill. It was a new email page being created on a dev system. With live data.

    Sure, we shouldn't be using live email addresses on a test system, and that is a wtf, but it's nothing extraordinary. And certainly nothing that isn't solvable. But it my company, the real wtf come from when you take a problem, and apply the "company" solution. It's a combinatory solution that's sure to blow with more force than any individual wtf.

    My solution: Why don't we change Exchange so that emails coming from the dev server are only delivered to internal mailboxes, and maybe a few whitelisted outside addresses? I'm sure there's a way to do that by IP address, and worse case, just throw up a second Exchange server dedicated only to testing. Maybe we can get fancy and prepend all subject lines with "[TEST - IGNORE]"?

    The company solution:  Every single web application will now need to have  a dedicated page. This page will query the database, and find all email addresses in the system. The user will then go one by one (or if we can figure it out, in bulk) and change the email addresses to something else. We'll need to keep running this page every time a data import happens, so make it part of the unwritten, convoluted and multistep "processes" that I just made up now.



  • By change the email address to something else, you mean change it to garbage so it bounces, or change it one-by-one because some people want to receive these test emails?
    If the former, just write a 25 line service which replaces all text before the '@' with a guid. If the latter, create a service which reads an configuration file telling it which emails to leave alone (dev team, etc.). Or get a little fancy and apply a regex for certain domains, etc.

    Or is this WTF more insidious than that and you're not even allowed to create something sane like this? If so, then write the service anyway, and tell everybody you'll take full responsibility of running the "magic page".


  • Trolleybus Mechanic

    @C-Octothorpe said:

    By change the email address to something else, you mean change it to garbage so it bounces, or change it one-by-one because some people want to receive these test emails?
     

    Both. So there needs to be a bulk update like you mention (turn everything to guid@domain.com, or  user@example.com). But some of those have to be excepted, since we don't want to clobber out the developer and QA email addresses.

    @C-Octothorpe said:

    If the former, just write a 25 line service which replaces all text before the '@' with a guid. If the latter, create a service which reads an configuration file telling it which emails to leave alone (dev team, etc.). Or get a little fancy and apply a regex for certain domains, etc.

    Assuming that all the emails go through a central library, and that there isn't copy-pasta'd code all over the place that sends emails...

    (Guess what shouldn't be assumed)

    @C-Octothorpe said:

    Or is this WTF more insidious than that and you're not even allowed to create something sane like this? If so, then write the service anyway, and tell everybody you'll take full responsibility of running the "magic page".

    Ding. The magic page needs to go into the database, and actually do UPDATE tbl SET email=....  Except that it won't be just one page. It will be one page per system, because each system uses a different table and a different column. And most systems uses more than one table-- different if you're, say, a prinicpal, a caretaker, or a member of the public.

    (What's that-- group common attributes together and only use divergent tables for divergent data? AHHAHAHAHAHAHAHAHAHAHAHAHAHA)

     



  • NO no no!

    Do not put the data scrub as part of your domain code.

    You import prod data to test, scrub data, now use it as though it is normal data.

    The scrubbing happens as part of your data load.

    Though I personally have no idea how you get a known start state for any tests so you can achieve a known result.

     



  • @KattMan said:

    The scrubbing happens as part of your data load.
    I thought of that after I posted and didn't bother correcting because I figured someone is going to mention that idea as well.



  • @C-Octothorpe said:

    If the former, just write a 25 line service which replaces all text before the '@' with a guid.
     

    I would anonymise the live data by replacing all emails with "@local.testbed" then set up my internal MX to collect all emails addressed to that domain and drop it into one mailbox that others could access.

    No.. scrub that.. I have done it. It prevented any emails from escaping, and ensured that any escaping would never have been delivere to live addresses.

    It concerns me that you use live production data in such a careless manner....


  • Trolleybus Mechanic

    @Cassidy said:

    I would anonymise the live data by replacing all emails with "@local.testbed" then set up my internal MX to collect all emails addressed to that domain and drop it into one mailbox that others could access.

     

    I'll try suggesting that, but it seems no one here knows how to use Exchange. Including the Exchange admin. Last time I wanted a new distro group, I got some blank stares and the suggestion that I do it myself.

    @Cassidy said:

    No.. scrub that.. I have done it. It prevented any emails from escaping, and ensured that any escaping would never have been delivere to live addresses.

    It concerns me that you use live production data in such a careless manner....

     

    It concerns me to. I'm just glad we farm all ecommerce processing off to a third party. 

     

     



  • @Lorne Kates said:

    I'm just glad we farm all ecommerce processing off to a third party Nagesh

    FTFY



  • @Lorne Kates said:

    I'll try suggesting that, but it seems no one here knows how to use Exchange. Including the Exchange admin. Last time I wanted a new distro group, I got some blank stares and the suggestion that I do it myself.
     

    erm.. okay... minor WTF in itself.

    Either way, that's no reason not to do it, it's just a reason why people don't currently know how to do it.

    What's the alternative? "we're getting fined for spamming because nobody knows how to correctly configure and operate our internal mail server..."



  • @Cassidy said:

    What's the alternative? "we're getting fined for spamming because nobody knows how to correctly configure and operate our internal mail server..."

    I don't know what is the jurisdiction for that company but in most countries it takes a lot of spam to get a fine. A more common problem is getting blacklisted and therefore not being able to send legitimate emails.


  • Trolleybus Mechanic

    @Speakerphone Dude said:

    @Cassidy said:

    What's the alternative? "we're getting fined for spamming because nobody knows how to correctly configure and operate our internal mail server..."

    I don't know what is the jurisdiction for that company but in most countries it takes a lot of spam to get a fine. A more common problem is getting blacklisted and therefore not being able to send legitimate emails.

    No, the real concern is pissing off the end users, who complain to the client, who get pissed off at the company, who then drop the company as their software provider.

     

     



  • @Lorne Kates said:

    the real concern is pissing off the end users, who complain to the client, who get pissed off at the company, who then drop the company as their software provider

    I remember when I used to believe that users satisfaction is more important than the senior management golf deals. Ah, the candor of youth.




  • 1) Update the email address to all be the company CEO's email address.

    2) Run the tests and flood his/her mailbox.

    3) Apply the Peter Principle and get promoted.

    4) Profit!

     

    - chooks

     

     


  • Trolleybus Mechanic

    @Speakerphone Dude said:

    @Lorne Kates said:
    the real concern is pissing off the end users, who complain to the client, who get pissed off at the company, who then drop the company as their software provider

    I remember when I used to believe that users satisfaction is more important than the senior management golf deals. Ah, the candor of youth.

     

    Silly you, user satisfaction with out system is ENTIRELY* determined by how rounded the corners are, and how clean the background gradients look.

    *(Unless they're using some versions of Opera, nearly any version of IE, are colorblind, or just plain blind, or are on a mobile device)

      (Or if they care more about just using the sysem than the UI )


  • Trolleybus Mechanic

    @chooks said:


    1) Update the email address to all be the company CEO's email address.

    2) Run the tests and flood his/her mailbox.

    3) Apply the Peter Principle and get promoted.

    4) Profit!

    kids don't try this at home

     

    I would never advise anyone to try this at home. Do it at work where you have a much larger dataset, and a enterprise-grade MTA.

     



  • @Lorne Kates said:

    @chooks said:


    1) Update the email address to all be the company CEO's email address.

    2) Run the tests and flood his/her mailbox.

    3) Apply the Peter Principle and get promoted.

    4) Profit!

    kids don't try this at home

     

    I would never advise anyone to try this at home. Do it at work where you have a much larger dataset, and a enterprise-grade MTA.

     

    ... and a few internet connections so you can round-robin them an get as much emails sent as possible before all the IPs are blacklisted.



  • @Lorne Kates said:

    No, the real concern is pissing off the end users, who complain to the client, who get pissed off at the company, who then drop the company as their software provider.

    That - I should have made it a bit clearer that the business impact is more than a few fines, it's loss of reputation and customer perception. We've sacked one marketing girl for (twice) CCing a fraction of our clientbase with spam targetted mails of business interest. Several asked to be removed from our mailing lists, several bomblasted their account managers (who were unaware at the time) and some "replied-all" giving us a right kicking (in front of other customers) for incompetance.

    .. and they were right.


Log in to reply