Sherlock Holmes and the Case of the Second Facepalm

kraagenskul

Names changed to protect the innocent (and my job.)

There is nothing worse than code that works on one machine and not another. It's even worse when it works in multiple machines but not all of them. In this case it worked everywhere but one place. And of course that was the place where we needed it.

It was one web page, but one the customer desperately needed to work. It worked on the development machines, our qa machine, the customer's qa machines, the staging machine, but not the production machine. Since I was in charge of building and deploying to most of these, the problem fell upon me to fix.

Since the ear was directly copied from the staging to production and the former worked, I doubted anything within the ear was the issue but I used an MD5 hash and check to make sure. Exact match. So it had to be in the configuration. I checked all the differences, even using compare software. Everything was the same where it should be, and different where it should be. I swapped some of the configurations around on the two boxes (such as the database), but staging always worked and production never did. At one point they were completely swapped, and I sat staring at them. The words of Sherlock Holmes came to me, although not exactly as written by Doyle:

"How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth"

It had to be the URL I was using on production as it this point it was the only difference on the two instances. I used the numeric IP address, and sure enough, it worked fine.Why the hell would that happen?

Most of the machines URLs were based off of our company name and the customer's name: staging_customername.ourcompanyname.com\captain

The production machine used the customers web site: captain.customername.com\captain

The word "captain" showing up twice was my first clue. I looked at the code for the page:

newPath =oldPath.substring(0, oldPath.indexOf("captain")) + "content";

After a nice face palm, I sent an email to the offending developer explaing the problem and asking them to use one of the functions we already had in place to grab the base URL. I realized I could have fixed it myself, but I wanted to make him aware of the built in functionality for such instances since he was relatively new.

This is the code that was checked in:

newPath =oldPath.substring(0, oldPath.indexOf("/captain")) + "/content";

And there's the second facepalm.

dhromed

Bitchslap him.

bjolling

@kraagenskul said:

It worked on the development machines, our qa machine, the customer's qa machines, the staging machine, but not the production machine.

My first reaction would have been to just put the staging machine in production. It's the way it's done in the company I work for... @kraagenskul said:

The production machine used the customers web site: captain.customername.com\captain

... but of course here our staging environment would have used the URL: captain.customername.test\captain

DOA

I remember the time I let a "colleague" set up a database connection in a PHP project. We had a very simple class for database connectivity complete with phpdoc generated documentation, so I figured he'd use that as it'd take him 3 lines to get data out of the database. Later I checked the source code and sure enough he had copy-pasted some example database connection code straight out of PHP's documentation and into production code, complete with hardcoded login and password.

Sometimes people make me homicidal.

Xanthus179

@DOA said:

Sometimes people make me homicidal.

I know exactly what you mean. All the other times I am most likely asleep.

chrismcb

@kraagenskul said:

Most of the machines URLs were based off of our company name and the customer's name: staging_customername.ourcompanyname.com\captain

I had a similar problem recently. Everything worked on my test machine. Everything worked on the testers machine, assuming you used the ip address. But use the name, and things failed. After two days of banging my head against the way... It worked fine if I used Firefox, but not if I used IE, yet IE worked fine on my machine.

Turns out "_" is an invalid character in a DNS name. So I would recommend NOT using "staging_customername"

@chrismcb said:

Turns out "_" is an invalid character in a DNS name.

Not quite; you can put arbitrary octets in DNS labels, underscore is prohibited by the convention for host names.

kraagenskul

@chrismcb said:

Turns out "_" is an invalid character in a DNS name. So I would recommend NOT using "staging_customername"

I changed the names of all the server just in case my manager actually reads the site. The name actually has a hyphen, not an underscore (staging-customername), but I've been doing a lotof Ruby programming lately and have underscores on my brain when it comes to naming.