Names changed to protect the innocent (and my job.)
There is nothing worse than code that works on one machine and not another. It's even worse when it works in multiple machines but not all of them. In this case it worked everywhere but one place. And of course that was the place where we needed it.
It was one web page, but one the customer desperately needed to work. It worked on the development machines, our qa machine, the customer's qa machines, the staging machine, but not the production machine. Since I was in charge of building and deploying to most of these, the problem fell upon me to fix.
Since the ear was directly copied from the staging to production and the former worked, I doubted anything within the ear was the issue but I used an MD5 hash and check to make sure. Exact match. So it had to be in the configuration. I checked all the differences, even using compare software. Everything was the same where it should be, and different where it should be. I swapped some of the configurations around on the two boxes (such as the database), but staging always worked and production never did. At one point they were completely swapped, and I sat staring at them. The words of Sherlock Holmes came to me, although not exactly as written by Doyle:
"How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth"
It had to be the URL I was using on production as it this point it was the only difference on the two instances. I used the numeric IP address, and sure enough, it worked fine.Why the hell would that happen?
Most of the machines URLs were based off of our company name and the customer's name: staging_customername.ourcompanyname.com\captain
The production machine used the customers web site: captain.customername.com\captain
The word "captain" showing up twice was my first clue. I looked at the code for the page:
<font color="black"><font size="2"><font face="Arial">newPath</font></font></font><font color="black" size="2" face="Arial"> =oldPath.substring(0, oldPath.indexOf("captain")) +
"content";</font>
After a nice face palm, I sent an email to the offending developer explaing the problem and asking them to use one of the functions we already had in place to grab the base URL. I realized I could have fixed it myself, but I wanted to make him aware of the built in functionality for such instances since he was relatively new.
This is the code that was checked in:
<font color="black"><font size="2"><font face="Arial">newPath</font></font></font><font color="black" size="2" face="Arial"> =oldPath.substring(0, oldPath.indexOf("/captain")) +
"/content";</font>
And there's the second facepalm.