"No, no, this is the test server, trust me!"
-
A minor WTF that happened to me last week. :)
I work together with a nice guy and from what I can see he produces nice code (I'm not that great myself so who am I to judge?) but sometimes he's a bit... reckless.
The other day I noticed a slight bug in one of the websites we maintain, so I went over to him and asked him to have a look at it. I presumed it was something with the database code, which he maintains. He told me he had already fixed the bug and I should look on the test system, but I already did and the bug was present there as well, his reply: "oh yes, I only tested it on my own PC"). A basic procedure we have for deploying a bugfix is to upload the new version to the test server and reset the data in the database with dummy data using a script.
So I watch while he starts the procedure and see him remotely log into the production server, so I tell him "hey dude I think you connected to the production server, you better start over before things get screwed up". But to my surprise he replies "no, no, this is the test server, truest me!". I propose to him to disconnect and then connect to the test server just to be sure, but again he tells me to "trust him". He then ran the test script and it echoed that the database was reset successfully. He goes to the test server's web address and notices that all the old test data is still there. Still standing behind him I see him thinkin "oh shit....", while he goes to the production server's web address. And yep, all the default data was in there.
Luckily we make hourly backups and the site is modified only about once a week, so in the end it was no biggy...
Though I hope he learned his lesson and is more careful in the future. xDmod: Fixed yo' paragraphing – dh
-
Be sure that from now on whenever he asserts something to you or to the group in a meeting, you help him out by saying "Trust me!"
-
Trust me.
I know what I'm doing.
-
@pbean said:
Luckily we make hourly backups and the site is modified only about once a week, so in the end it was no biggy...
This caught me off guard. I was expecting "...we realized that the backup hasn't worked for two years year and when it did work it encrypted the data with a key noone can find".I think I spend too much time here.
-
@DOA said:
@pbean said:
Luckily we make hourly backups and the site is modified only about once a week, so in the end it was no biggy...
This caught me off guard. I was expecting "...we realized that the backup hasn't worked for two years year and when it did work it encrypted the data with a key noone can find".I think I spend too much time here.
At the moment I actually expected such a situation to arise, trust me. But luckily the person in charge of the backups (imho one of the most important tasks, if not the most important) isn't some random failguy but actually knows what he's doing and even checks the backups by hand fairly often.
-
While I was arround a problem happened once on the production machine vs dev. After thatI made sure they are well labeled, made the application itself aware of whats going on and let you know about it in big bold friendly letters telling you not to panic.
-
hourly backups of the database?
I prefer flashback queries (if your DBMS supports them of course):
DELETE FROM some_table;
INSERT INTO some_table
(SELECT * FROM some_table AS OF timestamp sysdate - 1/24);Of course you can also only look at the data instead of replacing the table content with it.
-
Flashback doesn't work with schema changes. If their reset query is like some I have used.. i.e. drop all tables and recreate, flashback data is totally gone. Real backups are much more dependable.
And yes I did drop a column on a 10 million row table in production, and was sad I couldn't flash it back. Took 8 hours to get the row back via tape and hand-edit the few changes since the backup from server logs. Sad day.
-
@thetunaman said:
Took 8 hours to get the row back via tape
This, btw, is why I never make changes to production late in the day. Especially on Friday.
-
@DOA said:
@thetunaman said:
Took 8 hours to get the row back via tape
This, btw, is why I never make changes to production late in the day. Especially do shit on Friday.
FTFM
-
@thetunaman said:
And yes I did drop a column on a 10 million row table in production, and was sad I couldn't flash it back. Took 8 hours to get the row back via tape and hand-edit the few changes since the backup from server logs. Sad day.
So is the TRWTF that you dropped a column on a production database that you didn't mean to drop? Or is it that you are still allowed to touch production databases after doing something like that? Or is TRWTF that you still use tape backups?
-
It wasn't literal tape, was backed up to another hard drive array as a full region dump. The DBA called them "tapes"IIRC.
-
@thetunaman said:
It wasn't literal tape, was backed up to another hard drive array as a full region dump. The DBA called them "tapes"IIRC.
Alright, I'll let you off with a warning.. this time.
-
TRWTF is that he had write access to the production server.
-
@alegr said:
TRWTF is that he had write access to the production server.
?
I have access to production. I need to put things there. Duh.
-
@dhromed said:
I have access to production. I need to put things there. Duh.
I think his logic was anybody who doesn't make absolutely sure what server he's connected to shouldn't have access to production.
My employer's server admins are very meticulous in their procedures, and if they have any shadow of a doubt they are connected to the wrong server, they'll make a few double checks just to make sure. I know if I had production access and anyone, even the most dense individual looking behind me, said, "I think you're connected to production" when I intend to do something on a development environment, my response would never, ever be, "You're wrong. Trust me."
-
@RHuckster said:
I think his logic was..
Oh wait, he specifically, not just "developer".
I assume, anyway.
I have a personal account on the development server, and I have made Windows pink. There's definitely no mistaking then. Dito for the brown and green staging servers, as opposed to the live servers with generic accounts and default colouring.
-
Our production server's background is Red, which will be the last color you'll see if you do something wrong on it.
-
thats what you get for letting damn pirates at the controls - we have had nothing but problems since jack sparrow did a little php for us.
-
@XIU said:
Our production server's background is Red, which will be the last color you'll see if you do something wrong on it.
I colorize the bash prompt. Red for prod, yellow for staging, cyan for dev and green for my local machine.
-
@morbiuswilters said:
@XIU said:
No pink, huh. I thought I knew you better than I do, then.Our production server's background is Red, which will be the last color you'll see if you do something wrong on it.
I colorize the bash prompt. Red for prod, yellow for staging, cyan for dev and green for my local machine.
-
@belgariontheking said:
@morbiuswilters said:
@XIU said:
No pink, huh. I thought I knew you better than I do, then.Our production server's background is Red, which will be the last color you'll see if you do something wrong on it.
I colorize the bash prompt. Red for prod, yellow for staging, cyan for dev and green for my local machine.
The entire background is pink. If the prompt was pink, I wouldn't be able to see it.
-
@morbiuswilters said:
The entire background is pink. If the prompt was pink, I wouldn't be able to see it.
What do you need to see? Do you have such little confidence in the things that you type that you need to actually SEE them? pshhhh.
