The test environment broke, so just deploy it and see what happens
-
After several months of coding, we are several days into testing and both test servers crash - hard. After replacing most of the innards, they still crash - hard - after less than 60 minutes.
Finally, my boss gave up trying to run stress tests, or pretty much any other tests on the builds of all the last minute must-have's.
Erm, are we delaying the release?
No, just deploy it and see what happens.
-
@snoofle said:
No, just deploy it and see what happens.
Is this your warning to us to make sure that we have a decent amount of provisions stocked away, and the car gassed up and ready to go?
-
Most times it works.
How are the risks distributed? Is it a widely used project, that lots of people depend upon, or is it a new project, that people will start to use now? What will you lose if you wait to release?
Ok, your manager is probably overestimating this last one.
-
@snoofle said:
After several months of coding, we are several days into testing and both test servers crash - hard. After replacing most of the innards, they still crash - hard - after less than 60 minutes.
Finally, my boss gave up trying to run stress tests, or pretty much any other tests on the builds of all the last minute must-have's.
Erm, are we delaying the release?
No, just deploy it and see what happens.
The application is physically damaging servers, this means it is ready for Production. Mission accomplished!
-
@Mcoder said:
Most times it works.
Risk is a funny thing. Several years ago I sat and watched my boss re-booting an embedded computer during 5 minute gaps between production runs in a steel mill in order to update some software. He did this multiple times with no issue. So when I needed to do the same thing I wasn't to worried. Of course when I did it it crashed the computer and left some very expensive (and critical) equipment directly in the path of 10 tonnes of white hot steel rapidly coming down the path and no time to get things running normally. I survived that one by manually forcing I/O on the control computer to get the equipment out of the way and gain enough time to get things under control.
More recently I had planned to system test some changes to some code before going to site to deploy it. However screw ups by an intern meant I lost my chance to do testing (and also half a nights sleep). When I went to site again things crashed an burned. I spent about 6 hours on the phone trying to get support from the people back in the office in order to figure out what had gone wrong. No-one could figure it out, so I pulled everything out before I left. When I got back to the office the next day and actually tested things I spotted a trivial mistake that lead to everything failing.
So I am a great believer in testing. Or to put it another way, there is an old saying I heard a while back that goes like (and I am sure there are other versions):Trust in God, but tie your camel
-
-
@OzPeter said:
So I am a great believer in testing. Or to put it another way, there is an old saying I heard a while back that goes like (and I am sure there are other versions):
Trust in God, but tie your camel
I have experienced many sayings about testing, amongst which are:
- if you don't try it out on the testbed then you're unconcerned if things fail in production
- testing does not prove success - it exposes failure
- testing shows the presence of bugs - it does not actually fix them
- developers submit code, believing it works; testers accept it - believing it doesn't
- there is no such thing as defect-free code; there is simply code containing undiscovered defects
-
Oh yes. A heavy machine controller that is working now, but we wants to issue an update... Test that damn thing all you can, that won't be enough. Get the hell out of range and stop every machine you can while deploying.
But, another possibility is an update that fixes a bug in a money generating website, that nobody can use now (AKA it is losing its long time customers) becauseof this same bug... If testing will need more than a few minutes, forget about it!
As I said, what are the risks involved?
-
Could you imagine if this was Boeing?
Engineer: "Sir, we tested our unmanned passenger jet, and the prototype took off from the runway, but we can't find it on radar, and there's this big plume of smoke on the horizon."
Manager: "Hmm... I wonder where the plane went. If we don't have the plane, we can't test it!"
Engineer: "Well, sir, I'm pretty sure that plu--"
Manager: "Let's release it."
Engineer: "But sir--"
Manager: "RELEASE IT. Let's just see what happens."
-
@dhromed said:
So... just tie the camel.
What goes on between you and a consenting camel is none of my business and, quite frankly, TMI.
-
@rstinejr said:
I think a camel would look pretty spiffy in one of these:@dhromed said:
So... just tie the camel.
What goes on between you and a consenting camel is none of my business and, quite frankly, TMI.
-
@rstinejr said:
@dhromed said:
So... just tie the camel.
What goes on between you and a consenting camel is none of my business and, quite frankly, TMI.
Based on the words in red it appears that you are not on the same page.
For the html people among us: I've put a style property on the B tags and I am proud of it.
-
@OzPeter said:
Of course when I did it it crashed the computer and left some very expensive (and critical) equipment directly in the path of 10 tonnes of white hot steel rapidly coming down the path and no time to get things running normally. I survived that one by manually forcing I/O on the control computer to get the equipment out of the way and gain enough time to get things under control.
I just got a visual of this in my head.OzPeter reboots the machine.
*View of moving machinery coming to a halt*
He looks up and sees the jam, turns his head and sees white-hot metal start to come down a long conveyor line. He then turns around and presses a big red mushroom button. Nothing happens. He slams on it again and again.
*View of him from the ceiling from just above a duct carrying cables. One of them is frayed. Sparks shoot out of it.*
He looks at the line again. The metal is moving. Jumps into a seat in front of a terminal and starts typing like a madman. Mission Impossible music starts playing.
*Close up of his face. Sweat is dripping down*
*View changes to over his shoulder*
As OzPeter types away a 3D wireframes of the jammed equipment spins and swivels on the screen. Parts of it are flashing red.
*View of hot steel approaching jammed machine. The machine is twitching trying to move out of the way. View changes to our hapless operator*
Suddenly he stops typing, slams Enter, jumps up and hits the mushroom button again.
*View of the jammed equipment as seen by the approaching load. The machine screeches as it moves out of the way. View changes to OzPeter *
Mission Impossible music hits climax. He stares at the production line covered in sweat and trembling with the after effects of adrenaline.
He has survived another day at the job.
-
There are three keys to successful software development:
- Testing,
- Testing, and
- Testing.
-
@AndyCanfield said:
What about 4. testing ?There are three keys to successful software development:
- Testing,
- Testing, and
- Testing.
-
@toshir0 said:
@AndyCanfield said:
What about 4. testing ?There are three keys to successful software development:
- Testing,
- Testing, and
- Testing.
nope, three shall be the number thou shalt count, and the number of the counting shall be three.
-
You'd better have a good rollback policy, you'll be using it soon.
-
-
@AndyCanfield said:
There are three keys to successful software development:
- Testing,
- Testing, and
- Testing.
Now we just need to convince those that currently do:
- coding
- changing project scope
- crunch
-
@Cassidy said:
@AndyCanfield said:
There are three keys to successful software development:
- Testing,
- Testing, and
- Testing.
Now we just need to convince those that currently do:
- coding
- changing project scope
- crunch
I usually go by:
- understand what the hell you're building
- don't be an idiot
-
@dhromed said:
understand what the hell you're building
Yeah.. you see, some customers/clients need careful managing to reach this stage before you adopt their specs...
@dhromed said:
don't be an idiot
.. because sometimes they fail at this.
-
@DOA said:
@OzPeter said:
Of course when I did it it crashed the computer and left some very expensive (and critical) equipment directly in the path of 10 tonnes of white hot steel rapidly coming down the path and no time to get things running normally. I survived that one by manually forcing I/O on the control computer to get the equipment out of the way and gain enough time to get things under control.
I just got a visual of this in my head.OzPeter reboots the machine.
*View of moving machinery coming to a halt*
He looks up and sees the jam, turns his head and sees white-hot metal start to come down a long conveyor line. He then turns around and presses a big red mushroom button. Nothing happens. He slams on it again and again.
*View of him from the ceiling from just above a duct carrying cables. One of them is frayed. Sparks shoot out of it.*
He looks at the line again. The metal is moving. Jumps into a seat in front of a terminal and starts typing like a madman. Mission Impossible music starts playing.
*Close up of his face. Sweat is dripping down*
*View changes to over his shoulder*
As OzPeter types away a 3D wireframes of the jammed equipment spins and swivels on the screen. Parts of it are flashing red.
*View of hot steel approaching jammed machine. The machine is twitching trying to move out of the way. View changes to our hapless operator*
Suddenly he stops typing, slams Enter, jumps up and hits the mushroom button again.
*View of the jammed equipment as seen by the approaching load. The machine screeches as it moves out of the way. View changes to OzPeter *
Mission Impossible music hits climax. He stares at the production line covered in sweat and trembling with the after effects of adrenaline.
He has survived another day at the job.
That is a little slice of awesome you served up right there.
-
@RichP said:
@DOA said:
@OzPeter said:
Of course when I did it it crashed the computer and left some very expensive (and critical) equipment directly in the path of 10 tonnes of white hot steel rapidly coming down the path and no time to get things running normally. I survived that one by manually forcing I/O on the control computer to get the equipment out of the way and gain enough time to get things under control.
I just got a visual of this in my head.OzPeter reboots the machine.
View of moving machinery coming to a halt
He looks up and sees the jam, turns his head and sees white-hot metal start to come down a long conveyor line. He then turns around and presses a big red mushroom button. Nothing happens. He slams on it again and again.
View of him from the ceiling from just above a duct carrying cables. One of them is frayed. Sparks shoot out of it.
He looks at the line again. The metal is moving. Jumps into a seat in front of a terminal and starts typing like a madman. Mission Impossible music starts playing.
Close up of his face. Sweat is dripping down
View changes to over his shoulder
As OzPeter types away a 3D wireframes of the jammed equipment spins and swivels on the screen. Parts of it are flashing red.
View of hot steel approaching jammed machine. The machine is twitching trying to move out of the way. View changes to our hapless operator
Suddenly he stops typing, slams Enter, jumps up and hits the mushroom button again.
*View of the jammed equipment as seen by the approaching load. The machine screeches as it moves out of the way. View changes to OzPeter *
Mission Impossible music hits climax. He stares at the production line covered in sweat and trembling with the after effects of adrenaline.
He has survived another day at the job.
That is a little slice of awesome you served up right there.
When it was first posted I did not read this but after RichP gave a thumbs up I did read it and it is actually awesome. Could probably sell that to a Hollywood bigwig; it would end up being a movie about a blind whale being rescued by a little girl and her turtle (in 3D) but still it's a good piece!
-
@Speakerphone Dude said:
@RichP said:
@DOA said:
@OzPeter said:
Of course when I did it it crashed the computer and left some very expensive (and critical) equipment directly in the path of 10 tonnes of white hot steel rapidly coming down the path and no time to get things running normally. I survived that one by manually forcing I/O on the control computer to get the equipment out of the way and gain enough time to get things under control.
I just got a visual of this in my head.OzPeter reboots the machine.
*View of moving machinery coming to a halt*
He looks up and sees the jam, turns his head and sees white-hot metal start to come down a long conveyor line. He then turns around and presses a big red mushroom button. Nothing happens. He slams on it again and again.
*View of him from the ceiling from just above a duct carrying cables. One of them is frayed. Sparks shoot out of it.*
He looks at the line again. The metal is moving. Jumps into a seat in front of a terminal and starts typing like a madman. Mission Impossible music starts playing.
...That is a little slice of awesome you served up right there.
When it was first posted I did not read this but after RichP gave a thumbs up I did read it and it is actually awesome. Could probably sell that to a Hollywood bigwig; it would end up being a movie about a blind whale being rescued by a little girl and her turtle (in 3D) but still it's a good piece!
All that molten metal, going in the way of a heavy machine carrying the blind whale into the trunk! That movie will be great!
