What is a test server?
-
Recently, we got a new feature request for our web application. To implement this change, we have 3 environments:
- Test
- Pre-Production (uses all production databases and servers, but is not live for the public)
- Production.
Our management and testing team somehow misunderstand the purpose of each environment.
- Day 1: Development has just released a new feature for testing in our testing environment
- Day 2: Management complains that our feature is late and there is no test version available. I tell them again that it is in our test environment and are ready for qa
- No word for 2 days
- Day 5: Management complains again that our feature is late and there is no test version available. I tell them again that is in our test environment and that we are waiting for qa
- Day 6: Management complains that our feature is not available in the pre-production environment and therefore is not testable. I ask them what is wrong with the test environment, but they decline to comment.
- Day 7: Not wanting to waste more time, I push the feature to the pre-production environment and it gets tested. Management gives the ok to go live.
- Day 8: We finally go live with the feature. Our sales team, upon checking our feature live in production for the first time, tells us the feature is wrong.
- Days later: We eventually get it right and push it live once and for all
After the release, I reviewed a report about my project and its latest feature. It said our feature was delayed due the feature's unavailability to test for 6 days.
My dream: Testing gets done in the test environment, and final approval by sales gets done in pre-production.
My reality: Testing gets done in pre-production, final approval gets done in production.To this day I ask them, "Can we do tests in the test environment?", and I still get no proper response.
-
Our first project on my first team on my first company just went live. There WAS a test server, and the features have been tested, but somehow almost every test failed to point that the features were either broken, or not meeting the specs. That, and the specs changing during and after it going into production, among other things.
-
@OhNoDevelopment said:
After the release, I reviewed a report about my project and its latest feature. It said our feature was delayed due the feature's unavailability to test for 6 days.
I hope your review contained something similar to "This report is telling lies. The feature was available to test for 6 days.".
-
@havokk said:
@OhNoDevelopment said:
After the release, I reviewed a report about my project and its latest feature. It said our feature was delayed due the feature's unavailability to test for 6 days.
I hope your review contained something similar to "This report is telling lies. The feature was available to test for 6 days.".
Not with production data it wasn't... or at least that part is not made clear. Is the test environment a recent copy of the production databases?
-
@atipico said:
Our first project on my first team on my first company just went live. There WAS a test server, and the features have been tested, but somehow almost every test failed to point that the features were either broken, or not meeting the specs. That, and the specs changing during and after it going into production, among other things.
All the code in the second place I worked was deployed directly to Live. It was also developed on a shared codebase. Not even a "Visual Source Safe" shared.
1) Log into Windows
2) putty to development machine
3) vi /codebase/FileToWorkOn.php
Needless to say, half the chatter in the office was "Can you save and exit FileToWorkOn.php?"
-
I think you've got common syndrome of "not done here" where you deploy code, QA tests it and gives you more work to fix or gives it the go. Don't get me wrong, lots of places work this way because a huge QA team needs to be justified. What I've found that usually works is to do all the CI against a test environment and do some automated functional testing there. In staging you don't want anything automatic but manual functional/stress testing (by QA of course).
Anyway, this three stages process was invented by whom? I've worked in several places and several projects and the same process, which seems so simple, somehow is always wrong.
The first problem is with the dev-test environment where this are always unmaintained servers with corrupted/inconsistent db data, unless you have someone specially assigned for this purpose. In my case I prefer to use a memory database which is cleaned up with the same data on each CI iteration.
The second problem is with the staging servers, which given sometime, will end up as the dev-test servers: unmaintained servers with corrupted/incosistent data and worst of all, with outdated data since you're probably not syncing your live db with the staging one. Also, that premise of having staging servers which are identicall to the live ones is pointless.
Finally, the whole process of deploying from one environment to another is such a waste of time unless you have a really good team who is able to automate the whole process but either way, it doesn't come without any friction (I remember a place where with each release, we had issues with people using the wrong names on their SQL scripts because the namespaces/schemas/database were named like xxx-dev, xxx-stg, xxx-live and when you executed an script in live with the table xxx-stg.user the whole process failed, hunt for the blame, fix the issue, and start all over again.
Is there a better way? I doubt that producing code in the live environment is the best thing if you want to maintain some sanity.
Can't we skip the stg environment? Sure, shall we run our stress/functional/integration tests againts live?
Maybe we can skip the dev-test environment? Go back to top.
-
@ubersoldat said:
we had issues with people using the wrong names on their SQL scripts because the namespaces/schemas/database were named like xxx-dev, xxx-stg, xxx-live and when you executed an script in live with the table xxx-stg.user the whole process failed, hunt for the blame, fix the issue, and start all over again.
Sounds like a good idea against oopsies involving pointing the right script at the wrong DB.
-
I worked for a large company doing mostly front-end web development. Their subversion system was, in my opinion, pretty good, but complex to pick up the first time, and I was one of a fleet of developers hired on contract to work with this. Before I go on a rant I would like to say that I generally liked the people and I consider this to be the place where my life really started, having spent 2.5 years previously in a relative hell, but anyway.
The system was as such:
- Take a branch of production, named after a feature-number given by lotus notes (I hear bad things about it, but in this case it was not the WTF).
- Every change made is committed to the branch. Your dev environment folder is switched into that branch.
- When finished, commit into the branch and merge the branch with the test branch to view in the test environment for someone else (e.g. author of the feature) to sign off.
- If wrong, go back to dev, commit into branch, merge into test again.
- If signed off, merge into production.
Every new contractor accidentally merged their dev branch into production when they intended to merge it into test at least once, which was quite easy to do. This was easy to fix - reverse merge.
The part I found baffling was when experienced people seemed to think they were above said protocol:
Case 1:
Long story short, I merged into test, told the 'customer' (I use quotes because they were in-house) it was ready for sign-off, and did something else. They messaged me back "It's not there...". I checked (indeed it wasn't). I checked svn and one of my colleagues has deliberately undone it. After some time investigating why they had deliberately copied over my code, it turned out they just thought it didn't look right, without consulting the feature request to find out why it was there, instead of checking svn for the feature number and questioning me. I spent two days undoing the damage he had done and redoing my code all over again (this was more investigation of people than development of code). He probably wasted 16 company man hours and 2 of my hours in the process. So TRWTF is that once I spotted what he had done I didn't simply assign it to him with a message to the 'customer' (CC'ing him) saying "My colleague wishes to negotiate this feature, I'll let you two decide what's going on.".
Case 2:
Long story short, another colleague had named a development branch after the filesystem being altered, as though that information wasn't already redundant while the feature number was missing altogether. TRWTF is that I didn't just politely say to my team leader in the next scrum "He broke protocol so it took me a long time to find anything."
Case 3:
On some Friday, somebody of rank close to CEO asked for item X, I developed, requested sign off, got it signed off, it went live, fine. Later I got a message "we want to change Y in X". There should be a new feature request if it's gone live, and we're supposed to have a 5-day SLA, but whatever, I was dealing with a much higher rank. Implement Y, merge dev into test, request sign off, get sign off, merge into production. Repeat three or four times that day. I got none of my other work done because constantly being taken off what I'm doing to focus on this takes up time. Why they couldn't test it in the test environment I don't know. Ultimately I don't think I spent much of my own time on it so it wasn't too bad. The following Monday one of my colleagues made some comment along the lines of "cor look at all the stuff Shoreline hasn't done". Fortunately before I thought to commit assault the team leader mentioned I had a bad day Friday. More mildly annoying than a WTF.
Case 4:
Busy day. Did some work from a feature request. Requested sign-off. Customer requested something out of scope to be done urgently. They should sign this off and make a new feature request but whatever. I did it and requested sign off. They requested something else out of scope, I did that and requested sign off. They requested a third scope-external item. Maybe my need to concentrate on what I was doing made me too terse to write the message more diplomatically, but my message in response said I had other things to work on and therefore we have an SLA. The response back came from their manager complaining about this and CC'ing various higher ranks, and using the phrase "do your job". TRWTF is how strongly I regret not standing up for myself and explaining SLA, concentration span, how they're wasting time, and how it looks like they're not doing their job properly, instead of just giving in. Maybe I just wanted it to go away so that I could get on with my work on what was already a busy day, maybe I lacked self-confidence from the 2.5 years I spent in relative hell.
So no real advice, just some rants.
-
@russ0519 said:
@havokk said:
I hope your review contained something similar to "This report is telling lies. The feature was available to test for 6 days.".
Not with production data it wasn't... or at least that part is not made clear. Is the test environment a recent copy of the production databases?
That's because if it was using the production database, it wasn't in testing, it was live.
OhNoDevelopment's place does QA on production, but puts some lipstick by calling it by another name.
-
Simply rename the environments: Development, Test and Production.
Just be sure you're vague about which changed to what.
-
@Shoreline said:
Case 1:
Long story short, I merged into test, told the 'customer' (I use quotes because they were in-house) it was ready for sign-off, and did something else. They messaged me back "It's not there...". I checked (indeed it wasn't). I checked svn and one of my colleagues has deliberately undone it. After some time investigating why they had deliberately copied over my code, it turned out they just thought it didn't look right, without consulting the feature request to find out why it was there, instead of checking svn for the feature number and questioning me. I spent two days undoing the damage he had done and redoing my code all over again (this was more investigation of people than development of code). He probably wasted 16 company man hours and 2 of my hours in the process. So TRWTF is that once I spotted what he had done I didn't simply assign it to him with a message to the 'customer' (CC'ing him) saying "My colleague wishes to negotiate this feature, I'll let you two decide what's going on.".
You couldn't just do a reverse merge back to the old version?
-
So many of the situations described here can be addressed:
1) use proper security to prevent "accidental" pushes. There should be only one [ok, with redundencies] place where a production push can be done, and it should NOT be on any developers desk or credentials. Successful teams often have this machine(s) secured in the NOC, and require multiple people to gain acess to the system [multi-factor by person authentication]
2) Grow a backbone. If there is a policy that "code must be validated in test environment", then there is NO push to anywhere from test, unless there is a formally recognized test acceptance. If *told* to do a push without it [and assuming you ae not in one of the locations where true slave/indentured labor is legal] then refuse. If it comes to that (and depending on your location, and employment terms) it can get very interesting. In a fair number of locales (and where people have not signed away the right) being fired because you would not violate an official company policy can really backfire on the company....
-
@Shoreline said:
I got none of my other work done because constantly being taken off what I'm doing to focus on this takes up time...
The following Monday one of my colleagues made some comment along the lines of "cor look at all the stuff Shoreline hasn't done".
Said colleague was correct - the reason you didn't get your other work done was simply because you'd been tasked on other work by someone playing the RHIP card.
I would have taken their requests to the TL or PM and requested re-prioritisation. At least then others would have been aware of what your time had been spent on (and said VIP can be named as accountable for holding other work back)
-
@TheCPUWizard said:
2) Grow a backbone. If there is a policy that "code must be validated in test environment", then there is NO push to anywhere from test, unless there is a formally recognized test acceptance. If *told* to do a push without it [and assuming you ae not in one of the locations where true slave/indentured labor is legal] then refuse.
+1
If someone more senior wants to circumvent policy then I want a signed document holding them accountable.
-
@russ0519 said:
@havokk said:
@OhNoDevelopment said:
After the release, I reviewed a report about my project and its latest feature. It said our feature was delayed due the feature's unavailability to test for 6 days.
I hope your review contained something similar to "This report is telling lies. The feature was available to test for 6 days.".
Not with production data it wasn't... or at least that part is not made clear. Is the test environment a recent copy of the production databases?
The test environment has a non-recent copy of the production database. This new feature being developed was using new database tables, so a completely up-to-date copy of production was not necessary.
There was no formal process defined. We could ship things directly to production for all they cared, with no one to be held accountable. We have 3 environments intended for specific reasons and I expected them to be used as such.
-
@OhNoDevelopment said:
There was no formal process defined. We could ship things directly to production for all they cared, with no one to be held accountable.
Organisational WTF there. Without clarity of ownership, expect frequent finger-pointing blamestorming sessions.
-
@Cassidy said:
@Shoreline said:
I got none of my other work done because constantly being taken off what I'm doing to focus on this takes up time...
The following Monday one of my colleagues made some comment along the lines of "cor look at all the stuff Shoreline hasn't done".
Said colleague was correct - the reason you didn't get your other work done was simply because you'd been tasked on other work by someone playing the RHIP card.
I would have taken their requests to the TL or PM and requested re-prioritisation. At least then others would have been aware of what your time had been spent on (and said VIP can be named as accountable for holding other work back)
+1 to that. "VIPs" can vacuum all your billable time away if you fail to keep your line manager informed. Especially so if they've got away with doing this to you before ("Development won't do my Really Urgent Change! Something about SLA or something?" "No problem! Speak to Shoreline: he always does stuff for us ASAP.")
As Cassidy said, if your line manager approves ("Yeah, you'd better do that VIP thing right now.") then it is THEIR potential problem later, not yours. It's also likely to be mentioned in reports at some level, so you'll be golden.
Corollary: never allow ANY "VIP" to get the idea that you'll do anything for them, anytime, and screw the process.
-
This reminds me of a wtf that I was a part of several years ago. I needed to make a change to a company administration system due to a change in the reporting protocols in Ireland. We only had one customer who dealt with Irish companies and they wouldn't allow us access to their data. We only got told about the change a few days before it went active so I didn't have time to set up a test database, so I got the customer's IT department to clone their database so they could test my changes.
A side wtf was that after discussing the changes with the customer, they had an entirely different understanding of what was needed to me, and my boss insisted I did it their way. I made the changes, gave them the new program, and they tested it on the cloned database. Of course I soon got a response that the changes had been done wrong, and they then told me it should have been done exactly as I had orignally said.
So, I made the new changes and sent it to them for testing. Shortly afterwards, I got a response saying I had done it wrong. I had reversed the check in the update script and changed all the non-Irish customers instead of the Irish ones. No biggy right? It had been run on the cloned databse so I could quickly reverse the check, re-test and all is well. Nope, the customer had run it against the live database without testing first. All non-Irish companies now had incorrect reporting dates set up, and of course there was no record of what the old dates were. Cue panic, now 12,000 odd companies in their database have invalid data.
Fix turned out to be not too bad, although it pushed it close to the deadline. First run a script that takes the correct dates for non-Irish customers from the cloned database and changes them in the live database, then run the amended update with the check the correct way round. A good illustration of why testing is important.
-
@Malenfant said:
A good illustration of why testing is important.
"if you don't test it before releasing into production, you don't mind if it fails after releasing it into production" .
I do think that the term "testing" can present the wrong perceptions - people often believe it's all about proving it works, and the coder's done that already, right?
No.
I prefer the (management bullshit) term "Quality Assurance" because people feel happier about discussing "we're just establishing that it meets the high quality standards we're know for so that we can continue to deliver quality products with confidence".
@Malenfant said:
We only got told about the change a few days before it went active so I didn't have time to set up a test database, so I got the customer's IT department to clone their database so they could test my changes.
That's more a PM issue than a testing issue - the quality of the product may suffer if time is short, and a lack of stakeholder analysis prior to the change to engage relevant parties
.@Malenfant said:
... my boss insisted I did it their way. ... and they then told me it should have been done exactly as I had orignally said.
I take it you pointed out to your boss that their decision to follow the client's design first before falling back to yours has cost your company money in wasted effort? I understand that the boss may have taken that course of action simply for the "told you so" factor to the client.
@Malenfant said:
So, I made the new changes and sent it to them for testing...
...Nope, the customer had run it against the live database without testing first.
And this is your problem.. how? You have an audit trail showing clearly that this was "submitted for their testing" not "I've completely tested without any access to your data whatsoever and it works 100%, you're good to go!" They did a production release without proper testing and no backout plan, they can fix what they broke.
-
I never said that any of this was my fault other than the obvious mistake of writing the check the wrong way round, which is mitigated somewhat by the severe time constraints I was forced to work under.
-
@Malenfant said:
]
I never said that any of this was my fault other than the obvious mistake of writing the check the wrong way round, which is mitigated somewhat by the severe time constraints I was forced to work under.
So it passed unit, behavorial, integration tests all within your environment [all of the above should be part of the automated build process, with a good percentage being done before it is possible to commit to SCCM......
-
@Zecc said:
Sounds like a good idea against oopsies involving pointing the right script at the wrong DB.
Surely that never happens!
-
@powerlord said:
You couldn't just do a reverse merge back to the old version?
Something about the way he'd copied over the code made it much less straightforward than that - I think his merge had conflicted with it. Otherwise yes, that's how I'd usually handle mistakes.
@TheCPUWizard said:
1) use proper security to prevent "accidental" pushes.
I strongly agree. In this case, we had an in-house web interface with a dropdown list "test, production". In my opinion, this was not the biggest problem though, and was easy to resolve (reverse-merge mentioned by powerlord).
@TheCPUWizard said:
2) Grow a backbone.
Again strongly agree. My backbone wasn't in great condition due to being treated generally badly for a while, and I was paranoid and insane at the time. That was some years ago though and I'm no longer so frightened of getting into a fight with management if they're going to do something dumb. I have anecdotes of that, but I'll save them for another time.
@Cad Delworth said:
As Cassidy said, if your line manager approves ("Yeah, you'd better do that VIP thing right now.") then it is THEIR potential problem later, not yours. It's also likely to be mentioned in reports at some level, so you'll be golden.
I call this game "follow the accountability". I don't know if there's a better name for it (please enlighten me, and not Ingress style), and it's not the same as the blame game, which is more about pointing fingers where you can get away with it. You can't win at it, but you can try to lose less than everyone else.
-
@atipico said:
Our first project on my first team on my first company just went live. There WAS a test server, and the features have been tested, but somehow almost every test failed to point that the features were either broken, or not meeting the specs. That, and the specs changing during and after it going into production, among other things.
Did you ever show the features to the users who wrote the specifications? Did you have a walkthrough with them, or let them play with the system first? If not, that's your fault, not theirs.
-
@Shoreline said:
I call this game "follow the accountability". I don't know if there's a better name for it
I play "broadcast the accountability", including situations where I was personally accountable for making a fuckup. I ain't blameless.
However, I was taking issue with this statement:
@Shoreline said:
Fortunately before I thought to commit assault the team leader mentioned I had a bad day Friday.
Your team leader didn't give the real reason: you had been tasked with additional work that was deemed higher priority due to the position of the requester, which means you hadn't cleared it through them (and execs now know they can assign ad-hoc stuff to you) or they had been informed but had reasons for not revealing this in the meeting.
I'd have named/shamed the exec in question: I see no reason why I have to hide my activities, and I didn't get the impression the exec was after this work on the sly.
-
@ubersoldat said:
[...] we had issues with people using the wrong names on their SQL scripts because the namespaces/schemas/database were named like xxx-dev, xxx-stg, xxx-live and when you executed an script in live with the table xxx-stg.user the whole process failed, hunt for the blame, fix the issue, and start all over again.
And nobody ever had the bright idea not to include the schema name in the invocation, or to use synonyms?
-
@Severity One said:
And nobody ever had the bright idea not to include the schema name in the invocation, or to use synonyms?
I'm in two minds over this:
- the more automated the process, the greater potential for
Knight Tradingautomated mistakes - the more manual the process, the greater number of manual checks - and the possibility of introducing errrors.
In short, TRWTF is the human element.
Kill all humans. It's the only way to be sure.
- the more automated the process, the greater potential for
-
@TheCPUWizard said:
2) Grow a backbone. If there is a policy that "code must be validated in test environment", then there is NO push to anywhere from test, unless there is a formally recognized test acceptance. If told to do a push without it [and assuming you ae not in one of the locations where true slave/indentured labor is legal] then refuse. If it comes to that (and depending on your location, and employment terms) it can get very interesting. In a fair number of locales (and where people have not signed away the right) being fired because you would not violate an official company policy can really backfire on the company....
I don't know what fantasy land you live in, but most of the English speaking world is on AT-WILL employment terms. This means either party can terminate the employment for any reason at any time, and a reason is not required to be given. Sure if you can prove they fired you for an illegal reason, you can sue but its probably not going to do a lot of good. It might make an interesting story at an unemployment hearing.
n.b. I would still refuse, but I probably wouldn't want to continue working for an employer that would fire you for insubordination for refusing to break company policy.
-
@esoterik said:
@TheCPUWizard said:
2) Grow a backbone. If there is a policy that "code must be validated in test environment", then there is NO push to anywhere from test, unless there is a formally recognized test acceptance. If *told* to do a push without it [and assuming you ae not in one of the locations where true slave/indentured labor is legal] then refuse. If it comes to that (and depending on your location, and employment terms) it can get very interesting. In a fair number of locales (and where people have not signed away the right) being fired because you would not violate an official company policy can really backfire on the company....
I don't know what fantasy land you live in, but most of the English speaking world is on AT-WILL employment terms. This means either party can terminate the employment for any reason at any time, and a reason is not required to be given. Sure if you can prove, in a court of law, they fired you for an illegal reason, you can sue but its probably not going to do a lot of good. It might make an interesting story at an unemployment hearing.Your facts are 100% correct, and I am speaking about at-wil situations where there is great flexability for TERMINATING and employee. In the legal arena there is a subtly difference between simply being terminated, and being FIRED. I specificly was talking about being fired, and based my comment on direct experience [I was not the person fired, but was involved in testimony] in 3 different [USA] states over the past 20 years...
-
@TheCPUWizard said:
In the legal arena there is a subtly difference between simply being terminated, and being FIRED.
Yeah and if you got fired instead of terminated in an at-will context, your employer's legal counsel has an idiot for a client!
-
@Cassidy said:
@Severity One said:
Yes, but there are laws against that.And nobody ever had the bright idea not to include the schema name in the invocation, or to use synonyms?
I'm in two minds over this:- the more automated the process, the greater potential for
Knight Tradingautomated mistakes - the more manual the process, the greater number of manual checks - and the possibility of introducing errrors.
In short, TRWTF is the human element.
Kill all humans. It's the only way to be sure.
The point is to keep code the same between testing, staging and production. If you need to rename tables or schemas when moving to production, you're doing something wrong.
- the more automated the process, the greater potential for
-
@Severity One said:
The point is to keep code the same between testing, staging and production. If you need to rename tables or schemas when moving to production, you're doing something wrong.
+1
@Severity One said:
Yes, but there are laws against that.
I'm convinced there are exceptions that permit legal circumvention in situations of abject stupidity.
