It's in production
-
Me and a DBA just spent 12 hours between us tracking down a performance problem, creating indices and tweaking some queries. When I sent email up the chain, I was told that we don't need to fix that because it's (the indices we created) in production.
Really?
Then Why TF are those indices NOT in any of the test databases, pre-prod databases, qa databases or end-user certification databases?
I had the dba do a full schema compare of production to each of the corresponding schemas (we use 4 different schemas for various types of data) in each of the above-mentioned databases, and found 487 differences!
I sent the email up the tree to common management and they blew a gasket.
Common management asked: How do these things happen?
I told them there's only one logical explanation: people are hacking stuff in production and not propagating changes backward.
They said that people should not be making changes in production.
No kidding, but we don't have a prod-equivalent db in which to test things, so folks test in our development test production environment.
They blew another gasket.
Whee!
-
At least the elastomeric sector is doing well.
-
The fact that gaskets are being blown and they aren't just shrugging it off with "eh, it works so who cares?" is at least a positive sign.
-
@snoofle said:
Me and a DBA just spent 12 hours between us
Knowing how hard the DBAs and SnoofleCorp work, that must have been a very busy 11h55m for you.
-
@snoofle said:
Common management asked: How do these things happen?
I told them there's only one logical explanation: people are hacking stuff in production and not propagating changes backward.
They said that people should not be making changes in production.
"That's impossible, we have a written policy saying not to do that!"
-
@snoofle said:
I told them there's only one logical explanation: people are hacking stuff in production and not propagating changes backward.
They said that people should not be making changes in production.
At least they understand this. How many clueless people would say "Well, when an issue comes up we need to fix it right away so we patch production. Oops, we forgot to tell dev what we did" and think that's perfectly acceptable.
-
@blakeyrat said:
The fact that gaskets are being blown and they aren't just shrugging it off with "eh, it works so who cares?" is at least a positive sign.
You know, I think that's the most insightful thing I've ever heard you say on here.
-
The production indices were probably developed and applied by a DBA troubleshooting a performance problem. Those guys seem to like to be in monitoring/ firefighting mode all of the time. They often don't ask "what other database instances might benefit from this change?". In fact, many of the DBAs I've worked with tried to wash their hands of nonproduction databases altogether; the excuse is "non-DBAs can touch this, so I can't really do my thing on it." Should the DBA have at least given some developer(s) a "heads up" about the indices? One might think so, but the DBA mindset doesn't necessarily agree. After all, they don't call you up every single time they apply a service pack or reboot a server, and to them it's all the same. They're just reactively doing things to keep their "boxes" going.
I'm not saying there isn't a WTF here, just that TRWTF is "administrator think,"i.e. the notion that just monitoring systems and waiting for a manager/ customer to whine about something is worthy of a full-time "administrator" or "engineer."
-
@blakeyrat said:
The fact that gaskets are being blown and they aren't just shrugging it off with "eh, it works so who cares?" is at least a positive sign.
I'm willing to bet that gaskets aren't the only thing being blown in this company.
-
@da Doctah said:
@blakeyrat said:
The fact that gaskets are being blown and they aren't just shrugging it off with "eh, it works so who cares?" is at least a positive sign.
I'm willing to bet that gaskets aren't the only thing being blown in this company.I'm willing to bet that they don't actually even buy gaskets, they just use that nasty red Form-a-Gasket crap.
-
@snoofle said:
people are hacking stuff in production and not propagating changes backward.
That's a WTF there for me. Changes should be deployed in the order testbed->staging->production, not production then backported to the testbed.
@snoofle said:
They said that people should not be making changes in production.
<pedant> then how does anything get updated</dickweed>
What they really mean is "only authorised people should be deploying tested and approved releases onto production infrastructure", but I'm guesing they're of the mindset "we said it should be this and thought by our word it would magically become this without any effort to ensure this actually happens".
The wider questions are: "who changed production, should they be permitted to change production, what measures are in place to prevent adhoc change like this?"
@snoofle said:
No kidding, but we don't have a prod-equivalent db in which to test things, so folks test in our development test production environment.
I think you mentioned in the past that DBAs were refusing to invest resource (both staff capability and tangible hardware) into creating a test infrastructure that closely reflected production. Is this about to change now? Will heads roll?
-
@DrPepper said:
How many clueless people would say "Well, when an issue comes up we need to fix it right away so we patch production. Oops, we forgot to tell dev what we did" and think that's perfectly acceptable.
It also shows there's no regular auditing being conducted to determine if the two differ, allowing one or two changes to accumulate to 488...
-
@bridget99 said:
I'm not saying there isn't a WTF here, just that TRWTF is "administrator think,"i.e. the notion that just monitoring systems and waiting for a manager/ customer to whine about something is worthy of a full-time "administrator" or "engineer.
Last time I had a say in something like that, we used the very few administrators as a shield against developers that were too numerous, and thus potentialy newb. Some had freee pass to look into stuf and help the admins, but that was earned, not a default. Yeah, they also kept everything working, but most of the time was spent negotiating with developers and trobleshooting bugs.
By the way, if I quote the last character of a post, the forum barks. WTF! I had to disable NoScript for an instant and the entire web is broken (no, this is not the only site that breaks)... Shouldn't it be the other way around?
-
@Mcoder said:
if I quote the last character of a post, the forum barks. WTF!
Surely it would be “Woof!” and not “WTF!”? I mean, I've heard a few dogs over the years, and they've never barked like you say the forum does…
-
@da Doctah said:
@blakeyrat said:
That cute red-head in accounting can blow my gasket any time.The fact that gaskets are being blown and they aren't just shrugging it off with "eh, it works so who cares?" is at least a positive sign.
I'm willing to bet that gaskets aren't the only thing being blown in this company.
-
-
It's not the blown gasket that gets you, it's loosening the head bolts, lifting the head off the block (and that's a two person job, at least), then cleaning the mating surfaces and the little bolt holes with a razor blade and a wire brush before reassembling the whole thing, sealing it with silicone and finally tightening each of the nuts in turn with a 65 in-lb torque wrench.
Wait... Is it possible to take a car analogy a too far?
-
Do you have a database refresh schedule? We have six times a year! We make copy of client's production box and put that copy in our local database. That way we get access to latest and greatest.
