Separation of Power



  • When I left work today, the dev DB seemed to be running a little slow. What I didn't know was the prod DB was running a little slow too.

    By the time I got home, without laptop, they were calling me to help diagnose the problem. We are supposed to have one dev db machine, (forget the other environments) and 6 prod db servers in a cluster. It turns out that they are all in the same data center, on the same power grid, on the same network, on the same rack, with only the IP (and RAM/CPU) being different.

    I don't actually know how they wired this up, and they won't admit to it, but the power supply on the dev DB server started dropping voltage and somehow something started stealing diverting power from one of the prod DB servers to the dev server. Then that power supply's output started dropping, so power was stolen from the next DB prod server and so on, until only one prod DB server was running and doing all the work (it must have had a more resilient power supply?)

    Upon discovering this, all the people (network, SAs, DBAs, prod managers, dev managers and me) on the phone decided to discuss how best to fix this, with ideas ranging from a full prod shutdown (in the middle of our busy period), on down.

    When I suggested that unplugging the dev DB server (with the bad power supply) would stop the power drain and everything should simply correct itself, I got lashed because we're crunching for a deliverable and we can't afford to slow down development. When I pointed out that if we miss an internal deliverable, nobody would care, but shutting our customers down in the middle of the crunch period might be noticed and affect the bottom line, there was an interminable silence.

    At that point, I told them they had a solution and I was signing off.

    Tomorrow, when I point out that the whole prod power setup needs to be redone, should be fun.



  • We need a "Best of Snoofle" collection. And this needs to be in it.



  • @Scarlet Manuka said:

    We need a "Best of Snoofle" collection. And this needs to be in it.
    Bah! We need a "Best of Snoofle" front page category!


  • Discourse touched me in a no-no place

     The physics on this don't quite work.



  • @Weng said:

     The physics on this don't quite work.

     

    Fucking electronics, how does it work?

     



  •  Ahh .. but PC power supplies have constant power drains. If the line voltage supplied drops, the current from the power line increases.

    Because of Ohms law, the voltage supplied from the power line will drop as the current from the power line increases .

    Eventually the power supplies drag the line voltage they see into the brownout region.

    One supply with the highest brownout threshold will then shutdown through low line voltage.

    Or one will  or overheat / blow up because they become less efficient because of the high current draw. (this one is evil ..see

    Once one power supply shuts down the others see a higher line voltage so carry on running with reduced reliability because of the high current and near-brownout line voltage.

     

     

    A long time ago I had the pleasure of exclusive use of an Apollo DN10000 RISC workstation for a lunchtime benchmarking session for a radio channel simulator I had written ... (150MIPS in a straight line - using a deep pipeline, 25MIPS on average code  and massive around $100k price tag. At the time our lab used a dual core 22MIPS IBM mainframe as its main computer for 100 users...) 

    The DN10000 had three redundant power supplies .

    It was temporarily installed in a lecture theatre of a UK research lab where the line voltage was quite high (I measured it later as 260 volts or so,  being paranoid about how my code could make the computer do the Star Trek thing) because some heavy electromagnets had been removed but nobody had got round to adjusting the line voltage.

    One of the three DN10000 supplies failed because the line voltage was out of spec.It started  drawing a large current. The computer carried on running. My attention was  drawn to the problem by the smoke coming out of the power plug (UK plugs have fuses in them which get hot ... )  One of the PSUs turned out to be too hot to touch.

     



  • @mikedjames said:

    f the line voltage supplied drops, the current from the power line increases.

    Because of Ohms law, the voltage supplied from the power line will drop as the current from the power line increases .

     

    Explain it to me like I'm a highschool senior*, because this seems nonsensical to me. IANAEE.

     

     

     

    *) That would be a Dutch highschool senior. You can use big words and don't have to anthropomorphize the components (such as PSUs that "see" a voltage, or charges that "want" to be together).

     



  • @dhromed said:

    *) That would be a Dutch highschool senior. You can use big words and don't have to anthropomorphize the components (such as PSUs that "see" a voltage, or charges that "want" to be together).

    The PSU doesn't like to be mocked by your condescending tone.



  • @mikedjames said:

    Ahh .. but PC power supplies have constant power drains. If the line voltage supplied drops, the current from the power line increases.

    Because of Ohms law, the voltage supplied from the power line will drop as the current from the power line increases .

    Eventually the power supplies drag the line voltage they see into the brownout region.

    One supply with the highest brownout threshold will then shutdown through low line voltage.

    Or one will  or overheat / blow up because they become less efficient because of the high current draw. (this one is evil ..see *) 

    Once one power supply shuts down the others see a higher line voltage so carry on running with reduced reliability because of the high current and near-brownout line voltage.

     

    This must be what we sound like to managers.

     



  • Bizarre. I can't imagine a scenario where a computer power supply will cause AC power to sag unless something really bizarre is going on (like you're running off of a little Honda generator, or the AC wiring was patched in with CAT5, or, I dunno). Or, maybe someone home-brewed a modular cluster that shared power supplies "because Google started that way".

    Now if you have a UPS or PDU (Power Distribution Unit) that can do load shedding, that could explain losing machines one by one.

    Another possibility would be heat. if the intense data crunching on the dev DB server spun up all the drives (and something was going on with the AC in the colo room) caused the production server right above it to overheat, dumping load onto the remaining production servers, etc...

     



  • @dhromed said:

    Explain it to me like I'm a highschool senior*, because this seems nonsensical to me. IANAEE.

    Here's a wikipedia article on the subject: http://en.wikipedia.org/wiki/Voltage_drop

    Voltage drop is given by the following formulas:

     Single Phase voltage drop

    Where L is the length of the conductor.

    R is  Resistance factor per NEC Chapter 9, Table 8, in ohm/ft (relates to wire gauge)

    I is the load current in amps.

    So the short answer is voltage drop is directly proportional to current.

     



  • @dhromed said:

    @mikedjames said:

    f the line voltage supplied drops, the current from the power line increases.

    Because of Ohms law, the voltage supplied from the power line will drop as the current from the power line increases .

     

    Explain it to me like I'm a highschool senior*, because this seems nonsensical to me. IANAEE.

    *) That would be a Dutch highschool senior. You can use big words and don't have to anthropomorphize the components (such as PSUs that "see" a voltage, or charges that "want" to be together).

     

    Power supplies have an internal resistance (the resistance of their own components plus other things).  If you decrease the resistance of the external load, increasing the current drawn, the power supply's resistance makes up a bigger proportion of the total circuit resistance so there is a higher voltage drop inside the PSU, so lower output voltage. 

     That explanation is pretty much literally true for a circuit consisting of a battery and a variable resistor, but I think it gets more subtle when you start looking at inductance and active components and such.



  • @blakeyrat said:

    The PSU doesn't like to be mocked by your condescending tone.
     

    I am a thirsty horse! And you know how people get cranky when they are a little dehydrated?



  •  @frits said:


    So the short answer is voltage drop is directly proportional to current.

     

    Yea, but now plop in some real world numbers:

    12 AWG Cu wiring: 1.6215 ohms/1000 ft. (12 AWG would be the required minimum gauge for a 20A branch circuit for the National Electrical Code, Copper is standard, if they're using Al wiring for internal distribution, increase your equipment insurance *now*)

    Assume the distribution panel has a "nominal" voltage (probably valid, if the rest of the colo in this scenario is not experiencing blackouts).

    Assume 100' of wire (including neutral return path, so the load is 50' away from the panel)

    Assuming neutral is properly sized (for 3-phase wye-connected loads with PF != 1.0, such as datacenter loads,  neutral should be oversized)

    Assuming 0 ohm connectors, outlets, splices, etc. (again, the argument is regarding line loss).

    20 A branch circuit drawing exactly 20A:

    Referencing Mr. Ohm

    R: 1.6215 ohms/1000ft   or: .0016215 ohms/ft * 100 ft = 0.016215 ohms of resistance.

    V= I*R 

    V= 0.016215 * 20 A = 32.43 mV.

    Hmm, current draw doesn't seem to be the case, considering the

    Granted, the distance might not be 100'. Ok, if it's 1000' we have 324 mV of drop. 

     



  • @RichP said:

    blah, blah, blah...

    I'm experiencing blakeyrat levels of boredom with this thread.



  • @RichP said:

    <snip>

    I'm getting more like 8 volts for your example (1.98 ohms/1000')

    (2 * 100 *1.98 * 20)/1000 = 7.92



  • @orange_robot said:


    Power supplies have an internal resistance (the resistance of their own components plus other things).  If you decrease the resistance of the external load, increasing the current drawn, the power supply's resistance makes up a bigger proportion of the total circuit resistance so there is a higher voltage drop inside the PSU, so lower output voltage. 

     That explanation is pretty much literally true for a circuit consisting of a battery and a variable resistor, but I think it gets more subtle when you start looking at inductance and active components and such.

     

    Higher load on the output can actually DECREASE the power lost in a switching mode power supply (reference things like continuous versus discontinuous mode operation, etc.). The power supply just increases the duty cycle of the switch circuit. Modern PCs don't use linear regulators for their PSUs (or for their onboard supplies, for that matter). For a linear reculator, yes, it will need to dissapate I * (Vin-Vout) watts worth of power. A switcher operates much more efficiently (power losses include parasitic losses in the magnetics, switching losses, and the ohmic loss across the switch itself, which is very low resistance -- Rds(on) of tens of millivolts are typical).

    The engineers that designed the power supply made sure to design it so that the output voltage is constant regardless of the current draw. The overall drop across the supply will always be Vin-Vout. That's kind of the purpose of the supply.

    You are absolutely correct that things get more complicated when moving out of the DC world and considering inductance, power factor, etc.

    ...Ok, I promise, I'll quit annoying all of you on this topic for a while!

     



  • I am lacking a puzzle piece here. Because I still don't quite get it.


    [...]

    wiki browsin'

    [...]

     

    @mikedjames said:

    PC power supplies have constant power drains.

    Ah, there it is. PSUs will just suck in more juice i.e. lower their resistance dynamically based on fluctuations in the line voltage, yes?

    Because then current increases (back) and the already present voltage drop increases a little more and you get a feedback loop.



  •  @RichP said:

    R: 1.6215 ohms/1000ft   or: .0016215 ohms/ft * 100 ft = 0.016215 ohms of resistance.

    You're off by a factor of 10 here. I don't think that makes a fundamental difference, the numbers remain small, but IANAEE.



  • @RichP said:

    ...Ok, I promise, I'll quit annoying all of you on this topic for a while!
     

    Bollocks. Keep talking.



  • You know how in the MST3K episode Gorgo, there's that annoying radio reporter guy who just won't shut up? And towards the end of the episode the bots run out of riffs and just start screaming "SHUT UP!" over him?

    This thread is like that.



  • Whichever resourse Google gave me lists 1.6215 ohms/1000'

    Wikipedia lists 12 AWG as  1.588 ohms/ft

    My trusty, dusty, Electronics Engineers Handbook lists anywhere from 1.59 to 1.65 ohms/1000 ft, depending upon the hardness of the Copper (ASTM B1-56, B2-52, or B3-63).

    Check your reference, I think you'll find it reads 1.98 mohms/ft.

    Sanity-checking those numbers: 100' of 12 AWG with a 10A load will drop 1980 Volts. That would mean 100' of 12 AWG would not work for wiring a household 120 VAC circuit.

     



  • @blakeyrat said:

    MST3K
     

    What's that?



  • @dhromed said:

    @blakeyrat said:

    MST3K
     

    What's that?

     

    Mystery Science Theater 3000, basically a dude and some puppet robots making cracks over old crappy sci-fi (and horror) movies.



  • @RichP said:

    Whichever resourse Google gave me lists 1.6215 ohms/1000'

    Wikipedia lists 12 AWG as  1.588 ohms/ft

    My trusty, dusty, Electronics Engineers Handbook lists anywhere from 1.59 to 1.65 ohms/1000 ft, depending upon the hardness of the Copper (ASTM B1-56, B2-52, or B3-63).

    Check your reference, I think you'll find it reads 1.98 mohms/ft.

    Sanity-checking those numbers: 100' of 12 AWG with a 10A load will drop 1980 Volts. That would mean 100' of 12 AWG would not work for wiring a household 120 VAC circuit.


     

    At the risk of boring Blakeyrat in a thread he doesn't have to read:

    Please read my edited post (which I'm sure was edited waaaay before your reply but whatever).  Your calculations are off by a factor of ten.  Also I think the difference in K factor has to do with temperature.  The NEC chart (which I referenced) is at 75 C.  Your references may be at 30 C or something.

     



  • @locallunatic said:

    robots

    What's that?



  • Good catch on the 10x error. My internal floating point unit has been known to do that before. Main point was it's not on the order of ohms per foot.

    You may be right on the temperature dependence. IIRC my chart was at 25C. I'm also assuming solid cable (usually valid for premise wiring), not stranded.

    Back to the OP, though, line loss should not affect the PSUs under normal conditions, given typical operating ratings for server PSUs.Of course, he could have 6 power strips daisy-chained off of a 100' extension cord borrowed from the janitors closet as a temporary power setup.

    Snoofie, any updates? I can only imagine what you found behind the rack!



  • @RichP said:

    Sanity-checking those numbers: 100' of 12 AWG with a 10A load will drop 1980 Volts. That would mean 100' of 12 AWG would not work for wiring a household 120 VAC circuit.
     

     That's the least sane sanity check I've ever seen; I was compeled to create an account just to reply to that.  10A dropping 1980V would require 198ohm, or 123,750' of 1.6ohm/1000' 12 gauge wire.  There are at least two factor-of-10 errors in the analysis above that resulted in a 32mV drop.  100' (which is rather too long for a 20A run of 12 gauge wire, 10 would be better) of 1.6ohm/100' cable would have a resistance of 0.16ohm.  20A passing though that resistance would result in a volatge drop of 3.2V, or 2.66% for a 120V circuit.  This is of course out of spec for a 2% drop, but in spec for a 5% drop, assuming all terminations contribute no resistance.



  • @RichP said:

    I can only imagine what you found behind the rack!
     

    Ribs, pectorals, a beating heart and some lungs.

     

     

    Or that's what I found anyway. She stopped screaming/whining/gurgling shortly afterwards, so I think the bitch got the message.



  • @snoofle said:

    By the time I got home, without laptop, they were calling me to help diagnose the problem. We are supposed to have one dev db machine, (forget the other environments) and 6 prod db servers in a cluster. It turns out that they are all in the same data center, on the same power grid, on the same network, on the same rack, with only the IP (and RAM/CPU) being different.

    I don't actually know how they wired this up, and they won't admit to it, but the power supply on the dev DB server started dropping voltage and somehow something started stealing diverting power from one of the prod DB servers to the dev server. Then that power supply's output started dropping, so power was stolen from the next DB prod server and so on, until only one prod DB server was running and doing all the work (it must have had a more resilient power supply?)

    I think I can guess.  They've daisy-chained them all, using the power-out-to-monitor plug on the back of each server to supply power in to the next one.

    Just thank your lucky stars the dev server doesn't have the monitor-off power saving setting enabled!




  • Exactly the point. The earlier post had a typo of 1.98 ohms per foot, not per 1000 feet, which would be insane values of resistance.

    Two issues with my post:

    A) I wasn't clear about the value I was illustrating (the 1.98 ohms/foot)

    B) Frits corrected his typo long before I got around to hitting "post" rendering the rant invalid anyway.

    Back to the original premise: Would 3.2V drop cause the servers in question to fail? unlikely.



  • All: I got some additional info today. I don't know what parts they used, but somehow the (external) power source they were using has an outlet for additional equipment. They plugged one of the prod servers into that. Although I don't know the details, I kept hearing the phrase daisy-chained. The dev server was at the end of the chain. When it's power supply start to flip out, apparently the draw/surge/whatever caused a chain reaction until all the daisy chained servers (except the one plugged directly into the power supply) shut down.

    Needless to say they brought in a separate UPS for each machine.



  • @frits said:

    Here's a wikipedia article on the subject: http://en.wikipedia.org/wiki/Voltage_drop

    Voltage drop is given by the following formulas:

     Single Phase voltage drop

    Where L is the length of the conductor.

    R is  Resistance factor per NEC Chapter 9, Table 8, in ohm/ft (relates to wire gauge)

    I is the load current in amps.

    TRWTF is people who express physics formulae using unitless quantities and then have to define their units externally. Especially when they then only define the units for some of their quantities, even though not all the others are unitless.


  • @pjt33 said:

    TRWTF is people who express physics formulae using unitless quantities and then have to define their units externally. Especially when they then only define the units for some of their quantities, even though not all the others are unitless.

    Yep.  TRWTF. You found it. :|



  • @snoofle said:

    Needless to say they brought in a separate UPS for each machine.

    More daisies!



  • @snoofle said:

    Although I don't know the details, I kept hearing the phrase daisy-chained.

    Ahh.  Smug mode.



Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.