I might be TRWTF



  • So we have a couple of servers.

    • they need to be set reasonably close to the correct time
    • they're airgapped, so they can't just ask time.windows.com for the current time
    • the mission-critical software running on them will shit bricks if the time changes too much
    • all of this necessitates periodically changing the time manually, by small increments

    I checked them for the first time in a while today and calculated that they were slow by 5:21. This is how I fixed it:

    Given the conditions, it seems like it was a pretty reasonable thing to do...



  • What Google does is atleast as much a wtf. They smooth time to remove leap seconds.. As in they have a custom dns that evens it out. It annoys me that time is that complicated.
    Also: why the fuck can't your servers keep accurate time?


  • Notification Spam Recipient

    @anotherusername said:

    seems like it was a pretty reasonable thing

    I dunno, those are pretty high values of time, isn't that like, 3000 ms? You sure your mission-critical software can handle the jump?



  • why don't you set up a private ntp server?



  • @swayde said:

    Also: why the fuck can't your servers keep accurate time?

    Because they can't check with a time server to figure out whether they're running fast or slow, and adjust.

    Computer clocks are really bad at keeping time, apparently.

    I dunno.

    @Tsaukpaetra said:

    I dunno, those are pretty high values of time, isn't that like, 3000 ms? You sure your mission-critical software can handle the jump?

    It probably causes a bunch of timeouts, but nothing severe. "Big" jumps cause it to completely lose its cookies, and crash. (Luckily, it's running concurrently on both servers. One of them going down doesn't create too much of a disturbance in the Force.) Even so, the alternative (before I figured out the loop syntax) was to jump it forward/back in increments of 15 seconds or so, and that didn't crash it. As long as I spaced them around a minute apart, that is. Adjusting the clock by several minutes in manual increments of 15 seconds is pretty tedious, though.

    @Jarry said:

    why don't you set up a private ntp server?

    Connected to what time source?

    The servers are NTP servers, but their clocks have to be set somehow, and the internet isn't an option.



  • @anotherusername said:

    Computer clocks are really bad at keeping time, apparently.

    ?

    Just standard quartz crystal counting, like pretty much everything else that tells time in 2016. It's hard to get quartz wrong.



  • Do you not have any clocks in your house that don't set themselves to the internet anymore or something? They drift. Sometimes a little, sometimes a lot. Even if it's a little, it adds up to minutes wrong over a few weeks or a month or so.

    Every clock that I have drifts by some nonzero amount. The one on the dash of my car does. The one on my microwave does. The digital alarm clock in my room does. I set them all to my computer or phone, since those get the correct time from the internet.



  • @blakeyrat said:

    Just standard quartz crystal counting, like pretty much everything else that tells time in 2016. It's hard to get quartz wrong.

    It was a problem in the nineties, where they could loose like a minute per day - when off.

    @anotherusername said:

    Every clock that I have drifts by some nonzero amoun

    I've not had this problem in this decade.
    Your servers are shit.
    If they are really ntp servers they should get time from a GPS source (or something akin to that)



  • @anotherusername said:

    Connected to what time source?

    The servers are NTP servers, but their clocks have to be set somehow, and the internet isn't an option.

    let me see if i understand the problem, you have a bunch of servers without internet conection, and you need all of them to have the same time?

    you turn one of them into the "time master" and the rest ask that one for the time. the master doesn't ask anyone about the time, it just tell the rest of the servers its own time.



  • also, it has to be physically labeled as "Time Master" otherwise it won't work



  • <awaits the really predictable and unfunny "time lord" joke.>



  • @Jarry said:

    as "Time Master"

    Or time lord



  • I can't see any content in this reply 😵



  • </>

    you have been burned 😛



  • And its clock rhythm is going to be a multiple of four. Duh-duh-duh-duh. Duh-duh-duh-duh. Duh-duh-duh-duh.



  • @swayde said:

    If they are really ntp servers they should get time from a GPS source (or something akin to that)

    They could, but wasn't specced to do it. It's not really that important, so manually adjusting them is Good Enough.



  • Dammit


  • Garbage Person

    We have CRAZY time drift problems with our VMs at WtfCorp. Even with regular syncs, occasionally you get a VM that drifts so far out that it can't talk to ActiveDirectory.



  • The time you spend adjusting the clock probably costs a few orders of magnitude more than a few cheap RTCs or GPS recievers.



  • What's this, bizarro world? I thought this was a non problem everywhere. Is AD very sensitive about time? or hypervisor misconfiguration? I thought they shared the HW RTC?



  • @Jarry said:

    you turn one of them into the "time master" and the rest ask that one for the time. the master doesn't ask anyone about the time, it just tell the rest of the servers its own time.

    That's what's done, but the master clock still has to get the correct time from somewhere.



  • oh, i see. is it labeled as "Time Master" ??


  • Garbage Person

    Probably VMware cluster misconfiguration.



  • Yes, it's configured to use its time as the authoritative time source. All the client PCs have NTP clients and sync their clocks to it. Client PCs can have problems connecting and staying connected if their clocks are very badly out of sync with the server.



  • "Oh no, my Kerberos ticket-granting ticket is already expired/not yet valid and I just got it! However will I trade it for passes to get vouchers to get chits to get tokens to maybe authenticate? D'oh well, I'll just sit here and not log in..."



  • "People who put hidden things in the raw are stupid dicks!" – Blakey, circa one month ago



  • @swayde said:

    Is AD very sensitive about time?

    Yes, although I can't imagine how it could drift that far out if it's syncing daily. :wtf:



  • @TwelveBaud said:

    ticket-granting ticket

    Whenever I work with Kerberos, this term always makes my head hurt.


  • :belt_onion:

    I had a firewall reconfiguration block my main NTP server at home. Things got interesting on the domain in a couple days or so, the computers were a good 5 or so minutes out of sync.



  • Host/AD having wrong time can also cause a lot of disruption for Hyper-V VMs which have host clock synchronization enabled but also synchronize over the internet. Fun times when your VM system clock jumps 5 minutes back and forth constantly.



  • @anotherusername said:

    they're airgapped, so they can't just ask time.windows.com for the current time

    If you're in charge of your local DNS, spoofing time.windows.com and *.pool.ntp.org to resolve to your local time master saves a shitload of easily-overlooked reconfiguration.



  • On danger of useful information – GPS receivers can act as a pretty high-stratum timesource, as can dedicated time radio signal receivers. (The buzzterm for me would be DCF-77 to get Ze Offizial German Taime from Fränkföört, this obviously does not scale to leftpondia.)


  • BINNED

    @brotherelf said:

    Ze Offizial German Taime from Fränkföört

    Jawohl!


  • BINNED



  • @swayde said:

    Is AD very sensitive about time?

    A client's time has to be within 5 minutes of that of its domain controller:

    "To prevent 'replay attacks,' Kerberos V5 uses time stamps as part of its protocol definition. For time stamps to work properly, the clocks of the client and the domain controller need to be in sync as much as possible ... If the difference between a client clock and the domain controller clock is less than the maximum time difference that is specified in this policy, any time stamp that is used in a session between the two computers is considered to be authentic.
    ...
    Default: 5 minutes."


  • Notification Spam Recipient

    @anotherusername said:

    Do you not have any clocks in your house that don't set themselves to the internet anymore or something? They drift. Sometimes a little, sometimes a lot. Even if it's a little, it adds up to minutes wrong over a few weeks or a month or so.

    Every clock that I have drifts by some nonzero amount. The one on the dash of my car does. The one on my microwave does. The digital alarm clock in my room does. I set them all to my computer or phone, since those get the correct time from the internet.


    I use to have a watch with face clock and a digital display. The face clock would fall about five minutes behind by the end of the month. It was a shit watch. My current watch automagically keeps time now and is usually accurate to the second with my computer's time. Except for those times when it inexplicably forgets what time zone its in and its suddenly on Australian time.


  • Discourse touched me in a no-no place

    @swayde said:

    What Google does is atleast as much a wtf. They smooth time to remove leap seconds.. As in they have a custom dns that evens it out. It annoys me that time is that complicated.

    Why are they using DNS? Wouldn't NTP be better for that sort of thing?



  • Ach weel.. :headdesk:



  • @brotherelf said:

    On danger of useful information – GPS receivers can act as a pretty high-stratum timesource, as can dedicated time radio signal receivers.

    That would work, and the idea was bandied about when this whole setup was going in, but it ended up not happening. Setting the clock by hand once every month or two is not really that inconvenient.

    @DogsB said:

    My current watch automagically keeps time now and is usually accurate to the second with my computer's time. Except for those times when it inexplicably forgets what time zone its in and its suddenly on Australian time.

    Sounds like it has GPS in it. Its clock still drifts, but it can calculate very precisely how much it drifts and adjust its time accordingly.



  • Disagree that abstracting away a concept that 99.9999% of people do not care about is a :wtf:


  • Java Dev

    Doesn't wall-socket alternating current also function as a time source? Though I wouldn't be surprised if computers don't have access to it because the PSU shields it from them.



  • @PleegWat said:

    Doesn't wall-socket alternating current also function as a time source?

    Allegedly. Although it requires some dude at the power plant to adjust its clock on a daily basis, and the only reason it's kept around anymore is for legacy raisins. Apparently a lot of clocks are hard-wired to the mains and use the AC for timekeeping porpoises. My reaction is a mixture of :wtf: and :facepunch:.

    @PleegWat said:

    I wouldn't be surprised if computers don't have access to it because the PSU shields it from them

    You misspelt "big-ass UPS". But yeah... computers mostly use low-voltage DC, hopefully smoothed out to the point where you couldn't use it to keep time even if you wanted to. Nothing saying that they couldn't have designed them differently from the get go, though...

    'course, computers can usually run off either 60 Hz or 50 Hz, so it'd have to keep correct time no matter which it was plugged into. And I'm not sure if they necessarily try in other countries to keep the average number of cycles per day relatively constant like they do here.


  • Java Dev

    @anotherusername said:

    Allegedly. Although it requires some dude at the power plant to adjust its clock on a daily basis, and the only reason it's kept around anymore is for legacy raisins. Apparently a lot of clocks are hard-wired to the mains and use the AC for timekeeping porpoises.

    I know my wall-powered alarm clock keeps significantly better time than the battery-powered quartz clock on my desk. Yes, having an analog quartz clock sitting next to my computer is probably TR :wtf:.

    @anotherusername said:

    You misspelt "big-ass UPS"

    I considered mentioning that as well. But the only way it could work even without one is if the MB had a separate AC clock signal from the PSU, and I'm pretty sure it doesn't.



  • I meant technically. It's bad that the software handling of leap seconds is so bad they have to lie about them. I don't expect the user to care.


  • Discourse touched me in a no-no place

    @anotherusername said in I might be TRWTF:

    The servers are NTP servers, but their clocks have to be set somehow, and the internet isn't an option.

    There's various radio sources that you can use (depending on where you are in the world) or you can hook up a GPS unit. Both are using known one-way broadcasts of time information, and so are ideal in an air-gapped situation; you can prove that no information is leaking out of your organisation. Well, unless management wear tinfoil hats to keep the orbital mind control lasers from rearranging their midichlorians or something…

    We used to run NTP servers (still do; they're even public for now) and with the advent of cheap GPS receivers it became trivially easy to set up.



  • solution looks blank from my phone


  • :belt_onion:

    @dkf I know of an organization (you've probably heard of them) that enforces specific-length gaps between VGA cables so the transmissions can't be read. The tin-foil-hattedness of people knows no bounds


  • Discourse touched me in a no-no place

    @sloosecannon said in I might be TRWTF:

    The tin-foil-hattedness of people knows no bounds

    True, but they can put a data-diode in if they really want. GPS lets them get a (really!) good time signal into the building for very little expenditure.


  • :belt_onion:

    @dkf said in I might be TRWTF:

    @sloosecannon said in I might be TRWTF:

    The tin-foil-hattedness of people knows no bounds

    True, but they can put a data-diode in if they really want. GPS lets them get a (really!) good time signal into the building for very little expenditure.

    But someone might be able to hack the diode! What then?

    Filed Under: Those Martians, man........


  • Discourse touched me in a no-no place

    @sloosecannon said in I might be TRWTF:

    But someone might be able to hack the diode! What then?

    At that point, they can also magic themselves into the datacenter and access the servers' consoles directly. What then?


Log in to reply