A look back at Y2K - Share your WTF stories here



  • This is a little retrospect into what happened the night of December 30, 1999 that I feel will get some definite "WTF?s".

    While everyone else was partying like it was 1999, I spent the night in my company's war room braced for any Y2K issues. Being naive, right out of college I took a job at a major railroad company doing green screen work. The mainframe was ready. The minutes ticked by. At 12:00:00, everything sat pretty with no major problems. A collective sigh eminated.

    12 minutes later, the red phone rang. A nightly script had crapped itself with the error "Date out of Range". Programmers scambled and midnight phone calls were made. Systems were analyzed to see if anything else had died and just didn't report.

    7 minutes later, it turned out that there was this one field in the code for the script that failed that needed to be updated yearly to say what year it was. (WTF? Anyone heard of getDate() or the COBOL equivalient)?) It was permanently fixed and calm settled again.

    3 minutes later, the red phone rings again. This time, it was much worse. The entire Dallas, TX yard had lost power. Obviously, this wasn't a coding problem on our end, but our war room was the only phone number for support that was written down. Still, we started backtracking and tried to grab the sessions that the Dallas yard was running so that nothing was lost. Everything in that yard halted. Managers nervously watched the news to see if Dallas as a whole had lost power or if some nuclear facility was about to blow.

    Nothing was reported.... we worked and waited....

    Two hours later, we got the news. Some trigger happy Texan decieded to celebrate in the new year by firing off his 6 shooter and accidentally blew a hole in the power transformer that linked to the train yard. Yee-hawtf?.

    Other than that, Y2K was uneventful. I wouldn't report it as a "dud" that the media called it because many qualified programmers, engineers, and other profesionals had worked tirelessly to prepare systems for Y2K and the "dud" was a testament to the quality of their work.



  • <quote>
    This is a little retrospect into what happened the night of December 30, 1999
    </quote>

    Of course it was quiet ... it was December 30th. You should have seen it the next day/night ;-)



  • "Yee-hawtf?"  ROFLMAO!



  • Boy, that's some memories!  I was working for a Major City police department, something like 1800+ officers.  I was network admin/SQL database admin/database developer.  We applied all the patches that we could, reviewed everything we could find the source for, all-in-all we were in pretty good shape.  Fortunately all of my databases used date datatypes, so it was pretty much a non-issue for me.

    Now, I should stress that there were two sides to our computer group, the PC network and the mini/mainframe network.  The latter held all the criminal records, controlled the mobile data terminals (MDTs) in the cruisers, etc.  The former, our side, was mainly administrative and held very little actual criminal information.  Secretarial use, email, etc.

    The "millenium" rolled over, and as far as we could tell, the micro side stood tall.

    A little bit after midnight, the dispatch/911 system went down.

    While prepping for Y2K, they had identified some non-compliant code and fixed it.  This was in the days of edit/compile/link.  They fixed the code, they compiled the code, but they didn't do the final link edit.

    Fortunately the dispatchers were veterans that pre-dated the new dispatch system and switched over to paper notes to keep track of units without difficulty.


    A second story -- all of the MDTs were Motorola 386s with something like a 10 or 20meg hard drives (yes, megabytes) that ran Windows 3.1 or WFW 3.11, I don't remember.  But all they ran was a terminal emulator that tied to the radio that tied to the HP 3000s used by Dispatch.  Both 3.1 and 3.11 were non-Y2K compliant, but Motorola would have been happy to sell us a patch at something like $500 per unit.  I don't know how many cruisers we had, but I think we can safely say 700 or more.

    Well, after a little investigating it turned out that when the officer's logged on to the computer, the terminal software downloaded the date and time from the HP.  Thus, no Y2K issue on a non-compliant piece of hardware.



  • Some time before Y2K, I set the bios of my PII to 2000+ date.

    No issue.
    Happily ticked along.


    Then I tried it on the old 486.

    No issues.
    Happily ticked along.


    So, factually nothing happened.



  • In 1999, we made a project that was in theory relatively simple, but included a lot of technologies that were completely new to us, e.g. MS SNA Workstation.
    Needless to say, there were delays and doubts. But finally we made it. Then, during the final test, the project leader decided to test the system with the clock set to Feb. 29th, 2000, to make sure this "special" date causes no troubles. Worked perfectly, of course. Afterwards, he switched the date back to (say) Aug. 16th, 1999 - and BANG, an important process of the system froze.
    It turned out that this process had a timer that would wait for a certain point in time; since the clock was turned back to 1999, it would have waited till Feb. 29th, 2000 to continue.



  • As far as I recall, nothing bad happened, especially since an
    extraordinary amount of effort went into fixing things, but we were
    prepared anyway.



    A few giant trucks parked up outside with backup computers, a generator
    and a few gallons of diesel, and LOTS AND LOTS OF BOTTLED WATER.



    I really have no idea. They were used a few months later when our water
    mains had an accident, so we were using it partly to drink, but mostly
    to flush the toilets with.




  • In the Summer of 1999, I was a (the) tester in a group doing "Sustaining Engineering" on a legacy OS that our company had acquired through a merger. We subcontracted out most of the actual Y2K investigation to another company, who pored through our entire Unix source code base (4.2 BSD), plus our frameworks and application code. Other than a couple of issues with the year '2000' being displayed as '19100', and a single instance where some cron jobs would potentially be executed twice, they didn't really find anything.

    We patched all of the cosmetic issues and minor bugs, tested the fixes thoroughly, created patch packages for all four hardware platforms we supported, and made them available to customers a good long while before the end of the year. End of story, right? Not exactly.

    As the end of the year approached, our management started to feel more and more pressure to have a "plan" for dealing with any last minute emergencies. Despite the fact that our group had repeatedly demonstrated an inability to get *any* software changes out the door in less than two weeks (including building, testing, and packaging), it suddenly became critical for all of us on the team to be available, all night on December 31st, so if there were any problems at customer sites, we could fix them right away.

    Despite repeatedly explaining that...

    1. Nothing critical was likely to go wrong at the OS level when the clock hit midnight, even for customers that hadn't installed the patches.
    2. We had in fact fixed even the cosmetic issues in patches that had been available for months.
    3. No matter what came up, we'd never get another patch out in less than a week.
    4. Many of us had already made other plans for New Years.

    ...the decision came down from above that the whole team needed to be on site or on call for the whole of our nominal holiday time between Christmas and New Years, and the week afterward.

    Fortunately, right around that time I got a call from a friend at a startup, who had a job for me, if I was interested. So off I went to greener pastures, and the clock ticked over, and the whole rest of the team was on-site or on-call the whole time, and not one of our customers had any kind of Y2K-related issue at all. Most of them, in fact, were off work on holidays when the transition happened.

     



  • I saw customers create issues due to Y2K....

    Novell Netware's NDS uses timestamps to help merge replicated data.  Well, if you set the date of a production server forward beyond the year 2000, nothing happens.  Except, internally is it making time stamps with the year 2000 on them.  When you set the clock back to 199x, the system needs a way to deal with the out-of-order timestamps.  Fortunatetly, Novell thought of this.  The system will never issue a timestamp older than any existing timestamp.  So, in 1998, it will start putting Y2K timestamps on data.  They call this "synthetic time" and the server beeps and give an error message something like "Synthetic time is being issued on partition: XYZ" every two minutes.  Nothing bad happens and synthetic time runs slower than real time, so ventually the system gets back on real time.  But, with sythetic time a full to years ahead of real time, it would take four to six years for the beeping to stop.

    Fixing this problem isn't very difficult, but it does take a very long time to explain to most people what is wrong with their system, and most people are pretty freaked out by the error message.



  • I Returned a video to the video store on the 3rd of January 2000. Apparantly I returned it 100 years to early.



  • @Founder said:

    I Returned a video to the video store on the 3rd of January 2000. Apparantly I returned it 100 years to early.


    And now you are rich with negative late fees?



  • @jsmith said:

    I saw customers create issues due to Y2K....

    Novell Netware's NDS uses timestamps to help merge replicated data.  Well, if you set the date of a production server forward beyond the year 2000, nothing happens.  Except, internally is it making time stamps with the year 2000 on them.  When you set the clock back to 199x, the system needs a way to deal with the out-of-order timestamps.  Fortunatetly, Novell thought of this.  The system will never issue a timestamp older than any existing timestamp.  So, in 1998, it will start putting Y2K timestamps on data.  They call this "synthetic time" and the server beeps and give an error message something like "Synthetic time is being issued on partition: XYZ" every two minutes.  Nothing bad happens and synthetic time runs slower than real time, so ventually the system gets back on real time.  But, with sythetic time a full to years ahead of real time, it would take four to six years for the beeping to stop


    Ouch! Who is crazy enough to do Y2K testing on the production server itself?? The very reason you do a test is because you don't know what is going to happen! And to then roll the clock back and procede with business as usual? Silly, silly people.



  • @RayS said:

    Ouch! Who is crazy enough to do Y2K testing on the production server
    itself?? The very reason you do a test is because you don't know what
    is going to happen! And to then roll the clock back and procede with
    business as usual? Silly, silly people.



    The very reason you do it on the production server is that you want to make sure that the productions server survives Y2K. A test server could be... different.


  • @ammoQ said:

    @RayS said:

    Ouch! Who is crazy enough to do Y2K testing on the production server
    itself?? The very reason you do a test is because you don't know what
    is going to happen! And to then roll the clock back and procede with
    business as usual? Silly, silly people.



    The very reason you do it on the production server is that you want to make sure that the productions server survives Y2K. A test server could be... different.


        Indeed.  At the time, I was system mangler/programmer/bottlewasher for a 911 dispatch center.  I was training a new guy we'd hired to help me with the sysman/programming stuff.  We had already done Y2K testing and fixes and were pretty confident that all would be well.  My trainee wanted to do a test on his own, which was fine with me.  He was sharp and showed spirit, which I liked.  He was following instructions that I had given him for setting the date on our VAXCluster.  (For those who never heard of this, it was a two-node minicomputer arrangement.  In this case the dispatch production machine was one node and our development/test machine was the other.)
        He got into the cluster management application, typed the command to set the time on both nodes, realized this was a bad idea (don't test on a running production machine without notifying people, right?), and hit control-C.  (He did not hit enter first, just had the command sitting there waiting to be sent.)  *Normally* control-C is an abort.  But for some WTF-reason in this case it actually transmitted the command and the time on BOTH machines got changed.  It only took him a minute to realize what happened because things got very weird.  He immediately fixed the time, but by then it was too late.
        In the span of a minute or less, the busy (and near-real-time) production dispatch application did these "normal" tasks:
    *  Closed the history files for the current (November) month, and created new ones for December.
    *  Wrote a few history records to the December files.
    *   Closed the history files for December, and created new ones for January.
    *  Continued writing history records to the January files.
        Of course it was happy as a clam until he fixed the date back to November, at which time it saw the inconsistency with the history files and puked it's guts out.  My trainee then got first-hand experience restoring files from backup tapes.  I'm a masochist and this was technically so fascinating that I LOVED every minute of it.  But he felt horrible and extremely guilty, even though technically it wasn't his fault.  Even I had no idea that the cluster management app considered control-C to be a transmit (I reported this to the OS engineers, but haven't heard if it's been fixed yet). 
        When the dust had settled, we reported to management that we would get through Y2K with NO problems.  We just could never go back.  :-)










  • @ammoQ said:

    @RayS said:

    Ouch! Who is crazy enough to do Y2K testing on the production server
    itself?? The very reason you do a test is because you don't know what
    is going to happen! And to then roll the clock back and procede with
    business as usual? Silly, silly people.



    The very reason you do it on the production server is that you want to make sure that the productions server survives Y2K. A test server could be... different.

    What I meant was shorthand for "on the production server without rolling back to a backup after performing the testing. Of course you test the hardware itself and the specific hardware/softwre combination, but you do it in a controlled manner, putting the server back to the exact same pre-testing state after testing.


  • @RayS said:

    @ammoQ said:
    @RayS said:

    Ouch! Who is crazy enough to do Y2K testing on the production server
    itself?? The very reason you do a test is because you don't know what
    is going to happen! And to then roll the clock back and procede with
    business as usual? Silly, silly people.



    The very reason you do it on the production server is that you want to make sure that the productions server survives Y2K. A test server could be... different.

    What I meant was shorthand for "on the production server without rolling back to a backup after performing the testing. Of course you test the hardware itself and the specific hardware/softwre combination, but you do it in a controlled manner, putting the server back to the exact same pre-testing state after testing.


    I completely agree, yet a lot of people had to learn that the hard way ;-)


  • @jetcitywoman said:


        Indeed.  At the time, I was system mangler/programmer/bottlewasher for a 911 dispatch center.  I was training a new guy we'd hired to help me with the sysman/programming stuff.  We had already done Y2K testing and fixes and were pretty confident that all would be well.  My trainee wanted to do a test on his own, which was fine with me.  He was sharp and showed spirit, which I liked.  He was following instructions that I had given him for setting the date on our VAXCluster.  (For those who never heard of this, it was a two-node minicomputer arrangement.  In this case the dispatch production machine was one node and our development/test machine was the other.)
        He got into the cluster management application, typed the command to set the time on both nodes, realized this was a bad idea (don't test on a running production machine without notifying people, right?), and hit control-C.  (He did not hit enter first, just had the command sitting there waiting to be sent.)  *Normally* control-C is an abort.  But for some WTF-reason in this case it actually transmitted the command and the time on BOTH machines got changed.  It only took him a minute to realize what happened because things got very weird.  He immediately fixed the time, but by then it was too late.
        In the span of a minute or less, the busy (and near-real-time) production dispatch application did these "normal" tasks:
    *  Closed the history files for the current (November) month, and created new ones for December.
    *  Wrote a few history records to the December files.
    *   Closed the history files for December, and created new ones for January.
    *  Continued writing history records to the January files.
        Of course it was happy as a clam until he fixed the date back to November, at which time it saw the inconsistency with the history files and puked it's guts out.  My trainee then got first-hand experience restoring files from backup tapes.  I'm a masochist and this was technically so fascinating that I LOVED every minute of it.  But he felt horrible and extremely guilty, even though technically it wasn't his fault.  Even I had no idea that the cluster management app considered control-C to be a transmit (I reported this to the OS engineers, but haven't heard if it's been fixed yet). 
        When the dust had settled, we reported to management that we would get through Y2K with NO problems.  We just could never go back.  :-)

    lol, that is hilarious



  • 19103

        In 2003, I worked as a teaching assistant for a second level intro computer science course.  I don't recall exactly what the week's assignment was but in some part of it, it required the display of the date.  Somehow, I always got the dumb ones.  This guy in particular managed to take the system date provided by Java, extract the year, subtract from 1900, then re-concatonated it into 19103.  This pissed me off, but not as much as when the head TA said that since "no Y2K bugs" wasn't listed on the criteria, I couldn't dock him points for it.



  • what school was that were it's OK to write programs that don't work correctly for the current date?



  • Actually, I think it is a dud. Unless you didn't use the actual no. of second since the Unix epoch (also known as the actual way time is stored on the machine), you were completely safe.



  • Actually, this problem will occur again in 2030 or something like that. 

    If I recal Microsoft Access is one of those applications that was never fixed correctly.

    So look for a mini-y2k just before you retire.   :)

     

     

     



  • @Kev777 said:

    Actually, this problem will occur again in 2030 or something like that. 

    If I recal Microsoft Access is one of those applications that was never fixed correctly.

    So look for a mini-y2k just before you retire.   :)

    No calling it a mini-y2k.  This is going to be the big one, and I'm depending on it to fund my retirement! :P



  • Yeah, the 2038 bug, its the time when time_t (C/C++) bottoms out, of course in reality by then systems won't even blink at storing numbers that big, so it doesnt matter anyways heh ...

    I was working as an embedded software engineer at the time, writing the firmware for custom dual-head laser jet, and ink jet printers ... These printer systems were large, used for various kinds of test printing by the government and paper manufacturers and what have you ... They ran on embedded x86 Linux systems with a lot of custom breadboards, microcontrollers, etc ... The problem was on every piece of paper it printed to, it printed a time and date stamp to ensure proper records.  We took several production systems that we were still building and changed the dates on them to a minute before Y2K hit, and watched ... The funny part was even though the ONLY thing it would effect was the time and date stamp on the paper, all my coworkers were anticipating all the motor controls to go berzerk and start shooting sparks and what not ... Eventually the minute passed and what do you know, the proper date was bring printed on the paper, all was well ... The amusing part was watching the expressions on my coworkers faces, you'd think someone had promised to buy them lap dances and ended up forgetting their wallet or something :P



  • @Oscar L said:

    @Kev777 said:

    Actually, this problem will occur again in 2030 or something like that. 

    If I recal Microsoft Access is one of those applications that was never fixed correctly.

    So look for a mini-y2k just before you retire.   :)

    No calling it a mini-y2k.  This is going to be the big one, and I'm depending on it to fund my retirement! :P

    Me too. :)

    I say mini because it won't have has much hype as y2k did.   So it will be over looked and likely cause many system failures.   Most problems will only be fixed after the fact and that is where the big bucks roll in for our retirements

    In less then 20 years we created the following layers (just for windows)

    Machine code, ASM, C,  DOS, Win API, C++, MFC, NET,  XML

    Imagine how many more layers will exist in 2038.  How is anyone going to know what layer to look at to even fix the problem?  

    It is going to get out of hand very quickly and you can bet that Microsoft will try to keep the code as platform dependant as possible! 

    Hopefully, the 2038 bug will be the one that actually f-ucks the worlds computer systems over.  hehee.. :)

     

     



  • I think the bigest WTF during y2k was all the usless bullshit I heard from all the high priced consultants that came in

    These guys billed the Beer company I was working for through the ass at over $200/hour just to sit in meetings all day and make useless suggestions and cause fear.

    I recall hearing one idiot with a very thick East Indian accent say "But, Vut is the verisk to da business?" over and over again from the board room.

    While I was fixing the problems he made the real money just talking about them and never provided on line of usefull information.  

     

     

     


Log in to reply