Unique datetime


  • Garbage Person

    So I just got handed this problem to design a solution for.

    Client wants a report with a filename containing a date time stamp of a format like ErrorReportMMDDYYYYHHmm.txt

    We may, under certain circumstances, produce more than one per minute in bursts. The filename, for obvious reasons, must be unique.

    We need a way to generate unique psuedodates using the current time and date as a seed.

    Generalize to arbitrary date formats and resolution/precision.

    What. The. Fucking. Falcon?

    ('make the client accept a reasonable file name' isn't, apparently a valid solution)


  • FoxDev

    ErrorReportMMDDYYYYHHMM.txt
    ErrorReportMMDDYYYYHHMM(1).txt
    ErrorReportMMDDYYYYHHMM(2).txt
    ...
    ErrorReportMMDDYYYYHHMM(65535).txt
    ErrorReportMMDDYYYYHHMM(-65536).txt

    ?



  • Add milliseconds? And if needed, microseconds?


  • Discourse touched me in a no-no place


  • ♿ (Parody)

    @Weng said:

    make the client accept a reasonable file name' isn't, apparently a valid solution

    Has anyone asked the client what you should do when you have more than one report per minute?



  • /Version1s
    /Version2s
    /Version3s
    /Version4s
    /Version5s
    

    // should never have more than 5 per minute


  • Garbage Person

    Point is that the stupid filename can't change even one character from a valid date time in their insane format.

    So, essentially, this calls for a service that takes a format string and returns a previously unallocated timestamp in that format as near as possible to the current time.


  • ♿ (Parody)

    Hmm...txt...just concatenate the files? Have everyone write to the files as output comes up?



  • Have the second file overwrite the first, and when they complain...

    CLOSED_WORKINGASDESIGNED


  • Garbage Person

    Yes. We ask man clients this. It comes up roughly every other implementation. The answer is always "don't worry computers aren't that fast" or "I'm not going to ask my client that, it makes us sound incompetent. Just do something. " (our interactions are proxied by sales and account management goons). Every once in awhile they'll grudgingly allow us to insert a unique ID.

    To date, the most common solution is" throttle the job so we do a maximum of 1 per minute ". Which often leads to hilarious processing backlogs which lead to our being yelled at for being terrible programmers and taking hours to generate a few simple reports.

    Vicious cycle of institutional derp.


  • Discourse touched me in a no-no place

    Can you have more than one 'report' per file? Or merge two reports to make them look like one?


  • Garbage Person

    Not without breaking their own encapsulation rules.

    Also, I think I just flagged you because derpy mobile interface.


  • Discourse touched me in a no-no place

    Keep a circular buffer of MAX_FREE (say 60).

    For every minute that goes by without a report being generated, chuck that minute at the end of the buffer.
    For every minute that doesn't get caught above

    1. the first report gets 'this' minute's timestamp.
    2. every other report 'this' minute pulls a slot off the top of the buffer.


  • @Weng said:

    Yes. We ask man clients this.

    What about the female clients?



  • ErrorReportMMDDYYYYHHmm{newGUID}?


  • Discourse touched me in a no-no place

    @chubertdev said:

    What about the female clients?

    Already covered.



  • Does it need to be an actual valid timestamp? Say can you use 02:61, 02:62, etc?

    Also, is it safe to assume that there will be periods of downtime where you're not generating reports? If you generate > 1/min 24/7 for long enough, you'll end up rolling your year past 9999 and then you'll really have fun (and no, I CBA to figure out exactly how many that would be).

    OK fine, 7985 years * 12 months * ~30.4 days/month * 24 * 60 = only 4,199,790,600 errors before their arbitrary formatting breaks down.


    Yes, yes Discourse would blow through that in a month.


  • Garbage Person

    Thanks to the nature of the beast, it's unlikely to step out more than a few minutes. Probably add some configurable sanity check bounds to keep anyone from bitching about the drift. Possibly go back in time, too.

    I.e. If it's 12:00, return 12:00, then 12:01, then 11:59, then 12:02, etc. This whole thing is so frigging stupid.



  • @Weng said:

    This whole thing is so frigging stupid.

    Yep. Arbitrary limits by non-programmers who won't listen to reason (or explain themselves on the off-chance that they have a good reason for what they want) usually are.



  • @PJH said:

    What about the female clients?

    Already covered.

    You must have fun with the ! operator using those parsing rules.


  • Discourse touched me in a no-no place


  • Banned

    Generate filename based on clock. If file exists, add one minute until you find one that doesn't. Each minute with double report would advance the file clock 1 minute ahead, but each minute without reports would bring it back.



  • Date your files 15 minutes "late". Each time you need to write a file, try "now - 15m", then "now - 14m", etc until you find an unused filename.

    If 15 minutes is too inaccurate for your client or their system, tell them to fix their shit.



  • Create a class to get the next filename. The class internally stores the last timestamp created as a singleton. When the GetFileName method is called, return the greater of (stored timestamp + one minute) and (current time rounded to minute).

    Time will artificially move foward too fast when called quickly and settle back to actual time when the flurry of activity stops.


  • Discourse touched me in a no-no place

    @Weng said:

    To date, the most common solution is" throttle the job so we do a maximum of 1 per minute ". Which often leads to hilarious processing backlogs which lead to our being yelled at for being terrible programmers and taking hours to generate a few simple reports.

    Create a series of subdirectories, 2[1] through n, where n is "high enough". The first duplicate for any given timestamp goes in 2/, and so on.

    [1] no solution would be complete without a reasonable-sounding but possibly wtfy, or perhaps a possibly wtfy but reasonable-sounding, part.



  • My gut response is to go the 'throttle the jobs to 1 per minute' route, but if a throughput of > 1/min is expected, then start viciously leaning on sales to tell them their shit stinks. Or find a way to tell them their shit stinks directly.

    Filed Under: I was under the impression they hired us for our skills, Apparently those skills do not include knowing why your shit won't work



  • @FrostCat said:

    Create a series of subdirectories, 2[1] through n, where n is "high enough". The first duplicate for any given timestamp goes in 2/, and so on.

    Oh man, that brings back memories...



  • @Weng said:

    our interactions are proxied by sales and account management goons

    Today's particular fucked-up time format is not TRWTF.



  • @Weng said:

    Client wants a report with a filename containing a date time stamp of a format like ErrorReportMMDDYYYYHHmm.txt

    We may, under certain circumstances, produce more than one per minute in bursts. The filename, for obvious reasons, must be unique.

    We need a way to generate unique psuedodates using the current time and date as a seed.

    Generalize to arbitrary date formats and resolution/precision.

    OK. The constraints are stupid, the spec is idiotic. Let's move on.

    The way I'd do that: maintain a count of issued filenames. Post-increment it every time you issue a filename; decrement it on every minute rollover, with a lower limit of zero. When asked for a filename, add the issued count to the current time before formatting it per the template.

    Provided filenames are not in fact issued faster than one per minute on aveage, this gets you filenames that reflect the time of issue as closely as possible, given the naming constraints; it maintains monotonic ordering for filenames; and when (not if) the marketing department's idiotic assumption about maximum filename issue rate turns out to be complete crap, it continues to generate usable filenames without blowing up in your face and eating all your storage.



  • Yeah, this is stupid question... I'm new here.

    Why don't you just overwrite the file?

    (The spec calls for unique file name with a resolution of only one per minute. So overwrite the file, and make it a known issue that the file name specified needs to change if this is not what the client wants.)



  • @Jerome_Viveiros said:

    Why don't you just overwrite the file?

    Because generally I try to avoid giving the Client an excuse to shriek like a banshee at me.

    "It's according to Spec" just gets "You should know better"
    Knowing better gets complaints about not adhering to Spec.



  • WTF problems call for WTF solutions. The solution I'd like to implement is to generate the filename with the current date and time, and if we find the directory already has one with that name, rename the existing filename to the previous minute. If we find there already is one for the previous minute, rename that one. Repeat until you shifted enough files to fit the new file.

    This way, we ensure that every file is created with the correct date and time. Whether they get to keep it is another matter.



  • You aren't the OP, are you? Doesn't matter anyway.

    In a situation like the one described, I'd make the client understand that a unique name is impossible in that format for more than one file per minute. Hacking silly workarounds would not be an option.

    They might be willing to change the format and might even admit that the spec was stupid. Of course, I'd be talking to the client; not posting about it here.



  • @Weng said:

    a date time stamp of a format like ErrorReportMMDDYYYYHHmm.txt

    Just noticed that this is not ErrorReportYYYYMMDDHHmm.txt - what are they smoking?



  • @Jerome_Viveiros said:

    I'd be talking to the client; not posting about it here.

    @Weng said:

    our interactions are proxied by sales and account management goons

    I gather that Weng is working at a place where the programmers don't get to talk to the client. Sales staff are frequently not inclined to listen to why something doesn't work, and just expect you to make it work.


  • Banned

    @Jaime said:

    singleton

    DIE!!!



  • In which case when marketing says “we don't want to bring this up with the customer, we would sound like incompetent” it might make sense to answer “well, you'll look more so if you don't, because the customer is unlikely to be happy with what we come up ourselves” and then proceed to implement a solution that the customer is almost guaranteed not to be happy with (like the renaming the previous report out of place).



  • Easy fix: use the decade as a counter in case the file exists. So if 121820141033 exists, use 121820241033. And if the app is used more than 10 years, make sure you're no longer working there...
    (Of course, you could do the same with centuries).
    </sarcasm>



  • Yeah, the real WTF is these guys using the American date format.



  • @Gaska said:

    >Jaime:
    singleton

    DIE!!!


    Quite so, because uniqifying timestamps across directories or against multiple filename templates is probably the Wrong Thing.

    Perhaps the right place to store Jaime's last-used-timestamps (plural! or, equivalently, my issued-filenames counts) would be as files in the same directories as the required ErrorReportYYYYMMDDHHmm.txt files.

    Or if that's too ugly, don't store anything all and just use a loop that tries to create the new file in noclobber mode, bumping the filename's embedded timestamp while creation fails with a "file exists". This will get increasingly slow once the average file creation rate does creep above one per minute but has the advantages of being very robust and not at all "clever".

    Also, I'd fail to correct that "accidental" misreading of the specified template as YYYYMMDDHHmm. I'm sure the only folks who actually care about that will be the same ones who end up glad when the files sort properly, so they'll never raise an issue ticket to get the "mistake" "fixed".

    Filed under: heavy penalties will be applied for "excessive" use of "scare" quotes



  • @flabdablet said:

    noticed that this is not ErrorReportYYYYMMDDHHmm.txt - what are they smoking?

    All their Christmases come at once


  • Garbage Person

    @flabdablet said:

    Just noticed that this is not ErrorReportYYYYMMDDHHmm.txt - what are they smoking?
    The format string varies client to client. But MMDDYYYY is most common.

    Added wrinkles for you guys to chew on:

    • The file is transmitted to a client-facing FTP immediate after creation
    • The files are logged, archived, and tracked by name, including their contents. That way when the customer goes "WE DIDN'T GET A SHIPPING ACKNOWLEDGEMENT FOR ORDER X YOU ARE OUT OF COMPLIANCE", we can say "It was in ShippingReport121820140824.xml" and they can say "WE NEVER GOT THAT" and we can say "We transmitted it to your FTP at <time> successfully and they can go completely silent on the matter until end of the year scorecarding, when they bring up 'chronic late shipments' with the sales rep and we get our butts chewed. (Hell, we get our butts chewed because manufacturing is actually chronically shipping things late and we aren't, I don't know, falsifying the reports or something).

    Incidentally, the use of FTP/SFTP sites in both direction to transfer data is literally the only thing we do for interchange. No web service calls. Nothing synchrnous. Nothing that offers a quick and easy immediate 'yeah we got it, thanks'. This is not because we don't have the capability built, but because sales has worked with FTP/SFTP for decades and understands it, and is not willing to talk about anything new. Even when it turns out to be what the client asked for first.


  • Discourse touched me in a no-no place

    @trithne said:

    "It's according to Spec" just gets "You should know better"Knowing better gets complaints about not adhering to Spec.

    Sometimes it's OK to point out the Catch-22. "Which of these impossible-to-fulfill-at-the-same-time conditions is least important?"


  • Discourse touched me in a no-no place

    @Weng said:

    we can say "We transmitted it to your FTP at successfully and they can go completely silent on the matter until end of the year scorecarding, when they bring up 'chronic late shipments' with the sales rep and we get our butts chewed.

    And you're not showing them the logs at this point and saying "sorry, yell at someone else" why?


  • Banned

    @flabdablet said:

    Perhaps the right place to store Jaime's last-used-timestamps (plural! or, equivalently, my issued-filenames counts) would be as files in the same directories as the required ErrorReportYYYYMMDDHHmm.txt files.

    Y U NO USE THE FILENAMES THEMSELVES


  • Garbage Person

    Oh, we do. It's astonishing how little they care. It's really just excuse making to browbeat our sales guys into better discounts to "apologize". There can't be any apologies if there aren't any real or imagined problems.


  • Discourse touched me in a no-no place

    Ah, that technique.

    Say, weren't you the one who was complaining about Amazon Prime yesterday? I'm sure it's just a coincidence I'm asking, and not in any way related to the topic.


  • Garbage Person

    Indeed.

    Another popular technique is "you may not retain detailed data for more than X days". And then on day x+15, they ask "what exactly caused order y to do z?"

    At which point, we do not have the data to answer that.



  • @Gaska said:

    Y U NO USE THE FILENAMES THEMSELVES

    Because it's easier to do arithmetic on a number and then format it using a template than to parse a number from a filename, do arithmetic on it and then format it using a template.


  • Banned

    1. n=0
    2. Generate file from current time + n.
    3. Check if file exists.
      True: save file.
      False: n++ and goto 2.

    What's so hard about it?


Log in to reply