Am I :doing_it_wrong:? Scheduled and event driven notifications


  • kills Dumbledore

    Problem domain:

    I have multiple clients (a few thousand, but I think many are inactive) who will be able to sign up for email notifications about new documentation.

    Each client can choose how often to be notified: daily, weekly, monthly, or whenever there is new documentation for them. This could be multiple documents per day. Each document is for a specific client.

    There will be a web service which is called by external systems each time a document is added, and will add the document details in to a database.

    My thinking on the client notification side is as follows:
    A service or some other long running process will be running. On startup it sets up a thread for each client which waits either for the delay period or for an event to be fired. The event will be signalled from the web service when a new document is added. When the timeout or event is fired, it will check the settings, update the notification frequency if changed, and go through the loop again.

    I set this up as a Windows service at first, but getting the web service to trigger the event was getting tricky. This would be the first Windows service I've written, and debugging and installing also look pretty complicated.
    Second attempt was a WCF service, but persistence is a bugger there. I found something to make the service act as a singleton, but when I try to run with that enabled, I put VS into a complete panic that's only recoverable by deleting the bin folder.

    So, a couple of questions: Is one thread per client a design that's going to bite me in the arse once I scale up from the 1 or 2 I'm currently testing with, and is one of the service solutions I outlined the right way to go or should I rethink my entire approach?


  • BINNED

    @Jaloopa said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    trigger the event

    Should this be event based? I mean can you come by by making the service check every x minutes that there is something to do? It's mail ... it's not a direct medium.


  • kills Dumbledore

    @Luhmann said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    can you come by by making the service check every x minutes that there is something to do?

    That would certainly simplify things. Wait some amount of time, send an email for each client who's past their notification period and has some new documents.

    I may need to remove these complicator's gloves


  • FoxDev

    @Jaloopa If you're setting it up as a web service (and you're hosting in IIS or similar), you shouldn't need to worry about threads, as IIS will just take one from its pool when it needs one.



  • @Jaloopa Sounds over complicated.

    Is the IPC really needed? How quickly are people expecting these updates? Couldn't you just query the database every half-hour to find out which articles and subscriptions have been added/changed? You actually don't want to be too timely, as you'll send out notifications for articles that got published accidentally and are private or deleted by the time the subscriber reads their email. (Twitter has that problem-- since the only way to edit a tweet is to delete and re-write it, people get email notifications for deleted tweets all the time.) I know developers love to make fancy real-time stuff, but it's not necessary 99.9% of the time.

    Threads are only needed for concurrent activities. Unless sending an email takes like 99% CPU power for like 3 full minutes, there's no need to put that in its own thread. Just have a single "scheduler" thread which sleeps until min( time_of_next_scheduled_event, polling_time ). (Even if you do need to spawn a thread to send an email, you can spawn that thread from this "scheduler" thread.) Another advantage here is if a schedule changes (someone sets it from weekly to daily), you don't have to track down the thread responsible and wake it up, you just need to add the new schedule to whatever Collection class you're storing it in, and the change'll become active the next time your "scheduler" thread wakes up for any reason.

    Your solution of one thread per subscriber isn't bad, per se, but certainly wasteful of computer resources (tracking threads, even sleeping ones, takes memory), and probably not even much savings in code to write (once you account for all the edge-cases, like someone making their schedule more frequent).

    Or an alternative: since you're using Windows, you also have this incredibly powerful tool called Scheduled Tasks. Why have a Service at all? Just create a simple CLI tool to send the update, and have the OS take care of the scheduling. Instead of IPC, your web server can simply create the Scheduled Task when the user requests it. (The Service might be useful if you plan to later port to Linux, because cron suuucks. But Windows Scheduled Tasks is really good.)


  • Trolleybus Mechanic

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    Well, it isn't as compli... wait

    @blakeyrat said

    @blakeyrat said

    @blakeyrat said

    @blakeyrat said

    @blakeyrat said

    Oh shit-- @boomzilla you forgot to switch accounts before posting again.


  • kills Dumbledore

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    The Service might be useful if you plan to later port to Linux, because cron suuucks

    No Linux ports if I have anything to do with it. We're all Microsoft stack here.

    I hadn't even considered Scheduled Tasks. That makes debugging a lot simpler too.

    I knew this was an overcomplicated design as soon as I started getting into the nitty gritty, just had the basic design already in my head and it can be hard to think your way out of that without someone else pointing out what an idiot you're being


  • kills Dumbledore

    @Lorne-Kates Yeah, I was the one to bring him out of retirement. I AM THE KING!

    Good to hear from you, @blakeyrat


  • Trolleybus Mechanic

    @Jaloopa said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    @Lorne-Kates Yeah, I was the one to bring him out of retirement. I AM THE KING!



  • @Jaloopa said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    The only risk of Scheduled Tasks approach is if you have a million of these all scheduled at the same time, none of the tasks have knowledge of any of the others. That means they could be simultaneously trying to send a million emails all at once and they all time-out as a result.

    Odds are you'd never scale that big, but might be worth benchmarking when you're testing this solution just in case.


  • kills Dumbledore

    @blakeyrat I'm now thinking one scheduled task to loop through and send an email for each client that needs it. It will all be plain text emails with no attachments, so shouldn't be a massive drain.



  • BTW if you want to go a little outside of Microsoft's recommendations, you can create a worker thread inside your WebRequest that updates the article to notify all the people who are subscribed to it.

    Something like this horrible psuedo-code:

    HandleArticleUpdate()
    {
     update_article();
     notify = new NotifyThread( article_id );
     notify.Run();
    }
    
    class NotifyThread() : Thread
    {
     Run()
     {
      sleep_five_minutes();
      check_if_article_still_public();
      send_notification_to_subscribers();
     }
    }
    

    This works, but Microsoft doesn't recommend it because threads created this way don't end up inside the ASP.net threadpool and thus could slow the performance of IIS if your server gets really busy. It also means that some notifications might get lost if the web server is killed (which, hopefully, shouldn't ever happen but... be aware of the trade-offs.)

    You can then use Scheduled Tasks for the daily and weekly updates, and something like the above psuedo-code for the "real-time" updates.

    EDIT: you could also easily get double-notifications if you're not super careful, so maybe this is a terrible idea. It's been awhile since I've been designing software, sorry.



  • @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    It also means that some notifications might get lost if the web server is killed (which, hopefully, shouldn't ever happen but... be aware of the trade-offs.)

    You can probably spin off a separate process if that's an issue. Not sure about perf, security, and general ickyness implications of that, though.



  • @Maciejasjmj You could perhaps recycle the same quick and dirty CLI you're using to power the daily and weekly updates and spawn one of those from the WebRequest, that's true. I didn't think of that.


  • BINNED

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    Scheduled Tasks. Why have a Service at all?

    I guess it is possible but I simply have not seen it done properly with Scheduled Tasks. A service eventually forces proper start, stop and logging/troubleshooting stuff. You can have a service without it but everybody supporting it will hate you. As a Dev you can have a GUI based application turned into a Scheduled Task and never look back. No clean stop, no error or troubleshooting steps necessary but my will the service department hate your guts. And they are damn right.


  • BINNED

    @Jaloopa said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    I hadn't even considered Scheduled Tasks. That makes debugging a lot simpler too.

    Initial debugging might be easier but long term serviceability ...

    Did I tell you I hate all these Scheduled Tasks that should be done in proper services?

    A Service can be remotely monitored and corrected by every server management tool available for Windows. A Scheduled Task is not easily monitored. If your task is hanging you might be able to detect it. You might be able to stop or rather kill it.

    BTW a proper service for me implies that it will always stop in a proper state and can be started again on boot without interaction. It doesn't hang it crashes, hard and fast. But that's ok because you just kick it again and it starts working again. It has troubleshooting tools and logging so you can later find out why it failed if necessary.



  • @Luhmann said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    A Scheduled Task is not easily monitored.

    Well. I'd like to write a better reply but NodeBB's composer is completely broken right now so I can't quote you.

    Scheduled tasks do indeed have settings to kill and optionally reboot the task if it locks-up or takes over X amount of time. So that argument's out the window.

    The Scheduled Task feature in Windows is build to do EXACTLY THIS THING WE'RE DOING, so not using it in this case would be the WTF.

    Your unnamed monitoring tools might not work correctly with Scheduled Tasks, but that's not the software developer's problem-- that means you don't have very good monitoring tools. (Not that there's much to monitor at all, really. Schedule Tasks will already put an event in the event log if a task fails, and surely your monitoring tool can read that, right?)

    You bring up a good point that he should talk to whoever maintains the servers before settling on a solution, but. I still see no reason to disqualify Scheduled Tasks.


  • BINNED

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    Scheduled tasks do indeed have settings to kill and optionally reboot the task if it locks-up or takes over X amount of time. So that argument's out the window.

    And I haven't seen that working correctly. Sure it might just be the shitty apps I see running as scheduled tasks. Yet I see locked up applications that are run through scheduled tasks daily.

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    Your unnamed monitoring tools

    I didn't name any monitoring tools because it's a general statement. I'm not defending the use of Scheduled Tasks so I don't really feel compelled to provide evidence that they can be monitored as easily as services can. Yes if there result ends up in event log so that something. But as pointed out above I'm skeptical about the ability of scheduled tasks to detected and correct. Let alone that server monitoring tools will allow to control (stop/start) services, this is even a basic Windows feature. To pull this trick with scheduled tasks it requires some additional tooling.

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    but that's not the software developer's problem

    Sure it is. If you choose a certain solution but don't consider the service impact you're a shitty developer.

    So

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    talk to whoever maintains the servers before settling on a solution

    Alright then!

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    I still see no reason to disqualify Scheduled Tasks.

    Oh, they still have use cases. I'm just very weary if someone suggests a solution based on Scheduled Tasks.

    Scheduled tasks are often used as a stop gap solution to leverage something from an interactive application to something that runs unattended. Except that it was made to run attended. That means errors that pop-up blocking shit. The application not properly cleaning up after an error or after a crash. Sure you can have that shit with a service as well but with a service that kind of problems allow me to just kick it back to dev and make the responsible fix his shit. Because scheduled tasks can be fixed manually by human interaction that is most of the time the option that is choose above actually fixing the issues below.



  • @Luhmann said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    Let alone that server monitoring tools will allow to control (stop/start) services, this is even a basic Windows feature.

    Well you can restart the Task Scheduler service in Windows, if you have permissions, and if you think it'll do any good. Scheduled Tasks is implemented as a Service, so I guess that'll make you happy.

    @Luhmann said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    Scheduled tasks are often used as a stop gap solution to leverage something from an interactive application to something that runs unattended.

    Ok, but nobody's suggesting doing that in this thread.

    It sounds like you've gotten burned once and have forever bucketed the feature into the "reject" pile. Well fine. That's your opinion. I don't share it, neither do any Windows developers or system administrators I know.


  • BINNED

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    Well you can restart the Task Scheduler service in Windows

    I'm doubt that will have any influence on the tasks that are running as separate processes.

    @blakeyrat said in Am I :doing_it_wrong:? Scheduled and event driven notifications:

    "reject" pile

    It's more a red flag. I didn't claim it can't be done properly. It is the same like someone offering to implement something in NoSQL or in the Cloud without proper arguments. Sure there are right ways to do it but there are a plenty ways to do it wrong too.



  • Scheduled task feels dirty. Lika a temp hack until you get the real thing going.

    I've personally had best luck with a service app that has an internal scheduler. I'm sure there are. Net libraries around that can act as crons. Even if not, sleep interval should work nicely in your case.

    Shove that into a windows service, give it a little api to receive events (OWIN or something) and you're good to go.


Log in to reply