Long running threads in ASP.Net (website)
-
I've dabbled in threads before, and more or less know how they work. But I'm not sure about long-running (ie: 24 hours a day) threads.
So basically, I have a thread that:
- Starts a loop
- does something
- Waits for a WaitHandle signal (or does an async wait, depending on configuration)
- When it gets that signal, goes to top of loop.
On AppStart, the website kicks off the thread, and the thread runs happily in the background.
Am I going to get my ass kicked by "you shoulda known" .Net gotcha? Like, every 4 hours when the app_pool recycles, my thread dies until I had a handler restart it? Or threads running long that 8 hours set a server on fire? Anything like that?
-
@lorne-kates A thread is essentially just another callstack running alongside your main one, within the confines of your process. It will die the same way your main callstack will: if there's an unhandled exception, or if normal control flow exits its main function. There aren't "hidden shoulda knowns" that will specifically terminate a thread. (Any stuff that will terminate your entire process still applies, of course.)
What you need to worry about with threads running is race conditions. Because the thread runs concurrent with your main code, with access to the same memory, it's possible that your two threads (main and background worker) can try to use the same objects at the same time, causing inconsistencies that can be very hard to debug. The best way to handle this is to make sure that your worker thread never touches anything that the main thread can also touch, but if you do this 100%, there's really very little reason to have a thread at all rather than a separate process. If you want the threads to communicate with each other, it's best to use a threadsafe queue. Failing that, use the
lockstatement to protect access to shared data. (Which can come with pitfalls of its own, natch.) But generally, when it comes to sharing state between threads, "the only winning move is not to play."
-
@lorne-kates I believe the advice is not to run long-running threads in ASP.Net but instead to ask a Service (or another type of application at least) to run them on your behalf.
That said I've done short-running threads for an app that needed to respond to the client post-haste, but still do a bit of extra work once the response was sent. The threads lived long enough to perform a DB insert, with retries a maximum lifetime of 1 minute, and it wasn't a huge deal and didn't seem to affect performance.
Like you mention, though, since the AppPool doesn't know about your custom threads, when it recycles it'll just dump them on the floor (or perhaps lock-up entirely, I don't remember which). If the work the threads are doing is critical, you probably want to avoid that. (In my case, it was an analytics application where AppPool recycles happened maybe once a week and dropping a dozen or so events wasn't a big deal, and the threads couldn't stall for longer than a minute anyway.)
Assuming your thread can be cancelled and resumed, it should be fine. Just cancel it when the AppPool is recycling and start a new one when the first post-recycle page load happens.
-
@masonwheeler said in Long running threads in ASP.Net (website):
There aren't "hidden shoulda knowns" that will specifically terminate a thread. (Any stuff that will terminate your entire process still applies, of course.)
And an app pool recycle like @Lorne-Kates mentioned (or the app pool hitting its idle timeout, if that's not set to 0) terminates the process.
For our long-running threads, we have a dedicated Windows service to own/manage the thread, rather than trying to have ASP.NET do that without dropping it.
-
Long time ago, when I was in NET land, I remember trying to do this exact thing.
It bit me in the ass. The code was flaky and difficult to debug. I ended up moving my background processing into a service and having ASP.NET talk with it through WCF.
I don't remember the details. but I think in my case, I actually tried to spawn and control threads from web requests. If you only start one thread on start and don't touch it again, you might fare better. But you'd probably still be better off keeping background processing away from your web server and in an environment that is designed to handle it.
-
Echoing what has been posted before.... If I have material that must persist longer than he request/response (or worse case "session" - I love stateless, so minimize sessions), then I will invariably host it outside of IIS (as others said, most commonly as a Windows Service)
-
@lorne-kates in addition to what others have said: When your app pool recycles you might have two instances running, as iis gives the "dying" process some time to exit, while the next process starts up for new request. So then you could have cross-process synchronisation to worry about as well.
-
@unperverted-vixen said in Long running threads in ASP.Net (website):
For our long-running threads, we have a dedicated Windows service to own/manage the thread, rather than trying to have ASP.NET do that without dropping it.
This is also how we do it.
-
@blakeyrat Yeah, pretty much the same thing. Gathers up analytics data and dumps it in a queue (request, referer, page being looked at, other shit like that). The thread at it's leisure will write that down to the database. Another thread, at it's leisure, picks up a chunk of records from the database, sends it to a remote system, then marks it as "sent".
@everyone_else I was originally thinking a Windows Service. The design I got was originally "how about make a hidden webpage, and use SQL Job Agent to use a CLR to call that page on a timer". The compromise was a threaded solution-- but I'm worried it'll be flaky.
I'll pitch moving it into a service. We do have other stuff running as a service, so I don't see why this can't be. There is some shared code with the website, but nothing that can't be referenced in an .dll.
The question is now, what is the recommended way of doing interprocess communication between the website and the service? Right now, the website gathers the data, then dumps it into a (static but thread-safe) queue (basically a List<Of OurDumbClass>). The most obvious seems to be "have the website write shit into the db and then have the service pick it up". But the whole point of threading it out was so the web response doesn't get stalled/delayed waiting on a db write that has nothing to do with the user's request.
-
@lorne-kates said in Long running threads in ASP.Net (website):
The question is now, what is the recommended way of doing interprocess communication between the website and the service? Right now, the website gathers the data, then dumps it into a (static but thread-safe) queue (basically a List<Of OurDumbClass>). The most obvious seems to be "have the website write shit into the db and then have the service pick it up". But the whole point of threading it out was so the web response doesn't get stalled/delayed waiting on a db write that has nothing to do with the user's request.
Our sites still communicate to the offloading service via .NET Remoting. For obvious reasons, I wouldn't recommend that approach at this point. :face_with_stuck-out_tongue:
I know at least one person above mentioned WCF, but I'm pretty sure that's deprecated too. I think at this point the recommended approach is to have the service self-host ASP.NET Web API, and use that for the transfer.
-
@lorne-kates said in Long running threads in ASP.Net (website):
Yeah, pretty much the same thing. Gathers up analytics data and dumps it in a queue (request, referer, page being looked at, other shit like that).
And the long-running thread is the queue?
Yeah I wouldn't do it that way. My solution used it because we didn't do jack with the data other than store it... so we did basic verification stuff (looked for the Do Not Track header, the opt-out cookie, send the W3C privacy policy header, etc.), spawned the worker thread (with the raw URL/IP data), then returned the 200 response. The worker thread then did some very basic parsing and stored it in the database. It was short-lived enough that the thing worked.
I have two suggestions for you:
-
If the server isn't virtual, write a Windows Service that looks at the web server logs and inserts data from that. (Make your web server super super simple.) This works with a physical server or a cloud machine that pretends to be a physical server, but not as well from something like a Azure Web Application, since the "web log" is a big cloud blob and it's hard to do random seeks into a big cloud blob.
-
Move the entire queue into a Service, and use some simple IPC to get your data there.
@lorne-kates said in Long running threads in ASP.Net (website):
The question is now, what is the recommended way of doing interprocess communication between the website and the service?
I'd probably just use TCP/IP to localhost over a known port. More than that's probably overkill. You'd still have to serialize the data though.
-
-
@lorne-kates said in Long running threads in ASP.Net (website):
I've dabbled in threads before, and more or less know how they work. But I'm not sure about long-running (ie: 24 hours a day) threads.
So basically, I have a thread that:
- Starts a loop
- does something
- Waits for a WaitHandle signal (or does an async wait, depending on configuration)
- When it gets that signal, goes to top of loop.
On AppStart, the website kicks off the thread, and the thread runs happily in the background.
Am I going to get my ass kicked by "you shoulda known" .Net gotcha? Like, every 4 hours when the app_pool recycles, my thread dies until I had a handler restart it? Or threads running long that 8 hours set a server on fire? Anything like that?
I would just move the long running code to a Windows Service solely because the possibility of "application pool recycling".
-
@unperverted-vixen said in Long running threads in ASP.Net (website):
@lorne-kates said in Long running threads in ASP.Net (website):
The question is now, what is the recommended way of doing interprocess communication between the website and the service? Right now, the website gathers the data, then dumps it into a (static but thread-safe) queue (basically a List<Of OurDumbClass>). The most obvious seems to be "have the website write shit into the db and then have the service pick it up". But the whole point of threading it out was so the web response doesn't get stalled/delayed waiting on a db write that has nothing to do with the user's request.
Our sites still communicate to the offloading service via .NET Remoting. For obvious reasons, I wouldn't recommend that approach at this point. :face_with_stuck-out_tongue:
I know at least one person above mentioned WCF, but I'm pretty sure that's deprecated too. I think at this point the recommended approach is to have the service self-host ASP.NET Web API, and use that for the transfer.
Or since you're using database anyway, dump the parameters (like index of table your "OurDumbClass" is referring to) to a parameter table, and just send a custom command to your service, and have it lookup whatever it needs from table.
-
As far as technology for communication to the Service - IMPO: It very much depends, I would never give a general "you should use XXX" type answer.
That being said, I am a bug fan of very thin layers (server and client side - a matched pair) that deal with just the communication aspect [ZERO "logic"]. This has saved my bacon as scenarios change and different technologies become more/less appropriate for the communication. Imagine as a hypothetical that you later want to move to Azure and use AppFabric - having a design that allows you to create/test/deploy a pair of DLL's without impacting the reminder of the ecosystem could be a make/break condition.
-
@lorne-kates said in Long running threads in ASP.Net (website):
The most obvious seems to be "have the website write shit into the db and then have the service pick it up". But the whole point of threading it out was so the web response doesn't get stalled/delayed waiting on a db write that has nothing to do with the user's request.
Have the website spin off a new thread on each request that does that DB writing. It won't block. It might occasionally fail, but it's analytics so not mission crticial.
Alternatively, you could use MSMQ or similar. That handles the queueing AND the IPC.
And in theory, you can (I am informed, though I have never actually seen it done) have MSMQ feed into an IIS-hosted asp.net application, instead of actually calling over HTTP. Fuck if I know how THAT works, though.
-
@thecpuwizard said in Long running threads in ASP.Net (website):
That being said, I am a bug fan of very thin layers
-
@weng said in Long running threads in ASP.Net (website):
Have the website spin off a new thread on each request that does that DB writing.
I was thinking of that, but there was concern that if the website gets hammered, all those threads would kak the server.
MSMQ
Funnily enough-- we have MSMQ as part of our architecture, but we're in the process of getting rid of it. Mainly because Microsoft is slowly moving the technology we use to be entirely cloud/365 based. So it soon won't be feasible to use MSMQ from Azure back to the customer's home office.
-
@lorne-kates said in Long running threads in ASP.Net (website):
@thecpuwizard said in Long running threads in ASP.Net (website):
That being said, I am a bug fan of very thin layers
Not sure why you reacted with a "wtf" to my opinion that very thin layers (veneers) are helpful in situations like this.....
-
@thecpuwizard said in Long running threads in ASP.Net (website):
@lorne-kates said in Long running threads in ASP.Net (website):
@thecpuwizard said in Long running threads in ASP.Net (website):
That being said, I am a bug fan of very thin layers
Not sure why you reacted with a "wtf" to my opinion that very thin layers (veneers) are helpful in situations like this.....
I think it was the bug part...?
-
@tsaukpaetra said in Long running threads in ASP.Net (website):
@thecpuwizard said in Long running threads in ASP.Net (website):
@lorne-kates said in Long running threads in ASP.Net (website):
@thecpuwizard said in Long running threads in ASP.Net (website):
That being said, I am a bug fan of very thin layers
Not sure why you reacted with a "wtf" to my opinion that very thin layers (veneers) are helpful in situations like this.....
I think it was the bug part...?
I didn't see the typo.... Need a new pair of glasses, seems the screen on the machine is shrinking ;)
-
@thecpuwizard said in Long running threads in ASP.Net (website):
@tsaukpaetra said in Long running threads in ASP.Net (website):
@thecpuwizard said in Long running threads in ASP.Net (website):
@lorne-kates said in Long running threads in ASP.Net (website):
@thecpuwizard said in Long running threads in ASP.Net (website):
That being said, I am a bug fan of very thin layers
Not sure why you reacted with a "wtf" to my opinion that very thin layers (veneers) are helpful in situations like this.....
I think it was the bug part...?
I didn't see the typo.... Need a new pair of glasses, seems the screen on the machine is shrinking ;)
Nah, you're just getting bugger...
-
