Simple job management system
-
So we have a website (PHP, Wordpress-based, outsourced to cheap Indian developers, you know the stuff). The website generates "tasks" that need to be processed by a custom, backend application (.NET, Linux/Windows compatible, developed by cheap Southern-European developers) that needs to pass output back to the frontend somehow while the task is running (not just at the end).
Currently we're using a plain old MySQL database as a queue: frontend writes a row to the table, backend checks for new rows every 5 seconds, backend writes results in rows in other database, simple stuff. This has two obvious disadvantages:
- Both sides need to continuously poll for messages from the other instead of just waiting. We probably do a workaround with HTTP calls or something, but then again the tasks usually take several minutes so an extra 5 seconds won't be a big deal.
- We'll probably want to expand to multiple job servers, and there's lots of hairy details there (dealing with interrupted jobs and crashes, for example).
And it just seems like a solved problem in general. There are many, many products related to job queues, but most are much more complex than I need. So I need your recommendations.
So far I've seen http://gearman.org/ (seems to be exactly what I want, but it's in - gasp - php), https://www.hangfire.io/ (pretty much a .NET equivalent), RabbitMQ ("Message queue" is not what I was directly looking for but I guess people use them for that), and Apache Kafka.
-
@anonymous234 said in Simple job management system:
Both sides need to continuously poll for messages from the other instead of just waiting.
Well, I mean you could write up something that does the polling for you and hides it away if that's such a big deal.
I could probably do something pretty quick in an hour using websockets and EF (or whatever POCO database connector).
The gearman link seems to indicate it's written in C, their sample client connector thing is what's in PHP. Apparently there are many clients available.
