How should I handle this polling load?
-
Previously, I rolled a web application (in the JS/SPA style with a PHP server backend) as kind of an extension to our company's billing system, so that we could pull client authentication into the billing system via RADIUS (we're a WISP). Now, the task on my plate is to add a "polling" module that will retrieve signal & connected tower information from the client equipment via SNMP, check the signal on the tower side (also via SNMP), and ping the client to check for jitter/packet loss. I've got a mostly-working way to have the application tell the server "poll client X", and then the PHP backend goes off and gathers the information I need on a one-off basis... but the requirement is also to periodically poll all 7,000 of our customers, to build up baseline information over time. Given that a single client SNMP/Ping qualification process is going to take 30-60 seconds (based on the ping testing we want to do, mostly), and the periodic poll interval being tossed around is 6 hours, I'm going to have to find some way to do the polling in parallel.
I know I'm kind of in WTF territory re-inventing Cacti/Nagios here, but there are Reasons to want to tie this straight into our SQL backend rather than mucking around with middleware RRD files (not to mention the difficulty of keeping dynamic IPs straight within a Cacti/Nagios type solution). I've tried to understand the PHP Threads extension documentation, but it's way over my head, and my Google-fu isn't coming up with anything that helps.
So, at the risk of getting mocked, I'm gonna go from longtime lurker to first time poster and ask... anyone have any pointers to make PHP Threading work for this task? Or suggestions as to a better way to (1) get SNMP data from a device by IP address with a common community, (2) make an SQL lookup based on that data to find the address of the second device to SNMP query, (3) then ping the first device 100 times in 20 seconds and record min/avg/max/mdev for that ping test? I'm not dead-set on PHP or anything, but have basically no programming experience in anything other than PHP & JS, so idiot's-guide-level pointers would be appreciated if another solution is better. I'm certainly willing to do the legwork to learn another toolset, I just have no idea where to start looking.
-
The poll interval is 6 hours? The execution time exceeds the timeout on most web requests? Wouldn't it be better to just use a cron job and pick the results up with the PHP script?
-
If I were solving this problem in .NET (shoot me if you must) I would write a system service that has several threads doing the polling, as .NET has a bunch of stuff making multi-threaded easy for you. Writing the results directly to the DB and letting your PHP site stick to being about displaying the information rather than gathering it.
As you probably don't have $1400 to sink on an MSDN license (required for making services, or at least, VS professional, you could get it for about $40 a month if you rather as VS online), you can probably do the exact same thing in Java or something.
-
Erm, I guess I underspecified.
Two different types of polling should happen:
- On demand, "poll Client X right now" type polling that is initiated via the web application
- Periodic, once every 6 hours, "poll all clients" type polling that is initiated via cron (or probably a Scheduled Task, if the Web Server which is IIS/Windows is what triggers it)
I'm just trying to conceptualize a way to let 2) basically just spawn off 7,000 copies of 1) and still get it done in a reasonable amount of time without eating my server's resources alive. But I'm a little leery of just doing 7,000 individual polls in sequence because that's 1-2 hours of polling (mostly waiting around for the pings or SNMP return, especially from clients with shaky connections) and I'm not at all sure management won't change the interval on me later.
I guess one other thing I forgot, but is kind of pertinent... It would make sense from a "not bombing my access points with SNMP requests" standpoint to build the mass polling process in such a way that it SNMP'd the clients, then kind of built a table of Client A - AP A, Client B - AP A, Client C - AP B, and so forth so it could just hit each AP once and then parse out the per-client information from each AP as needed. I'm not sure this is a "deal breaking" requirement, though, as SNMP is pretty lightweight and I can specifically ask for just one OID per client (and not have to walk the whole tree for each time I'm trying to retrieve client data from the AP), so just hitting each AP 10-50 times probably wouldn't kill things.
-
We actually do have a MSDN license via our partner program silver competency, so I could probably see that working. Any pointers on a good starting point for ".NET system services" for dummies for a poor PHP convert like me?
-
2) Periodic, once every 6 hours, "poll all clients" type polling that is initiated via cron (or probably a Scheduled Task, if the Web Server which is IIS/Windows is what triggers it)
This requirement may not be entirely accurate, or the best interpretation of the problem. If I were you I would see about reconceptualising it to "all data is refreshed no less than once every 6 hours", that will let you spread the load out over that 6 hour period and queue up the data in a consistent way, adding threads and increasing the rate (and hence server load) as needed programmatically if you detect any items not refreshed within X% of the time limit, and vice versa if you are getting too much work done too fast (unnecessarily overworking your servers)
write it in such a way that 6 isnt a magic number (should be pretty easy to do, the time limit WILL change, probably getting longer when you need to poll 70,0000 instead)
This certainly sounds like something you CAN write in a php client/server application, but it also sounds like something that you REALLY shouldn't. The single item poll side of things makes sense as it's triggered from the PHP client, but I would probably want to think about having that call a web-service method somewhere else, allowing you to separate off that and any other slow/intensive tasks to different server(s) if you need to scale up.
polling large amounts of things like this really seems like something that should be done elsewhere, away from your site, with the data plugged into a shared database for the client to consume.
-
We actually do have a MSDN license via our partner program silver competency, so I could probably see that working. Any pointers on a good starting point for ".NET system services" for dummies for a poor PHP convert like me?
if you have customers paying to use any element of this, or the data it produces, or anything like that, you aren't licensed to make it with licenses granted from the Microsoft Partner Network. Applications used exclusively internally (i.e. to help other areas of your business deliver stuff for customers) are fine though, so bear that limitation in mind.
Tutorial: Create a Windows service app - .NET Framework
In this tutorial, create a Windows service app in Visual Studio that writes messages to an event log. Add features, set status, add installers, and more.
That's a walkthrough from microsoft, it should help. Although googling for "Write a windows service c# .net" will get you a plethora of guidance.
also http://msdn.microsoft.com/en-us/library/cc221403(v=vs.95).aspx the background worker is a reasonably simple to use wrapper around threads allowing you to just define a task, pass it to the worker and let it go off and do it, letting your main service know when it's finished. It's what I would probably use, but that's because I've never used the concurrency framework (or at least, the concurrency framework proper)
Learn about the Task-based Asynchronous Pattern (TAP), and compare it to the legacy patterns: Asynchronous Programming Model (APM) and Event-based Asynchronous Pattern (EAP).
here is the concurrency framework, it seems complicated, although not as bad as direct thread management. possibly overkill, particularly for a newbie.
Hopefully this is enough information.
one last thing:
DONT USE VB
-
What @algorythmics said.
The poll-1 can be written in PHP but to be run using php/cli (it's a scripting language after all).
For the periodic poll, (Thread) Pool mechanism, where each task could be the same as poll-1.Disclaimer: I understand enough of PHP to NOT recommend doing it in PHP. My choice would be Java + SNMP4J.
-
(we're a WISP)
Well the solution is obviously to find the nearby Wisp Mother and kill her first, you'll just prolong the fight if you go after the Wisps first.
-
f I were solving this problem in .NET (shoot me if you must) I would write a system service that has several threads doing the polling, as .NET has a bunch of stuff making multi-threaded easy for you. Writing the results directly to the DB and letting your PHP site stick to being about displaying the information rather than gathering it.
If you wanna stay all open source-y, you could implement this in Ruby or Python as well without much trouble. You don't need "real" threading, you need "send a request and wait for a response" threading, and even the fake stupid idiot threading support in Ruby is good enough for that use-case.
PHP for a task like this is a non-starter. Sorry.
-
PHP for a task like this is a non-starter. Sorry.
Well, at least I've got confirmation that I'm not a total slobbering idiot -- I got to the point where I said "I'm pretty sure i'm trying to use a hammer to drill a hole here"
-
Well you could do "threading" in PHP by just running a shitload of PHP scripts simultaneously, but that's stupid. Like, write your PHP so you can execute it on the CLI with like "php DoWispShit.php servername" and then kick it off from a shell script or Windows Scheduled Task or PowerShell or VBScript or something. (I don't know what OS we're talking here.)
Consider this a chance to grow your toolbox.
-
I like how over the course of the thread you have populated your profile slightly more each post (people joining the thread late wont see it, but I DID! because I am a cool cat who was HERE IN THE HEART OF THE ACTION! unlike you spoons!)
-
DONT USE VB
But you could use VB .NET if you wanted to. (I'm a c# boy, but be fair.)
-
But you could use VB .NET if you wanted to. (I'm a c# boy, but be fair.)
VB .NET exists, but it's what I meant when I said "DONT USE VB".
VB works really hard to encourage you to write and structure things badly. Easily? Sure, for simple tasks, but invariably badly.
It tries too hard to be like english, and clouds your code as a result when working with anything slightly complex.
Also, in many areas (mine included) putting VB.Net on your CV makes you unemployable.
It's like writing a quick demo app all in one form in WinForms, yeah it's a lot easier in the short term, you just better hope to hell the short term is all this is going to be used for. You don't want to end up working in 18 months time on a 100,000 line class called Form1. VB is like that but with the entire language.
-
Oh come on. VB.net isn't that bad at all. You're crazy. You really think VB.net is more harmful to a resume than fucking PHP? I'd hire the VB.net guy in a millisecond.
-
Except VB .NET still compiles to the same end result as C#, and supports threading - and this is probably the simplest use case for threading in existence. There really is no possibilities for race conditions, you just want a fire and forget process that essentially is a glorified timer.
Recommending against any language that accomplishes the goal is TRWTF, there's no reason to hate on VB in this case.
-
@blakeyrat said:
PHP for a task like this is a non-starter. Sorry.
Well, at least I've got confirmation that I'm not a total slobbering idiot -- I got to the point where I said "I'm pretty sure i'm trying to use a hammer to drill a hole here"
Err.. not that fast. Remember we're dealing with @blakeyrat here, so everything he says is to be taken with a grain of salt (specially when open-sourcey and linuxy things are mentioned).
I've done a monitoring dashboard for keeping track of online/offline status for a bunch of network devices, using PHP and SNMP. But it was only retrieving sysDescr (standard SNMP oid) each 5 minutes for less than 100 devices, and storing offline/sysDescr in a DB. The script run in average for less than 1 minute.
For this situation you don't have hammers. You only have odd-shaped drills and your task is to use one of them without drilling your foot.
Whatever the language, don't do it in the webserver process. If possible, as mentioned before, in another machine.
-
Well you could do "threading" in PHP by just running a shitload of PHP scripts simultaneously, but that's stupid.
That would be stupid, as is the assumption PHP doesn't have threading capabilities.
-
Oh come on. VB.net isn't that bad at all. You're crazy. You really think VB.net is more harmful to a resume than fucking PHP? I'd hire the VB.net guy in a millisecond.
Except VB .NET still compiles to the same end result as C#, and supports threading - and this is probably the simplest use case for threading in existence. There really is no possibilities for race conditions, you just want a fire and forget process that essentially is a glorified timer.
Recommending against any language that accomplishes the goal is TRWTF, there's no reason to hate on VB in this case.
All true, but the point I made isn't about what it compiles to, it's about what it encourages you to do. Webforms and MVC both use the same programming languages, but one encourages you to seperate display from logic very definitely, while the other encourages you to kill yourself (quoting another user for that last bit, but I found it pretty miserable to work with myself).
If I was picking a new language to work with for a problem, I would probably want to avoid the one that has a stigma attached, even if some/all of that stigma is imagined (and I agree, I think at least most of it is). I think C# has most support for encouraging good practice, and is the more transferable of the 2 languages to learn.
-
All true, but the point I made isn't about what it compiles to, it's about what it encourages you to do. Webforms and MVC both use the same programming languages, but one encourages you to seperate display from logic very definitely, while the other encourages you to kill yourself (quoting another user for that last bit, but I found it pretty miserable to work with myself).
WebForms does encourage you to separate display from logic, it just doesn't force you to. WebForms was designed to make quick-and-dirty internal DB-backed apps for businesses-- think Access apps, but on the web. For that purpose, WebForms does an excellent job.
If you're using it to make, say, Twitter-- well, it's not WebForms' fault you picked the wrong tool for the job.
If I was picking a new language to work with for a problem, I would probably want to avoid the one that has a stigma attached, even if some/all of that stigma is imagined (and I agree, I think at least most of it is). I think C# has most support for encouraging good practice, and is the more transferable of the 2 languages to learn.
There's not much stigma attached to VB.net, though. That's what's throwing me. You're the only one I've ever heard with such a strong opinion on it. I think you have VB.net confused with VB6, that's the only explanation.
-
All depends on the goal, the OP appears to have a simple use case that is meant as a simple worker, and there probably* is no GUI interface related to this tool (related to the process worker) - it doesn't really matter what platform or toolset you use as long as it gets the job done.
I also don't see anything about the OP caring about putting this on his resume, just a question about how to poll a client every x period of time / have a client poll him.
-
He could also be confusing it with VBA (The access app - his benefit, not yours.) stigma. That shits still real, but still required in real normal business.
It just sucks to have to deal with it.
-
That would be stupid, as is the assumption PHP doesn't have threading capabilities.http://php.net/manual/en/book.pthreads.php
I have no idea what you're implying, but I already knew PHP had threads. Duh? PHP is a kitchen sink, it has fucking everything. It's all awful, but it's there.
That doesn't mean it's a good fit for this task. If you disagree with that, maybe... present an argument? Just a suggestion.
-
I don't disagree.
I understand enough of PHP to NOT recommend doing it in PHP. My choice would be Java + SNMP4J.
--edit: If you knew PHP has threads, why:
@blakeyrat said:Well you could do "threading" in PHP by just running a shitload of PHP scripts simultaneously, but that's stupid. Like, write your PHP so you can execute it on the CLI with like "php DoWispShit.php servername" and then kick it off from a shell script or Windows Scheduled Task or PowerShell or VBScript or something. (I don't know what OS we're talking here.)
-
I don't see how that quote you posted from me is mutually-exclusive with me knowing PHP has threads.
EDIT: I also don't understand why you're concerned with "correcting" me when you also recommend a language other than PHP.
-
Because pedantic dickweedery.
-
If C# is an option, but license is a (potential) problem... wouldn't Mono fit the bill here? Or do I really suck at interpreting licenses?
-
I don't think there are any problems with licensing, just a problem of not knowing which technology to use and asking for advice.
-
It was noted before that licensing might be an issue so just pointing out that there might be an alternative. I'm also pretty sure I didn't say anything others didn't think about, but there was some confusion about licenses so hey, might as well add my .02 cents (no, that's not a typo).
Personally, I'd also point out that if you need something that's easy to get networking and multithreading working while being pretty high-performance, there's (fanboy hat, ON!) Qt. Even though it's C++ it's very easy to use, actually encourages you avoid all the mucking about with memory management, and the framework overhead is almost negligible. Also, should be fairly easy to plug existing C/C++ SNMP libraries in it.
Then again, it's a new thing to learn, and if C# is something OP is more comfortable with and has acceptable performance, by all means, it'll do the job.
-
Obligatory verizon link, except I'm at work, so you know.
-
I'm just trying to conceptualize a way to let 2) basically just spawn off 7,000 copies of 1) and still get it done in a reasonable amount of time without eating my server's resources alive.
The classic solution is a thread pool (which you can implement with processes if you want). You have a queue of jobs to do and a limited number of worker tasks. Each worker task pulls an item off the queue (waiting if the queue is empty), carries out the job, and then once it's done, goes back and gets the next item off the queue. You scale the number of workers to your hardware and the complexity of the jobs, and monitoring the average queue length lets you figure out if you're being too cheap (the queue length shouldn't grow too much). Periodically you just pitch all the items to process into the queue; the rest of the machinery should handle the rest from there.
In your case, stash the results in a database table at the end of each job, and maybe use some sort of priority queue trickery to handle the “do it now” jobs (though getting a good PQ can take more effort than a simple queue). Like that, when you show something to a user you can show them data immediately and at the same time give an estimate of how stale the results are while you recalculate them. It's acceptable to put the queue itself in the database though that does mean that you need to make the workers poll every few seconds while waiting instead of using some smart locking tricks. (Yet more tuning parameters…)
Loads of languages can do this sort of thing.
-
The licensing issue is as follows:
Visual Studio Express does not have windows service as a project type option. You can still write services, it just doesn't do the boiler-plate stuff for you. If you use Visual Studio Express everything is fine.
VS professional includes a project template that hands you a windows service on a silver platter, just tell it what to do. If you are using a Microsoft Partner Network license however, that license doesn't cover you using it in any way that interacts with customers, it can only interact with internal business users (e.g. producing reports for engineers on where there are faults they need to fix).
So, you CAN use C# license free for commercial purposes as much as you like, as long as you don't use VS professional.
-
So, you CAN use C# license free for commercial purposes as much as you like, as long as you don't use VS professional.
Brain... hurts...
So, if someone generates the boilerplate for me in VS professional, and I copy that code into MonoDevelop I'm fine? Oversimplifying of course.
-
Not sure, ask your local MSDN reseller/lawyer. I would hazard "Probably" but I think there might be slightly more to it than I am aware of (service installers and similar which might just not work outside of VS pro)
I learned these pitfalls whilst trying to convince people to buy MSDN recently.
-
Useless to me, just trying to clarify what seems to me to be a legal WTF. CBA to go much further than a question and maybe a Google search. @izzion might want to look it up though.
-
The licensing issue is as follows:
Visual Studio Express does not have windows service as a project type option. You can still write services, it just doesn't do the boiler-plate stuff for you. If you use Visual Studio Express everything is fine.
- I don't get how that's a "licensing" issue.
- The boilerplate takes like maybe 2 minutes to fill-in.
-
Fortunately for me, this qualifies as an "internal use" thing, so I don't really have to sweat the licensing issue (for now, at least,
)
-
I don't get how that's a "licensing" issue.
That's because it's explained in my second paragraph.
VS professional includes a project template that hands you a windows service on a silver platter, just tell it what to do. If you are using a Microsoft Partner Network license however, that license doesn't cover you using it in any way that interacts with customers, it can only interact with internal business users (e.g. producing reports for engineers on where there are faults they need to fix).
Relevant excerpts:
VS professional includes a project template
Microsoft Partner Network license
doesn't cover you using it in any way that interacts with customers,
and, from the OP's subsequent post:
We actually do have a MSDN license via our partner program silver competency
So, to recap licensing:
- VS pro includes some useful (granted not necessary) tools for this task, but requires a license
- OP mentioned he had a specific type of license with some specific limitations
- I explained that license may not work for him, depending on the situation, and suggested he bear that in mind.
- You didn't read thoroughly enough.
- I repeated myself.
better?
-
Fortunately for me, this qualifies as an "internal use" thing, so I don't really have to sweat the licensing issue (for now, at least, )
Glad that's how its going, I have actually worked with services and SNMP recently. SnmpSharpNet is a library that makes querying/modifying SNMP agents services quite easy, I suggest you look it up to help if you are going down the .NET route. It has plenty of tutorials for various SNMP operations.
see: http://www.snmpsharpnet.com/?page_id=111 there are also download links about the place on that site.
best of luck!
-
So use Express. Which lets you make a Windows Service and has zero restrictions on how it's used.
I still don't get the issue.
Just because a particular version of VS lacks a particular template doesn't mean it's against the law to use it to make that type of project.
-
Just because a particular version of VS lacks a particular template doesn't mean it's against the law to use it to make that type of project.
Funnily enough, I NEVER SAID THAT!. I was pretty clear that using the vs pro license that he has for commercial purposes is a violation of that license. I didn't say he couldn't use express (although I did say it's harder, which it is), I didn't say that it was illegal to use express. I said exactly, specifically, and entirely, that he should be careful to make sure, if he chooses to use his VS pro license (which he has) that he is not using it for direct commercial purposes (which he has, and isn't)
I kind of feel like I am saying "If you want to use Tony's sledgehammer to knock down that wall in your garden, get his permission, don't just break in and take it", and you are responding by saying "I don't understand, why don't you just knock down the wall with a lump hammer? Tony doesn't need to give you permission to knock down the wall." Both statements are correct, but you seem convinced that yours makes mine wrong.
-
It's clarifying discourse, blakey. No confusion here.
PHP: pthreads - Manual
