Anybody have experience with proxies?
-
If you're lazy, skip to the TL;DR;
I'm working on a bot for Twitch.tv - And it's coming along extremely nicely. It currently actively moderates 3 channels that have ~3-10k people, and does chat logging in another 10 (stress testing, each channel has > 50k users, highest being 250k events)
All appears well and good. When I finish up a few more features I'd like to start rolling it out in an official capacity. Now the rub:
Twitch.tv does rate limiting for messages by ip. 20 messages per 30 seconds for regular viewers, 100 messages per 30 seconds for moderators, If you send 50 messages as a moderator, then 1 message to a non moderated channel you will get IP banned for 8 hours (because it will bump against the 20 message limit instead of the 100 message limit.
A moderator bot is pretty useless if it's banned for 8 hours, and creating a hosted solution that is suspect to these issues will create significant issues when it comes to growth.
I've already been implementing a queuing system that will prevent it from spamming itself offline, setting message priorities to ensure that high priority items can get out faster if necessary. (IE: Moderator commands go through before chat response commands)
But I believe that when I complete this new bot, it will be about as popular as moobot / nightbot (the current 'go to' bots) due to its unique feature set. The hosted solution is going to be shipped as a moderator / event manager that includes streamer dashboards and blah blah blah.
TL;DR . . . but still long
In order to successfully host a bot for the twitch community, I'm going to need to take advantage of both queuing and proxies to enable sufficient numbers of commands to be issued to maintain a responsive bot.I have several general purpose domains available that are unused for anything. Is it possible to shanghai some of them to act as secondary proxies for sending commands to the twitch servers? (Most are associated with shared hosting via GoDaddy)
Is there something that could be easily managed to generate a reasonably large number of proxy ip's for me to send commands? (No sensitive information is going to be sent, uniqueness of IPs is the only requirement) I was wondering if there's any potential for TOR here, where each new connection would be associated with a new TOR ip - but able to preserve a connected IP until a termination request happens. (with my bot handling unexpected terminations to open a new connection)
Do I just have to bite the bullet and pay for proxy servers? I'd really like to keep as low overhead as possible as I'd very much like the bot itself to be absolutely free, and sell my advanced dashboard information / reports / other stuff.
-
tl;dr
Your proposed approach smacks of something which will come back and bite you; my advice is rethink your plan.
In order to successfully host a bot for the twitch community
There's your problem. Hosting is non-free because it's a service; it incurs real costs (even if they're small in practice, and recoverable through handling advertising, value-add services, etc). Be prepared to think in terms of selling VM-ized deployments of the service so that someone can add the bot to their channel and deal with all the banning issues that go with it; their instance of the bot, their credentials1, their problem (assuming that you've not blundered in your code, of course). You can also offer a free-to-use service, but that will have to have a strict rate limiter applied to it; the responses to the free-loaders may take significantly longer than if they host their own. (You might point this out in the messages you send back if they are being significantly delayed; a gentle advertising opportunity for your pay-for VMs?)
The advantage of this? You're not trying to hack around someone else's terms of service. That's a massive advantage, since the ToS is really a document interpreted by people, and only coincidently by software. You do not want to end up getting someone's legal department jumping down your throat, because that gets very expensive and time-consuming.
1 Assuming that's a meaningful concept to twitch.tv; I've heard of them, but not
wasted timeinvestigated what the reality is like.
-
I have been planning out how to handle multi user environments, with recommendations to use their own user name instead of mine (my hosted service can, but besides the point as twitch manages by ip), they would also have the option to self host the moderation portion to limit message caps. (Which might be sufficient across the board) My services would silently log into the channel for normal recording purposes which has no limiting.
Twitch uses an oauth process for users when coming from third party locations. This oauth token is generated on a per user basis by going to a specific site.
I am well aware of the real life costs associated with free services, and there will be a tipping point where free for one user is too much, but there are some specific goals I'm aiming for:
Market share, there are already two bots in the moderator space which do most of what people need
Education, the services I'm going to offer have rarely been offered to any steamer who was not a league of legends event with their whole in house team of development. I'm looking to get them hooked on high level and drill down reports
Service not offered by other bots, namely much more fluid integration with several products as well as my own services such as tournament and event management.Why is all of this important? I'm playing for a long game that will allow me market analysis and feedback for other projects I'm working on, while creating a revenue stream and getting my name out there in a potentially big way.
Proxies aren't against a term of service for twitch in the context of moderation. They are supporting a large platform with many streamers and viewers and just have sweeping rules to keep things sane.
That said, it's why I'm asking this question. Proxies I host should have significantly less overhead and be more stable than your run of the mill proxy free service, and a decent chunk of private services.
But yes, real world costs is noted, and this is me working out my roi.
-
I have experience with Squid Proxy
I use it for reverse proxies (many requests <> proxy <> server).
but I think it would work for you.
http://www.squid-cache.org/Library/wikimedia.html (white paper)
Hope this helps.
-
This seems to imply user consolidation to take advantage of server caching, making request seem to originate from the intermediary server. Unless I'm reading it wrong, I think this is the opposite of what I want as it would force many ip traffic to look like 1 ip, resulting in a faster ban.
Did I misread it?
-
I (me) used squid proxy as a reverse proxy solution.
I'm not telling you to do that.
You can use it for request forwarding or anonymizing or whatever your particular proxy situation might be.
-
Here is a configuration guide (all in one page)...you can use CTRL-F and search around for answers.
and on the frontPage second section is 'popular topics'...It may give you some ideas as well.
-
Ctrl...f? What is this feature? It. . It doesn't work here!
-
I'm un-social engineering you to freedom of the internets and the power of CTRL-F...go now and search, search away....!
-
Too sleepy. Ctrl+f another day. Freedom is too overwhelming.
But really, I'll look this over. I've looked at squid a few times in the past but never had any real use for it. I will review again and see if it's more relevant for this project. Thanks for the reminder it exists.
Reverse proxy - Wikipedia
