@Corvidae said:
The "textbook" way to handle clients like this is to spawn off a thread as each client connects and handle all the I/O in there. Since I need to be able to send messages across the pipe while blocking for input, I think I'd need two threads per client in order to handle both a send and receive queue (barring something like a shared write queue across all clients, but that causes its own bottlenecks). This doesn't sound like the best way to handle this, however, as the context switching alone would cause a ton of overhead.
I don't think this is the textbook way of handling multiple connections. It's more of a text book way to explain why threading may be useful. As with most things in text books, it's a great concept but probably won't work in practice...
I don't think that having one thread per connection is a good idea. Let's say that you're accepting 5000 connections. You won't be able to process all of the simultaneously, since you'll run into CPU and I/O bottle necks. If your server has 4 CPUs with 8 cores each, you can process 32 requests simultaneously. That's it. Spawning more threads makes it easy on you as a programmer, but I don't think that it'll work well.
I'd start out with having one thread accept connections and create a session state object for that thread. This thread should only accept the connection, but not process any data; all it needs to do is plop the session object into a queue. Create a thread pool with enough worker threads to keep all your cores busy and have it process the work queue.
Each session object should have a handle for the tcp/ip connection, which you can poll for data, and a message queue of data that needs to be send back to the client. You could either have a homogenous thread pool where every thread receives (and processes) messages, then sends out any responses that may be queued, or have one pool that receives/processes and one that transmits responses. To make this more efficient, you can make use of the select() API to pick out those threads that have transmitted data instead of looping over the entire set.
Once you deal with thousands of connections you'll also quickly run into the limitations of the operating systems tcp/ip stack. You'll have to fine tune all sorts of buffer and connection related settings to be able to hold this many connections open simultaneously.