You didn't specify OS, so I will assume it's Linux we're talking about. I think you need to do some research about non-blocking IO, say epoll or asio. It is currently the most effective and scalable way to work with multiple connections simultaneously.
You can start here, for example.