... simply move allDone.Set() to after the listener.EndAccept(...). Since the callback is executing on a separate thread, allowing the main loop to continue execution while you continue to process the current connections shouldn't pose a problem - just remember to take out locks when necessary!