If I restart the service I can connect again after a little while.
I am connecting with HTTP, and have made no changes to sessions, threads or queue length.
In response to MonkeyEdit2000's post - I was running the service on a XP Pro 32bit machine at first and experienced and had to reboot once as well. After that I moved to a 2008 R2 Foundation instead (the service has frozen as before but no reboot have been necessary).
As MonkeyEdit2000 writes, I havent been able to determine when it occurs. I usually leave the console logged in, since it logs me out after a while anyways. It does not, however, seem to matter whether I leave the console logged in or not, the service becomes unresponsive either way.