Runtime Failures on TCP Stress Testing: (libuv) kqueue(): Too many open files and more #12666
Description
I'm on OSX 10.8.4, x86_64, latest Rust on master. And to head off any concerns about my file handle limits:
$ ulimit unlimited I ran into this when stress-testing my web sockets library. After just a couple hundred connections, some connections were being refused at random. The only thing I could see was some output with RUST_LOG=debug saying (libuv) kqueue(): Too many open files. I've come up with a smaller test case that brings this out with some spectacular errors:
use std::io::net::tcp::{TcpListener, TcpStream}; use std::io::net::ip::{Ipv4Addr, SocketAddr}; use std::io::{Acceptor, Listener}; use std::io::IoResult; fn main() { let addr = SocketAddr { ip: Ipv4Addr(127, 0, 0, 1), port: 8989 }; spawn(proc() { let mut acceptor = TcpListener::bind(addr).listen(); loop { let mut stream = match acceptor.accept() { Ok(stream) => stream, Err(error) => { debug!("accept failed: {:?}", error); continue; } }; println!("{:?}", stream.read_byte().unwrap()); } }); for _ in range(0, 1000) { spawn(proc() { match TcpStream::connect(addr) { Ok(stream) => { let mut stream = stream; stream.write([1]); let mut buf = [0]; stream.read(buf); }, Err(e) => debug!("{:?}", e) } }); } } Fails less often with 100-200 connections. The errors are different every time I run this with RUST_LOG=debug.
Here's the end of one run:
(libuv) kqueue(): Too many open files bug(54921,0x10881c000) malloc: *** error for object 0x7fca4b620f70: pointer being freed was not allocated *** set a breakpoint in malloc_error_break to debug std::io::IoError{kind: OtherIoError, desc: "too many open files", detail: None} task '<unnamed>' failed at 'assertion failed: `(left == right) && (right == left)` (left: `140506823655696`, right: `140506824839680`)', /Users/harish/rust/rust/src/librustuv/homing.rs:110 (libuv) kqueue(): Too many open files std::io::IoError{kind: OtherIoError, desc: "too many open files", detail: None} task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199 Abort trap: 6 Another:
task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199 task failed during unwinding (double-failure - total drag!) rust must abort now. so sorry. Segmentation fault: 11 Another:
task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199 task failed during unwinding (double-failure - total drag!) rust must abort now. so sorry. Illegal instruction: 4 And lastly:
fatal runtime error: leaked managed memory (140518327703008 objects) (libuv) kqueue(): Too many open files task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:Abort trap: 6