Current pending calls limit is too small and it can be easily reached in very intensive async file io applications. There is a little hack in pyaio[1] which sleeps if Py_AddPendingCall returns < 0 but It's not totally clear to me why the size of pendind calls array is only 32. [1] https://github.com/felipecruz/pyaio

This array would still have a fixed size. Could you arrange your code so that you enqueue pending calls elsewhere, and call Py_AddPendingCall only once until the pending items have been processed?

I just took a look at https://github.com/felipecruz/pyaio/commit/127372ba0a6dbca4045256dcd653789ee35f6a78 and it looks a bit silly to me: if the completion handler is called in a separate thread, then the completion handler can just take the GIL and put the results in a regular Python queue. In short, using signals for any kind of high-speed I/O sounds completely archaic to me.

Just confirmed that signals is not a viable option. Is too slow, as Antonie already pointed. It's almost 5 times slower than with SIGEV_THREAD. The problem now is: If I use Py_AddPendingCall, the tests can hang sometimes. I can still follow Amaury suggestion tough. If I try to acquire the GIL PyGILState_Ensure() call PyObject_CallObject and release the GIL in the aio_completion_handler function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer.. This branch works only on 3.3 and newer: https://github.com/felipecruz/pyaio/tree/feature/check_no_pending_call This other branch will segfault 2.7.3 with just acquire and release GIL on the completion handler: https://github.com/felipecruz/pyaio/tree/feature/py27_gil_error Since this looks very strange, can someone confirm this behavior? Tested on a Ubuntu12 64.

> If I try to acquire the GIL PyGILState_Ensure() call > PyObject_CallObject and release the GIL in the aio_completion_handler > function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer.. Well, can you post a traceback?

Running test_aio_read [New Thread 0x7ffff7ff7700 (LWP 20681)] [New Thread 0x7ffff61ff700 (LWP 20682)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff61ff700 (LWP 20682)] sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34 34	../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S: No such file or directory. (gdb) backtrace #0 sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34 #1 0x000000000051b053 in PyThread_release_lock (lock=0x0) at Python/thread_pthread.h:346 #2 0x00000000004c868f in PyEval_ReleaseLock () at Python/ceval.c:265 #3 0x0000000000501974 in PyThreadState_DeleteCurrent () at Python/pystate.c:321 #4 0x0000000000502116 in PyGILState_Release (oldstate=PyGILState_UNLOCKED) at Python/pystate.c:652 #5 0x00007ffff660ed44 in aio_completion_handler (sigval=...) at pyaio/core.c:26 #6 0x00007ffff6409cdc in notify_func_wrapper (arg=0x7ffff00008c0) at ../sysdeps/pthread/aio_notify.c:45 #7 0x00007ffff7bc4e9a in start_thread (arg=0x7ffff61ff700) at pthread_create.c:308 #8 0x00007ffff71f14bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #9 0x0000000000000000 in ?? ()

Is it with a debug Python build? `PyThread_release_lock (lock=0x0)` is a bit worrying.

Yes! 2.7.3 build with pydebug.

Without debug backtrace #0 sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34 #1 0x00000000004d456e in PyGILState_Release () #2 0x00007ffff59b9cdc in notify_func_wrapper (arg=0x7ffff00008c0) at ../sysdeps/pthread/aio_notify.c:45 #3 0x00007ffff7bc4e9a in start_thread (arg=0x7ffff59b5700) at pthread_create.c:308 #4 0x00007ffff69b64bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? ()