This happens on a 32-bit build on a 64-bit system, which happens to have some interesting properties: for example, malloc() will happily allocate memory larger than Py_SSIZE_T_MAX. The crash is exactly triggered by the following snippet: if sys.maxsize < (1 << 32) and struct.calcsize('P') == 4: self.assertRaises(OverflowError, self.marshal(b'\ta\n\tb').expandtabs, sys.maxsize)

Here is a patch. The idea is to use unsigned arithmetic instead of signed, and to check for overflows by comparing with PY_SSIZE_T_MAX. (the exact reason of the crash is unknown, it looks like either a compiler bug or a mis-optimization as (i < 0) returns 0 while i is -0x7FFFFFFF)

What's the purpose of old_i? It looks like it's never used for anything. Other than than, the patch looks good to me. I'd guess that the "if (i < 0)" was simply optimized away. This isn't necessarily a compiler bug: if I understand correctly, a strict reading of the C standards says it's legitimate for a compiler to assume that code is written in such a way that signed-arithmetic overflow never happens, and gcc (for one) is known to take advantage of this. Also, it would be nice to cleanup the whitespace in this function while you're fixing it; at the moment it's showing me a mixture of tabs and spaces.

Actually, what's the purpose of old_j? It looks like that's not needed any more, either!

You are right, old_i and old_j are unused, they were part of another approach I tried and which failed. Attaching new patch with these 2 variables removed, and the function cleanly reindented (of course the patch is messier because of this).

Looks fine; I think this should be applied. It seems as though your reindentation left trailing whitespace on the blank lines in the function: the svn commit hook might complain about this...

Committed in trunk, py3k, 2.6 and 3.0. Thanks!