➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Created on 2019-09-23 07:48 by zmk, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 32000 merged gregory.p.smith, 2022-03-19 23:06
PR 32013 merged gregory.p.smith, 2022-03-20 20:31
PR 32015 merged gregory.p.smith, 2022-03-20 21:52
Messages (10)
msg352992 - (view) Author: marko kreen (zmk) Date: 2019-09-23 07:48 
Following code: ---------------------------- import binascii, zlib bigdata=memoryview(bytearray((1<<32) + 100)) print(binascii.crc32(bigdata)) crc = binascii.crc32(bigdata[:1000]) crc = binascii.crc32(bigdata[1000:], crc) print(crc) print(zlib.crc32(bigdata)) crc = zlib.crc32(bigdata[:1000]) crc = zlib.crc32(bigdata[1000:], crc) print(crc) --------------------------- Outputs with python3.7 $ python3.7 x.py 2575877834 2838121701 2838121701 2838121701
msg352993 - (view) Author: Александр Семенов (iamsav) Date: 2019-09-23 08:01 
I've got Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 2838121701 2838121701 2838121701 2838121701
msg352994 - (view) Author: marko kreen (zmk) Date: 2019-09-23 08:16 
Looking at the code, the bug is dependant on USE_ZLIB_CRC32 define and it's unfixed in master. Cause is zlib crc32 that takes length as integer, zlibmodule has workaround, binascii has not.
msg352995 - (view) Author: marko kreen (zmk) Date: 2019-09-23 08:49 
Found zlibmodule fix: https://bugs.python.org/issue10276
msg415577 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-03-19 22:25 
``` $ python3.8 Python 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import binascii, zlib >>> bigdata=memoryview(bytearray((1<<32) + 100)) >>> >>> print(binascii.crc32(bigdata)) 2838121701 >>> crc = binascii.crc32(bigdata[:1000]) >>> crc = binascii.crc32(bigdata[1000:], crc) >>> print(crc) 2838121701 >>> >>> print(zlib.crc32(bigdata)) 2838121701 >>> crc = zlib.crc32(bigdata[:1000]) >>> crc = zlib.crc32(bigdata[1000:], crc) >>> print(crc) 2838121701 ```
msg415578 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-03-19 22:35 
it depends on the build. USE_ZLIB_CRC32 causes it due to zlib's 32-bitness as noted my marko. $ ./python Python 3.11.0a6+ (heads/main-dirty:b3f2d4c8ba, Mar 19 2022, 15:32:04) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import binascii, zlib >>> bigdata=memoryview(bytearray((1<<32) + 100)) >>> >>> print(binascii.crc32(bigdata)) 2575877834 >>> crc = binascii.crc32(bigdata[:1000]) >>> crc = binascii.crc32(bigdata[1000:], crc) >>> print(crc) 2838121701 >>> >>> print(zlib.crc32(bigdata)) 2838121701 >>> crc = zlib.crc32(bigdata[:1000]) >>> crc = zlib.crc32(bigdata[1000:], crc) >>> print(crc) 2838121701 >>>
msg415628 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-03-20 19:28 
 New changeset 9d1c4d69dbc800ac344565119337fcf490cdc800 by Gregory P. Smith in branch 'main': bpo-38256: Fix binascii.crc32() when inputs are 4+GiB (GH-32000) https://github.com/python/cpython/commit/9d1c4d69dbc800ac344565119337fcf490cdc800
msg415636 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-03-20 21:47 
 New changeset 4c989e19c84ec224655bbbde9422e16d4a838a80 by Gregory P. Smith in branch '3.10': [3.10] bpo-38256: Fix binascii.crc32 large input. (GH-32000) (GH-32013) https://github.com/python/cpython/commit/4c989e19c84ec224655bbbde9422e16d4a838a80
msg415667 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-03-21 06:34 
 New changeset 58a7e130375776b192a99b013bc563205a639edc by Gregory P. Smith in branch '3.9': bpo-38256: Fix binascii.crc32 large input. (GH-32000) (GH-32013) (GH-32015) https://github.com/python/cpython/commit/58a7e130375776b192a99b013bc563205a639edc
msg415668 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-03-21 06:37 
3.8 and older are in security fix only mode. 3.9+ have been fixed. realistically this was a rare issue with multiple workarounds. The 3.9 and 3.10 fixes were strictly bugfix only. The 3.11 fix included some performance improvements.
History
Date User Action Args
2022-04-11 14:59:20adminsetgithub: 82437
2022-03-21 06:37:04gregory.p.smithsetstatus: open -> closed
resolution: fixed
messages: + msg415668

stage: patch review -> commit review
2022-03-21 06:34:49gregory.p.smithsetmessages: + msg415667
2022-03-20 21:52:15gregory.p.smithsetpull_requests: + pull_request30103
2022-03-20 21:47:05gregory.p.smithsetmessages: + msg415636
2022-03-20 20:31:12gregory.p.smithsetpull_requests: + pull_request30101
2022-03-20 19:28:39gregory.p.smithsetmessages: + msg415628
2022-03-19 23:48:10gregory.p.smithsetassignee: gregory.p.smith
2022-03-19 23:06:19gregory.p.smithsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request30089
2022-03-19 22:35:31gregory.p.smithsetstatus: closed -> open
versions: + Python 3.8, Python 3.9, Python 3.10, Python 3.11
title: binascii.crc32 is not 64-bit clean -> binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32
messages: + msg415578

resolution: works for me -> (no value)
stage: resolved -> needs patch
2022-03-19 22:25:10gregory.p.smithsetstatus: open -> closed

versions: - Python 3.8, Python 3.9
nosy: + gregory.p.smith

messages: + msg415577
resolution: works for me
stage: resolved
2019-09-23 09:18:28serhiy.storchakasetnosy: + nadeem.vawda
2019-09-23 08:49:04zmksetmessages: + msg352995
versions: + Python 3.8, Python 3.9
2019-09-23 08:16:36zmksetmessages: + msg352994
2019-09-23 08:01:35iamsavsetnosy: + iamsav
messages: + msg352993
2019-09-23 07:48:21zmkcreate