builtins: Audit bytes arguments #7631
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
As a followup from python#7589 (comment), I audited all occurrences of bytes in builtins.pyi by reading the corresponding C code on CPython main. Most use the C buffer protocol, so _typeshed.ReadableBuffer is the right type. A few check specifically for bytes and bytearray.
JelleZijlstra commented Apr 16, 2022
| def __new__(cls: type[Self], __x: str | bytes | SupportsInt | SupportsIndex | SupportsTrunc = ...) -> Self: ... | ||
| def __new__(cls: type[Self], __x: str | ReadableBuffer | SupportsInt | SupportsIndex | SupportsTrunc = ...) -> Self: ... | ||
| @overload | ||
| def __new__(cls: type[Self], __x: str | bytes | bytearray, base: SupportsIndex) -> Self: ... |
Member Author
There was a problem hiding this comment.
>>> int(memoryview(b"0xdeadbeef"), 16) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: int() can't convert non-string with explicit base >>> int(memoryview(b"123")) 123 Showing that the first overload accepts buffers but the second doesn't.
Collaborator
There was a problem hiding this comment.
| def from_bytes( | ||
| cls: type[Self], | ||
| bytes: Iterable[SupportsIndex] | SupportsBytes, # TODO buffer object argument | ||
| bytes: Iterable[SupportsIndex] | SupportsBytes | ReadableBuffer, |
Member Author
There was a problem hiding this comment.
>>> int.from_bytes([1, 2, 3]) 66051 >>> int.from_bytes(memoryview(b"123")) 3224115 | self, __sub: ReadableBuffer | SupportsIndex, __start: SupportsIndex | None = ..., __end: SupportsIndex | None = ... | ||
| ) -> int: ... | ||
| if sys.version_info >= (3, 8): | ||
| def hex(self, sep: str | bytes = ..., bytes_per_sep: SupportsIndex = ...) -> str: ... |
Member Author
There was a problem hiding this comment.
>>> b"xy".hex(memoryview(b"x")) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: sep must be str or bytes. >>> b"xy".hex(bytearray(b"x")) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: sep must be str or bytes. | @overload | ||
| def __getitem__(self, __s: slice) -> bytes: ... | ||
| def __add__(self, __s: bytes) -> bytes: ... | ||
| def __add__(self, __s: ReadableBuffer) -> bytes: ... |
Member Author
There was a problem hiding this comment.
>>> b"x" + memoryview(b"y") b'xy' | def __setitem__(self, __s: slice, __x: Iterable[SupportsIndex] | bytes) -> None: ... | ||
| def __delitem__(self, __i: SupportsIndex | slice) -> None: ... | ||
| def __add__(self, __s: bytes) -> bytearray: ... | ||
| def __iadd__(self: Self, __s: Iterable[int]) -> Self: ... |
Member Author
There was a problem hiding this comment.
This was wrong; ba += [1, 2, 3] fails
| opener: _Opener | None = ..., | ||
| ) -> IO[Any]: ... | ||
| def ord(__c: str | bytes) -> int: ... | ||
| def ord(__c: str | bytes | bytearray) -> int: ... |
Member Author
There was a problem hiding this comment.
>>> ord(memoryview(b"x")) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: ord() expected string of length 1, but memoryview found 
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
1 similar comment
Contributor
| According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉 |
srittau approved these changes Apr 16, 2022
Collaborator
srittau left a comment
srittau left a comment There was a problem hiding this comment.
Thanks, I didn't double check, but the changes look reasonable.
| def join(self, __iterable_of_bytes: Iterable[ByteString | memoryview]) -> bytes: ... | ||
| def ljust(self, __width: SupportsIndex, __fillchar: bytes = ...) -> bytes: ... | ||
| def join(self, __iterable_of_bytes: Iterable[ReadableBuffer]) -> bytes: ... | ||
| def ljust(self, __width: SupportsIndex, __fillchar: bytes | bytearray = ...) -> bytes: ... |
Collaborator
There was a problem hiding this comment.
Unfortunately, this will also accept memoryview at the moment, but having it more explicit can't hurt.
Member Author
There was a problem hiding this comment.
That's a mypy bug :)
Collaborator
There was a problem hiding this comment.
It's working as documented. In the past when reviewing I've always asked people to remove bytearray from argument types due to that.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
As a followup from #7589 (comment),
I audited all occurrences of bytes in builtins.pyi by reading the corresponding C code
on CPython main.
Most use the C buffer protocol, so _typeshed.ReadableBuffer is the right type. A few
check specifically for bytes and bytearray.