Skip to content

bpo-32222: Fix pygettext skipping docstrings for funcs with arg typehints #4745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Feb 26, 2018
Merged

bpo-32222: Fix pygettext skipping docstrings for funcs with arg typehints #4745

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
00405d2
Fix pygettext skipping docstrings for funcs with arg typehints
Tobotimus Dec 7, 2017
09abc0e
Add news entry
Tobotimus Dec 7, 2017
bcbfd74
Revert unwanted accidental changes
Tobotimus Dec 7, 2017
7f5d2c7
Revert accidental encoding change
Tobotimus Dec 7, 2017
eb2267b
Skip over nested parens in param list
Tobotimus Dec 7, 2017
6633e80
Properly reset paren count to zero
Tobotimus Dec 7, 2017
7e974fa
Add tests
Tobotimus Dec 13, 2017
30cc30d
Refactor logic
Tobotimus Dec 13, 2017
36779af
Removing trailing whitespace
Tobotimus Dec 13, 2017
cd6ea5c
Fix whitespace
Tobotimus Dec 13, 2017
038d01c
Add tests for class
Tobotimus Dec 13, 2017
cf12715
Funcs and classes use same logic
Tobotimus Dec 13, 2017
2f3dbf5
One integer count for all enclosures
Tobotimus Dec 14, 2017
fc0f1ea
Update news entry
Tobotimus Dec 15, 2017
8c633ff
Update news entry v2
Tobotimus Dec 15, 2017
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Tools-Demos/2017-12-07-20-51-20.bpo-32222.hPBcGT.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Fix pygettext not extracting docstrings for functions with type annotated
arguments.
18 changes: 13 additions & 5 deletions Tools/i18n/pygettext.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# Minimally patched to make it even more xgettext compatible
# by Peter Funk <pf@artcom-gmbh.de>
#
# 2002-11-22 J�rgen Hermann <jh@web.de>
# 2002-11-22 Jürgen Hermann <jh@web.de>
Copy link
Member

@merwok merwok Dec 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that the change here is a re-encoding from Latin-1 to UTF-8. If that is so, please remove the encoding line at the top of the file.
Copy link
Member

@serhiy-storchaka serhiy-storchaka Dec 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better to not include any unrelated changes. If you want to change the encoding of the file, it should be done in a separate issue. But non-default encoding may be a part of testing.
Copy link
Contributor Author

@Tobotimus Tobotimus Dec 7, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed it was a change of encoding, thanks for pointing that out; my text editor seems to save files in UTF-8 by default. I've reverted the encoding to ISO 8859-1.
# Added checks that _() only contains string literals, and
# command line args are resolved to module lists, i.e. you
# can now pass a filename, a module or package name, or a
Expand Down Expand Up @@ -208,7 +208,7 @@ def make_escapes(pass_nonascii):
global escapes, escape
if pass_nonascii:
# Allow non-ascii characters to pass through so that e.g. 'msgid
# "H�he"' would result not result in 'msgid "H\366he"'. Otherwise we
# "Höhe"' would result not result in 'msgid "H\366he"'. Otherwise we
# escape any character outside the 32..126 range.
mod = 128
escape = escape_ascii
Expand Down Expand Up @@ -340,13 +340,21 @@ def __waiting(self, ttype, tstring, lineno):
elif ttype not in (tokenize.COMMENT, tokenize.NL):
self.__freshmodule = 0
return
# class docstring?
if ttype == tokenize.NAME and tstring in ('class', 'def'):
self.__state = self.__suiteseen
# class or func/method docstring?
if ttype == tokenize.NAME:
if tstring == 'def':
self.__state = self.__funcseen
elif tstring == 'class':
self.__state = self.__suiteseen
return
if ttype == tokenize.NAME and tstring in opts.keywords:
self.__state = self.__keywordseen

def __funcseen(self, ttype, tstring, lineno):
# ignore anything until we see the closing parenthesis
if ttype == tokenize.OP and tstring == ')':
Copy link
Member

@serhiy-storchaka serhiy-storchaka Dec 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about the following example?

def foo(bar=()):
Copy link
Contributor Author

@Tobotimus Tobotimus Dec 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a very good point. I will rethink my approach
self.__state = self.__suiteseen

def __suiteseen(self, ttype, tstring, lineno):
# ignore anything until we see the colon
if ttype == tokenize.OP and tstring == ':':
Expand Down