bpo-28837: Fix lib2to3 handling of map/zip/filter

[stuarteberg]

The 2to3 tool will automatically wrap calls to map, zip, and filter with list():

# Original: zip('abc', '123') # After 2to3: list(zip('abc', '123'))

...but the tool misses some cases, if the call is followed by a "trailer" such as [0]:

# Skipped by 2to3: zip('abc', '123')[0]

This PR augments the lib2to3 "fixers" for map, zip, and filter to correctly handle such cases.

http://bugs.python.org/issue28837

[codecov]

Codecov Report

Merging #24 into master will increase coverage by <.01%.

@@ Coverage Diff @@ ## master #24 +/- ## ========================================== + Coverage 82.37% 82.37% +<.01%  ========================================== Files 1427 1427 Lines 350948 351009 +61 ========================================== + Hits 289088 289144 +56  - Misses 61860 61865 +5

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7ffb99...5cdd2db. Read the comment docs.

[stuarteberg]

@benjaminp: Are you the right person to ping about lib2to3 PRs?

[benjaminp]

What's the reason for this change?

[stuarteberg]

What's the reason for this change?

On line 100 (below), I need to clone the argument list:

new = Node(syms.power, [Name("map"), args.clone()])

If I leave the above line unchanged, then I could have written this, instead:

new = Node(syms.power, [Name("map"), ArgList([args.clone()])])

...but there's a problem with that: It doesn't clone the ( and ) tokens -- it creates them from scratch. As a result, one of the tests fails, due to an unpreserved prefix before the ) token:

====================================================================== FAIL: test_prefix_preservation (Lib.lib2to3.tests.test_fixers.Test_map) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/bergs/Documents/workspace/cpython-git/Lib/lib2to3/tests/test_fixers.py", line 3036, in test_prefix_preservation self.check(b, a) File "/Users/bergs/Documents/workspace/cpython-git/Lib/lib2to3/tests/test_fixers.py", line 3031, in check super(Test_map, self).check(b, a) File "/Users/bergs/Documents/workspace/cpython-git/Lib/lib2to3/tests/test_fixers.py", line 36, in check tree = self._check(before, after) File "/Users/bergs/Documents/workspace/cpython-git/Lib/lib2to3/tests/test_fixers.py", line 32, in _check self.assertEqual(after, str(tree)) AssertionError: "x = list(map( f, 'abc' ))\n\n" != "x = list(map( f, 'abc'))\n\n" - x = list(map( f, 'abc' )) ? --- + x = list(map( f, 'abc')) 

Hence, I think it's best to capture and clone the whole trailer. But if I'm missing something, please let me know.

[benjaminp]

Okay, thanks for the explanation.