bpo-32981: Fix catastrophic backtracking vulns (python#5955)

* Prevent low-grade poplib REDOS (CVE-2018-1060) The regex to test a mail server's timestamp is susceptible to catastrophic backtracking on long evil responses from the server. Happily, the maximum length of malicious inputs is 2K thanks to a limit introduced in the fix for CVE-2013-1752. A 2KB evil response from the mail server would result in small slowdowns (milliseconds vs. microseconds) accumulated over many apop calls. This is a potential DOS vector via accumulated slowdowns. Replace it with a similar non-vulnerable regex. The new regex is RFC compliant. The old regex was non-compliant in edge cases. * Prevent difflib REDOS (CVE-2018-1061) The default regex for IS_LINE_JUNK is susceptible to catastrophic backtracking. This is a potential DOS vector. Replace it with an equivalent non-vulnerable regex. Also introduce unit and REDOS tests for difflib. Co-authored-by: Tim Peters <tim.peters@gmail.com> Co-authored-by: Christian Heimes <christian@python.org>

Author: 3 people

Lib/difflib.py

@@ -1083,7 +1083,7 @@ def _qformat(self, aline, bline, atags, btags):
 
 import re
 
-def IS_LINE_JUNK(line, pat=re.compile(r"\s*#?\s*$").match):
+def IS_LINE_JUNK(line, pat=re.compile(r"\s*(?:#\s*)?$").match):
  r"""
  Return 1 for ignorable line: iff `line` is blank or contains a single '#'.
 

Lib/poplib.py

@@ -308,7 +308,7 @@ def rpop(self, user):
  return self._shortcmd('RPOP %s' % user)
 
 
- timestamp = re.compile(br'\+OK.*(<[^>]+>)')
+ timestamp = re.compile(br'\+OK.[^<]*(<.*>)')
 
  def apop(self, user, password):
  """Authorisation

Lib/test/test_difflib.py

@@ -466,13 +466,33 @@ def _assert_type_error(self, msg, generator, *args):
  list(generator(*args))
  self.assertEqual(msg, str(ctx.exception))
 
+class TestJunkAPIs(unittest.TestCase):
+ def test_is_line_junk_true(self):
+ for line in ['#', ' ', ' #', '# ', ' # ', '']:
+ self.assertTrue(difflib.IS_LINE_JUNK(line), repr(line))
+
+ def test_is_line_junk_false(self):
+ for line in ['##', ' ##', '## ', 'abc ', 'abc #', 'Mr. Moose is up!']:
+ self.assertFalse(difflib.IS_LINE_JUNK(line), repr(line))
+
+ def test_is_line_junk_REDOS(self):
+ evil_input = ('\t' * 1000000) + '##'
+ self.assertFalse(difflib.IS_LINE_JUNK(evil_input))
+
+ def test_is_character_junk_true(self):
+ for char in [' ', '\t']:
+ self.assertTrue(difflib.IS_CHARACTER_JUNK(char), repr(char))
+
+ def test_is_character_junk_false(self):
+ for char in ['a', '#', '\n', '\f', '\r', '\v']:
+ self.assertFalse(difflib.IS_CHARACTER_JUNK(char), repr(char))
 
 def test_main():
  difflib.HtmlDiff._default_prefix = 0
  Doctests = doctest.DocTestSuite(difflib)
  run_unittest(
  TestWithAscii, TestAutojunk, TestSFpatches, TestSFbugs,
- TestOutputFormat, TestBytes, Doctests)
+ TestOutputFormat, TestBytes, TestJunkAPIs, Doctests)
 
 if __name__ == '__main__':
  test_main()

Lib/test/test_poplib.py

@@ -308,9 +308,19 @@ def test_noop(self):
  def test_rpop(self):
  self.assertOK(self.client.rpop('foo'))
 
- def test_apop(self):
+ def test_apop_normal(self):
  self.assertOK(self.client.apop('foo', 'dummypassword'))
 
+ def test_apop_REDOS(self):
+ # Replace welcome with very long evil welcome.
+ # NB The upper bound on welcome length is currently 2048.
+ # At this length, evil input makes each apop call take
+ # on the order of milliseconds instead of microseconds.
+ evil_welcome = b'+OK' + (b'<' * 1000000)
+ with test_support.swap_attr(self.client, 'welcome', evil_welcome):
+ # The evil welcome is invalid, so apop should throw.
+ self.assertRaises(poplib.error_proto, self.client.apop, 'a', 'kb')
+
  def test_top(self):
  expected = (b'+OK 116 bytes',
  [b'From: postmaster@python.org', b'Content-Type: text/plain',

Misc/ACKS

@@ -356,6 +356,7 @@ Jonathan Dasteel
 Pierre-Yves David
 A. Jesse Jiryu Davis
 Jake Davis
+Jamie (James C.) Davis
 Ratnadeep Debnath
 Merlijn van Deen
 John DeGood

Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst

@@ -0,0 +1,4 @@
+Regexes in difflib and poplib were vulnerable to catastrophic backtracking.
+These regexes formed potential DOS vectors (REDOS). They have been
+refactored. This resolves CVE-2018-1060 and CVE-2018-1061.
+Patch by Jamie Davis.