Skip to content

Whitespaces can be stored in the conllu file, so that the original text can be reconstructed back #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 19, 2021
Merged

Whitespaces can be stored in the conllu file, so that the original text can be reconstructed back #71

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2f5319d
Keep the information on spaces
michnov Feb 15, 2021
ddb9f03
escaping whitespaces in SpacesAfter and SpacesBefore attrs
michnov Feb 16, 2021
8ccf7e5
Whitespace filling can be enbled by a parameter
michnov Feb 19, 2021
443d625
`fill_spaces=True` -> `normalize_spaces=False`
michnov Feb 19, 2021
dd8bb89
fixes after Martin's code review
michnov Feb 19, 2021
c3b28f3
bugfix, missing self
michnov Feb 19, 2021
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixes after Martin's code review
  • Loading branch information
@michnov
michnov committed Feb 19, 2021
commit dd8bb897b7bded6f8585d9593824526b221c34ea
18 changes: 6 additions & 12 deletions udapi/block/tokenize/onwhitespace.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ class OnWhitespace(Block):
preserve whitespaces by filling MISC attributes `SpacesAfter` and `SpacesBefore` (by default True)
"""

escape_whitespace_table = str.maketrans({' ':r'\s', '\t':r'\t', '\r':r'\r', '\n':r'\n'})

def __init__(self, normalize_spaces=True, **kwargs):
super().__init__(**kwargs)
self.normalize_spaces = normalize_spaces
Expand All @@ -39,14 +41,6 @@ def tokenize_sentence(string):
"""A method to be overriden in subclasses."""
return string.split()

@staticmethod
def escape_whitespace(string):
string = re.sub(r' ', r'\\s', string)
string = re.sub(r'\t', r'\\t', string)
string = re.sub(r'\r', r'\\r', string)
string = re.sub(r'\n', r'\\n', string)
return string

def process_tree(self, root):
if root.children:
raise ValueError('Tree %s is already tokenized.' % root)
Expand Down Expand Up @@ -95,9 +89,9 @@ def process_tree(self, root):
node = root.create_child(form=token)
node.ord = i

if i == 1 and len(spaces_before) > 0:
node.misc["SpacesBefore"] = self.escape_whitespace(spaces_before)
if not len(spaces_after):
if i == 1 and spaces_before:
node.misc["SpacesBefore"] = spaces_before.translate(escape_whitespace_table)
if not spaces_after:
node.misc["SpaceAfter"] = 'No'
elif spaces_after != " ":
Copy link
Contributor

@martinpopel martinpopel Feb 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, escape_whitespace_table could be a module-level constant, instead of class-level, but I have no strong preference here, so I will merge this now.
node.misc["SpacesAfter"] = self.escape_whitespace(spaces_after)
node.misc["SpacesAfter"] = spaces_after.translate(escape_whitespace_table)