- Notifications
You must be signed in to change notification settings - Fork 89
Closed
Labels
Description
Looks like the example with just question marks is good now:
>>> segmenter.segment("??") ['??'] but the example with double question marks as a token at the end of a sentence still loses the question marks:
>>> segmenter.segment("T stands for the vector transposition. As shown in Fig. ??") ['T stands for the vector transposition.', 'As shown in Fig.'] looks like this is the minimal repro:
>>> segmenter.segment("Fig. ??") ['Fig.']