The only valid 2-col splitter is: ' ' (space)

- helps if the words in 'words.txt' contain special UTF whitespaces, which otherwise lead to having >2 columns, - it will make the code a little more robust to 'dirty' dataprep,

Author: KarelVesely84

scripts/rnnlm/get_unigram_probs.py

@@ -130,8 +130,9 @@ def get_counts(data_sources, data_weights, vocab):
 
  with open(counts_file, 'r', encoding="utf-8") as f:
  for line in f:
- fields = line.split()
- assert len(fields) == 2
+ fields = line.split(' ')
+ if len(fields) != 2: print("Warning, should be 2 cols:", fields, file=sys.stderr);
+ assert(len(fields) == 2)
  word = fields[0]
  count = fields[1]
  if word not in vocab: