Spell Checker - Finding probability distribution-NLP

Finding Probability Distribution

Kaggle Dataset for Spelling Corrector

You need to download big.txt or Create Notebook.

 import re with open('/kaggle/input/spelling/big.txt','r') as fd: lines=fd.readlines() words=[] for line in lines: words+=line.split(' ') len(words) 

 Output: 1164968 

Explanation:

1. Load the /kaggle/input/spelling/big.txt
2. fd.readlines() readlines read all lines from file and make list of string. Example: lines = [ "I love NLP", "Spell checkers are helpful", "Python is powerful" ]
3. Iterate through that list for line in lines:
4. Now word+=line.split(' ') split line into words, using space as separator.
5. len(words) return the length of words in words[] list.

 import re with open('/kaggle/input/spelling/big.txt','r') as fd: lines=fd.readlines() words=[] for line in lines: words+=re.findall('\w+',line) len(words) 

 Output: 1115585 

re.findall('\w+',line) does the same thing split line into word and then make list of those word. Now, question arises whats the difference.

line.split(' ')

This will separate the line with words by spacing but it will add other character also like spacing, '\','*','.'&' etc.

 print(words[:100]) 

 Output: ['The','Project','Gutenberg', 'EBook','of','The','Adventures', 'of','Sherlock','Holmes\n','by','Sir','Arthur','Conan','Doyle\n', '(#15','in','our','series','by','Sir','Arthur','Conan','Doyle)\n', '\n','Copyright','laws','are','changing','all','over','the', 'world.','Be','sure','to','check','the\n','copyright','laws', 'for','your','country','before','downloading','or','redistributing\n','this','or','any','other','Project','Gutenberg','eBook.\n', '\n','This','header','should','be','the','first','thing','seen', 'when','viewing','this','Project\n','Gutenberg','file.','', 'Please','do','not','remove','it.','','Do','not','change','or', 'edit','the\n','header','without','written','permission.\n', '\n','Please','read','the','"legal','small','print,"','and', 'other','information','about','the\n','eBook','and'] 

re.findall('\w+',line)

\w+ matches any word made up of:

• Letters (A–Z, a–z)
• Numbers (0–9)
• Underscore _

It will not take other than these character.

 print(words[:30]) 

 Output: ['The','Project','Gutenberg','EBook','of','The','Adventures','of', 'Sherlock','Holmes','by','Sir','Arthur','Conan','Doyle','15', 'in','our','series','by','Sir','Arthur','Conan','Doyle','Copyright','laws','are','changing','all','over'] 

Lets check how many unique words

 print(len(words)) vocab=list(set(words)) print(len(vocab)) 

 Output: 1115585 38160 

Finding Probability Distribution

It means that how frequently word is repeated, what is the probability of getting the word.

 words.count('the') 

 Output: 79809 

There 79809 'the' word in list. Do remember that use lower() function because it can give 'The' or 'the' different value.
for line in lines:
words+=re.findall('\w+',line.lower())

Probability Distribution

 len(words)/words.count('the') 

 Output: 13.978185417684722 

Probability Distribution for first 10 word

 word_probability={} for word in vocab[:10]: print(word,words.count(word)) 

 Output: susan 1 Tillage 1 shortly 21 enlivened 2 1720 1 victors 3 shipments 2 Go 100 constitution 63 blur 1 

If we want probability of all word

 from tqdm import tqdm word_probability={} for word in tqdm(vocab): word_probability[word] = float(words.count(word)/len(words))