DEV Community

Cover image for Scrapping Google Playstore
Anuoluwapo Balogun
Anuoluwapo Balogun

Posted on

Scrapping Google Playstore

Purpose Of Project :

To analyze how each fin tech apps is being used and widely accepted by Nigerians, as well as understanding the impact fin tech companies has made on Nigerians finances and Nigeri's Tech Sector.

Project Process:

  • Web scraping data from playstore
  • Performing Exporatory Data Analysis On data
  • Visualizing data on Power BI and Tableau

Project Tools:

  • Notebook
  • Power BI
  • Tableau

Importing Necessary Libraries 

import time from selenium import webdriver from webdriver_manager.chrome import ChromeDriverManager
Enter fullscreen mode Exit fullscreen mode

Accessing PlayStore Using Selenium 

driver = webdriver.Chrome(ChromeDriverManager().install()) driver.get('https://play.google.com/store/search?q=fintech%20in%20nigeria&c=apps') time.sleep(10)
Enter fullscreen mode Exit fullscreen mode

Getting All The Apps In The Required Page 

SCROLL_PAUSE_TIME = 5 # Get scroll height last_height = driver.execute_script("return document.body.scrollHeight") time.sleep(SCROLL_PAUSE_TIME) while True: # Scroll down to bottom driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") # Wait to load page time.sleep(SCROLL_PAUSE_TIME) # Calculate new scroll height and compare with last scroll height new_height = driver.execute_script("return document.body.scrollHeight") if new_height == last_height: break last_height = new_height
Enter fullscreen mode Exit fullscreen mode

Scraping App Links 

links_fintech = [] elems = driver.find_elements_by_xpath("//a[@href]") for elem in elems: if "details?id" in elem.get_attribute("href"): links_fintech.append((elem.get_attribute("href"))) links_fintech = list(dict.fromkeys(links_fintech))
Enter fullscreen mode Exit fullscreen mode

Scraping The Necessary Informations From Each App 

list_all_elements = [] for iteration in links_fintech: try: driver.get(iteration) print(iteration) time.sleep(3) header1 = driver.find_element_by_tag_name("h1") downloads = driver.find_elements_by_class_name("htlgb") list_downloads = [] for x in range (len(downloads)): if x % 2 == 0: list_downloads.append(downloads[x].text) titles = driver.find_elements_by_class_name("AHFaub") comments = driver.find_element_by_class_name("EymY4b") list_elements = [iteration,header1.text, list_downloads.append(downloads[x].text), comments.text.split()[0]] for x in range (len(titles)): if titles[x].text == "Download": list_elements.append(list_others[x]) if titles[x].text == "Developer": for y in list_others[x].split("\n"): if "@" in y: list_elements.append(y) break list_all_elements.append(list_elements) except Exception as e: print(e)
Enter fullscreen mode Exit fullscreen mode

https://play.google.com/store/apps/details?id=ng.gov.cbn.speed.wallet.temp
https://play.google.com/store/apps/details?id=com.oxloan.loan.money.credit.nigeria
https://play.google.com/store/apps/details?id=com.cowrywise.android
https://play.google.com/store/apps/details?id=com.fastmoney.loan.credit.cash.nigeria
https://play.google.com/store/apps/details?id=com.wemabank.alat.prod
https://play.google.com/store/apps/details?id=com.ikonik.lamp_pay......

Creating A CSV File For Scraped DataFrame 

import pandas as pd df = pd.DataFrame(list_all_elements,columns=['URL', 'Name', 'downloads', 'install']) df_1 = df.to_csv('fintech_playstore.csv', header = True, index=False, encoding="utf-8")
Enter fullscreen mode Exit fullscreen mode 
df_1
Enter fullscreen mode Exit fullscreen mode

https://www.danielherediamejias.com/scraping-google-play-store-with-python-and-selenium/

https://play.google.com/store/search?q=fintechs%20in%20nigeria&c=apps

Top comments (0)