Python Forum

hi Team

I am new to Python and try to crawl gmail without using any google or gmail API.

please advise me how can i pass value to gmail id and password and then click using BeautifulSoup.

I am using IDLE python 3

THanks
Ratan

it would probably be easier with selenium to login and get the page content and then pass it to BS

from selenium import webdriver from bs4 import BeautifulSoup import time EMAIL = '' PASSWORD = '' CHROMEPATH = '' login_url = 'https://www.google.com/accounts/Login' browser = webdriver.Chrome(CHROMEPATH) browser.set_window_position(0,0) browser.get(login_url) time.sleep(2) browser.find_element_by_id("identifierId").send_keys(EMAIL) browser.find_element_by_id("identifierNext").click() time.sleep(2) browser.find_element_by_name("password").send_keys(PASSWORD) browser.find_element_by_id("passwordNext").click() time.sleep(2) soup = BeautifulSoup(browser.page_source, 'html.parser')

Hey thanks for this

I ll try selenium

tell me can we use something else other than sleep i mean is there any way to wait until the page loads

You can wait until an element exists

from selenium.webdriver.support.wait import WebDriverWait element = WebDriverWait(driver, 10).until( lambda x: x.find_element_by_id("someId"))

or

from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))

And there are other variations http://www.obeythetestinggoat.com/how-to...click.html

Hm! I was using selenium several times and there is a delay when opening a web page.  Perhaps it returns when this is done?

the time it takes to load a webpage can be different for each person, depending on your internet speed and computer, etc.

@OP I added the above example in for wait until elements

from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By from bs4 import BeautifulSoup import time EMAIL = '' PASSWORD = '' CHROMEPATH = '' login_url = 'https://www.google.com/accounts/Login' browser = webdriver.Chrome(CHROMEPATH) browser.set_window_position(0,0) wait = WebDriverWait(browser, 2) presence = EC.presence_of_element_located visible = EC.visibility_of_element_located browser.get(login_url) wait.until(presence((By.ID, 'identifierId'))) browser.find_element_by_id("identifierId").send_keys(EMAIL) browser.find_element_by_id("identifierNext").click() wait.until(presence((By.NAME, 'password'))) browser.find_element_by_name("password").send_keys(PASSWORD) browser.find_element_by_id("passwordNext").click() wait.until(visible((By.CSS_SELECTOR, '#gb > div.gb_zd.gb_5d > div.gb_yc.gb_lb.gb_xc.gb_Zd > div > div.gb_ib.gb_4c.gb_yg.gb_R.gb_pf.gb_pb > div > a > span'))) soup = BeautifulSoup(browser.page_source, 'html.parser') print(soup.prettify())