Hi!
I’m new with Python and I have been following some tutorials and based on one of them I have the following piece of code:

from scrapy import Selector from urllib.request import urlopen html = urlopen("https://www.pythonparatodos.com.br/formulario.html") sel = Selector(text = html.read()) lista = sel.xpath('//input[@type="text"]') print(lista) for selector in lista: print(selector)

Thanks in advance!

The URL is easily accessed through a browser but when I run the code I got the following errors:

Error:
C:\Users\username\Envs\webcrawler_part_ii\Scripts\python.exe C:/Users/username/PycharmProjects/webcrawler_part_ii/aula44_scrapy_lxml_exemplos.py Traceback (most recent call last): File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1346, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1257, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1303, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1252, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1012, in _send_output self.send(msg) File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 952, in send self.connect() File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1426, in connect self.sock = self._context.wrap_socket(self.sock, File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1040, in _create self.do_handshake() File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\username\PycharmProjects\webcrawler_part_ii\aula44_scrapy_lxml_exemplos.py", line 5, in <module> html = urlopen("https://www.pythonparatodos.com.br/formulario.html") File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen return opener.open(url, data, timeout) File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 517, in open response = self._open(req, data) File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 534, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain result = func(*args) File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1389, in https_open return self.do_open(http.client.HTTPSConnection, req, File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1349, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)> Process finished with exit code 1

I’ve found some possible solutions on google to this with certifi or with the two lines of code below but none of them worked. Can someone help me on this?

import ssl ssl._create_default_https_context = ssl._create_unverified_context

Use Requests and not urllib.
Site also need a User-agent or will not give access.

from scrapy import Selector import requests user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36'} html = requests.get("https://www.pythonparatodos.com.br/formulario.html", headers=user_agent) sel = Selector(text = html.content) lista = sel.xpath('//input[@type="text"]') #print(lista) for selector in lista: print(selector)

Output:
<Selector xpath='//input[@type="text"]' data='<input type="text" name="nome" size="...'> <Selector xpath='//input[@type="text"]' data='<input type="text" name="email" size=...'> <Selector xpath='//input[@type="text"]' data='<input type="text" name="celular" siz...'>

Hi All,
I try to post a question but can not find the icon to do it.
So I drop here and hopefully someone shows me where it is.
Thanks!

(May-09-2022, 10:31 PM)duonght Wrote: I try to post a question but can not find the icon to do it.

Up in right corner there is button Post Thread.
Click on it and write your question-

(May-09-2022, 02:58 PM)snippsat Wrote: Use Requests and not urllib.
Site also need a User-agent or will not give access.

from scrapy import Selector import requests user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36'} html = requests.get("https://www.pythonparatodos.com.br/formulario.html", headers=user_agent) sel = Selector(text = html.content) lista = sel.xpath('//input[@type="text"]') #print(lista) for selector in lista: print(selector)

Output:
<Selector xpath='//input[@type="text"]' data='<input type="text" name="nome" size="...'> <Selector xpath='//input[@type="text"]' data='<input type="text" name="email" size=...'> <Selector xpath='//input[@type="text"]' data='<input type="text" name="celular" siz...'>

Thank you! It works.
Out of curiosity, is there any specific reason for the failure with urllib?