scrapy-playwright
🎠Playwright integration for Scrapy (by elacuesta)
httpcore
A minimal HTTP client. ⚙️ (by encode)
| scrapy-playwright | httpcore | |
|---|---|---|
| 14 | 5 | |
| 1,326 | 527 | |
| 2.9% | 0.4% | |
| 5.9 | 4.9 | |
| 12 days ago | 18 days ago | |
| Python | Python | |
| BSD 3-clause "New" or "Revised" License | BSD 3-clause "New" or "Revised" License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
scrapy-playwright
Posts with mentions or reviews of scrapy-playwright. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2024-08-22.
- Current problems and mistakes of web scraping in Python and tricks to solve them!
Middleware libraries are written by the community and are extending their functionality. For example, scrapy-playwright.
- Announcing Crawlee Python: Now you can use Python to build reliable web crawlers
While libraries like Scrapy require additional installation of middleware, i.e, scrapy-playwright and still doesn’t work with Windows, Crawlee for Python supports a unified interface for HTTP & headless browsers.
- Scrapy Vs. Crawlee
Scrapy does not support headless browsers natively, but it supports them with its plugin system, similarly it does not support scraping JavaScript rendered websites, but the plugin system makes this possible. One of the best examples is its Playwright plugin.
- Web Scraping Dynamic Websites With Scrapy Playwright
scrapy-playwright is an integration between Scrapy and Playwright. It enables scraping dynamic web pages with Scrapy by processing the web scraping requests using a Playwright instance.
- Turning webpages into pdf
- Scrapy & splash guide
- Web scraping with Python
To integrate Playwright with Scrapy, we will use the scrapy-playwright library. Then, we will scrape https://www.mintmobile.com/product/google-pixel-7-pro-bundle/ to demonstrate how to extract data from a website using Playwright and Scrapy.
- which libraries/frameworks could be used for page interaction?
Scrapy-playwright
- Implementing a Selenium backend on a web app?
your website is a dynamic there is many integration on scrappy can help you This the best best one https://github.com/scrapy-plugins/scrapy-playwright
- Is Selenium still a good choice?
This concern should be lifted if you are a Scrapy lover. There is a Scrapy integration for playwright, that gives you a lot of freedom and lets you operate from a Scrapy spider.
httpcore
Posts with mentions or reviews of httpcore. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2025-12-14.
- Ask HN: What Are You Working On? (December 2025)
Httpx/httpcore are abandoned and their connection pooling etc implementations are completely broken https://github.com/encode/httpx/issues/3215 Many PRs trying to fix varying issues https://github.com/encode/httpcore/pulls But maintainer(s) are not intrested in fixing any of the issues.
- How to write a DRY combined sync / async Python package
There is a nice package called unasync that can do this for you, but the simplest and cleanest implementation I have found was in the encode/httpcore package:
- Current problems and mistakes of web scraping in Python and tricks to solve them!
You might expect me to write here an equally elegant solution for httpx, but no. About six months ago, you could do the "dirty trick" and change the basic httpcore parameters that it passes to h2, which are responsible for the HTTP2 handshake. But now, as I'm writing this article, that doesn't work anymore.
- Useful helper libraries
- RuntimeError: Event loop is closed
Doing a specific google for that returns https://github.com/encode/httpx/issues/914 which shows they fixed it in their core component HTTP Core via this pull request
What are some alternatives?
When comparing scrapy-playwright and httpcore you can also consider the following projects:
scrapy-splash - Scrapy+Splash for JavaScript integration
crawlee-python - Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.
fastcore - Python supercharged for the fastai library
scrapy-fake-useragent - Random User-Agent middleware based on fake-useragent
pg-nearest-city - Given a geopoint, find the nearest city using PostGIS (reverse geocode).