Turn Any Website into an API

1. crawler-buddy

   1 9 199 9.6 Python

   Crawling framework, RSS reader and parser

   

   For YouTube yet dlp is also used to augment results.

   I can crawl using requests, selenium, Httpx and others. Response is via json so it easy to process.

   The downside is that it may not be the fastest solution, and I have not tested it against proxies.

   https://github.com/rumca-js/crawler-buddy

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. cinescrapers

   2 3 8 9.5 Python

   Web scrapers for London indie cinemas

   

   This is relevant to my interests[0]

   Based on the website I was quite skeptical. It looks too much like an "indiehacker", minimum-almost-viable-product, fake-it-till-you-make-it, trolling-for-email-addresses kind of website.

   But after a quick search on twitter, it seems like people are actually using it and reporting good results. Maybe I'll take a proper look at it at some point.

   I'd still like to know more about pricing, how it deals with cloudflare challenges, non-semantic markup and other awkwardnesses.

   [0] https://github.com/Joeboy/cinescrapers

4. pydoll

   3 7 6,227 9.8 Python

   Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

   

   It's not a browser extension, but controlling the actual browser without using webdriver is already a thing.

   https://github.com/autoscrape-labs/pydoll

Suggest a related project