Robot Framework VS Scrapy

Fixing a bug is incomplete without preventing its recurrence. Root cause analysis (RCA), coupled with regression testing and documentation, ensures long-term reliability. Antony Marceles, Founder of Pumex Computing, emphasizes, “Fixing a bug is only part of the solution, preventing it from happening again is the real goal.” Marceles’ team uses regression tests via Robot Framework and code reviews with Gerrit to maintain quality, documenting fixes in Confluence to share insights. A 2023 Forrester report found that teams with strong RCA practices reduce recurring bugs by 35%.

Documentation is your best friend. It provides comprehensive guides, examples, and API references to help you navigate the library effectively. Here you can access it, as well as the Robot Framework documentation.

The Robot Framework is an acceptance testing tool that is easy to write and manage due to its key-driven approach. Let us learn more about the Robot Framework to enable acceptance testing.

Well, I work with software quality and despite not having a strong foundation in automation, one fine day I decided to make a change. I have been working with Robot Framework for a few months - and that's when I got a taste of the power of python. Some time later, I dabbled a little with Cypress and Playwright, always using javascript.

Robot Framework

I've used Lua/Busted in a data-heavy environment (telemetry from hospital ventilators). I've also used robot: https://robotframework.org/

I can't say whether any of these will work, but maybe one of: PyAutoGui pytest-qt Robot Framework + plugins

User-Agent: Scrapy/2.11.0 (+https://scrapy.org)

Last week, I worked on code refactoring in Scrapy, which is an essential practice in larger and more complex projects. Refactoring not only improves code maintainability but also makes it easier for other contributors to understand and extend the project. This task was a good starting point for me to verify that I had the Scrapy project correctly set up locally, as refactoring of codes should not break existing functionalities.

In the past three months, I worked on various open source projects, including my own project Repo Context Packager, Math Worksheet Generator and Open Web Calendar. This month, I want to challenge myself to work on a larger and more widely used project - Scrapy, a Python module for web crawling.

What I have seen it is hard to tell what "serious scrapers" use. They use many things. Some use this, some not. This is what I have learned reading webscraping on reddit. Nobody speaks things like that out loud.

There are many tools, see links below

Personally I think that running selenium can be a bottle neck, as it does not play nice, sometimes processes break, even system sometimes requires restart because of things blocked, can be memory hog, etc. etc. That is my experience.

To be able to scale I think you have to have your own implementation. Serious scrapers complain about people using selenium, or derivatives as noobs, who will come back asking why page X does not work in scraping mechanisms.

https://github.com/lexiforest/curl_cffi

https://github.com/encode/httpx

https://github.com/scrapy/scrapy

https://github.com/apify/crawlee

Scrapy is a robust and scalable open-source web crawling framework. It is highly efficient for large-scale projects and supports asynchronous scraping.

Language: Python | GitHub: 52.9k stars | link

One might ask, what about Scrapy? I'll be honest: I don't really keep up with their updates. But I haven't heard about Zyte doing anything to bypass TLS fingerprinting. So out of the box Scrapy will also be blocked, but nothing is stopping you from using curl_cffi in your Scrapy Spider.

Install scrapy (Offical website) either using pip or conda (Follow for detailed instructions):