How to open a headless browser with AgentQL

Headless browsers are powerful tools for web automation, testing, and scraping. They let you run a browser without the need to spin up a visual interface. This can allow scripts to execute faster or in the background. AgentQL's SDKs leverage headless browsers to interact with web pages and execute queries without the need for a visible user interface.

Overview

This guide shows you how and when to use a headless browser, and how to execute queries inside one with AgentQL.

Why use a headless browser?

Headless browsers offer several advantages:

  • Speed: They're faster than full browsers as they don't render visuals.
  • Resource efficiency: They use less memory and CPU.
  • Automation: Perfect for running tests or scripts without manual intervention.
  • Server-side operation: Can run on machines without a GUI.

AgentQL and Playwright

AgentQL's SDK uses Playwright, a powerful browser automation library, to handle headless browsing. Playwright supports multiple browser engines and provides a rich API for web automation.

Running a query in a headless browser

Here's a basic example of how to use AgentQL with a headless browser:

import agentql from playwright.sync_api import sync_playwright  # Initialise the browser with sync_playwright() as p:  browser = p.chromium.launch(headless=True)  page = agentql.wrap(browser.new_page())   page.goto("https://scrapeme.live/shop/")   SEARCH_QUERY = """  {  products[] {  product_name  product_price(integer)  }  }  """  response = page.query_data(SEARCH_QUERY)   print("RESPONSE:", response)   # Close the browser  browser.close()

This script does the following:

  1. Initializes a headless browser
  2. Navigates to a webpage
  3. Creates an AgentQL query
  4. Executes the query and retrieves the result
  5. Closes the browser

Related content