Skip to main content

​
Enhancing AI Applications with Web Data

Learn how to integrate ScrapeGraphAI with your AI and LLM applications to enhance their capabilities with real-time web data.

​
Common Use Cases

  • RAG (Retrieval Augmented Generation): Enhance your LLM responses with up-to-date web content
  • AI Assistants: Build domain-specific AI assistants with access to web data
  • Knowledge Bases: Create and maintain dynamic knowledge bases from web sources
  • Research Agents: Develop autonomous agents that can research and analyze web content

​
Integration Examples

​
RAG with LangChain

from langchain import LLMChain from scrapegraph_py import Client from pydantic import BaseModel, Field from typing import Optional  class ArticleSchema(BaseModel):  """Schema for article content"""  title: str = Field(description="Article title")  content: str = Field(description="Main article content")  author: Optional[str] = Field(description="Article author name")  date: Optional[str] = Field(description="Publication date")  summary: Optional[str] = Field(description="Article summary or description")  # Initialize the client client = Client()  try:  # Scrape relevant content  response = client.smartscraper(  website_url="https://example.com/article",  user_prompt="Extract the main article content, title, author, and publication date",  output_schema=ArticleSchema  )   # Use in your RAG pipeline  text_content = f"Title: {response.title}\n\nContent: {response.content}"  docs = text_splitter.split_text(text_content) # Most text splitters expect string input  vectorstore.add_documents(docs)   # Query your LLM with the enhanced context  response = llm_chain.run("Summarize the latest developments...")  except Exception as e:  print(f"Error occurred: {str(e)}")

​
AI Research Assistant

from scrapegraph_py import Client from pydantic import BaseModel, Field from typing import List  class ResearchData(BaseModel):  title: str = Field(description="Article title")  content: str = Field(description="Main article content")  author: str = Field(description="Article author")  date: str = Field(description="Publication date")  class ResearchResults(BaseModel):  articles: List[ResearchData]  # Initialize the client client = Client()  try:  # Search and scrape multiple sources  search_results = client.searchscraper(  user_prompt="What are the latest developments in artificial intelligence?",  output_schema=ResearchResults,  num_results=5, # Number of websites to search (3-20)  extraction_mode=True # Use AI extraction mode for structured data  )   # Process with your AI model  if search_results and search_results.articles:  analysis = ai_model.analyze(search_results.articles)  print(f"Analyzed {len(search_results.articles)} articles")  else:  print("No articles found in the search results")  except Exception as e:  print(f"Error during research: {str(e)}")

​
Best Practices

  1. Data Freshness: Regularly update your knowledge base with fresh web content
  2. Content Filtering: Use our filtering options to get only relevant content
  3. Rate Limiting: Implement appropriate rate limiting for production applications
  4. Error Handling: Always handle potential scraping errors gracefully
⌘I