How to Use Private Data in Generative AI: End-to-End
Solution for RAG with CrateDB and LangChain
Marija Selakovic, Developer Advocate, CrateDB
Christian Kurze, VP Product, CrateDB
1
What is Generative AI?
Generative AI is a set of artificial intelligence methodologies that can produce novel
content that resembles the training data they were exposed to.
The content could be anything spanning from synthesizing text, generating
code, realistic images, music and more.
Text Instructions
Text
Prompt
Images Images
Audio Audio
Training Generate
Billions of
Video Parameters
Video
Foundational Model
2
Huge Potential, but also Challenges of Generative AI
Quality & Reliability: Hallucinations, accuracy,
timely input data
Ethical & Societal: Deepfakes, misinformation, bias
in AI-generated content require policies and controls
Computational Costs & Environmental Impact:
High power required to run large generative AI
models
Intellectual Property & Copyright: Generated
content resembles human-created work
Managing & Governing AI: Frameworks to manage
the development and deployment of generative AI
technologies
3
The Importance of Current, Accurate, Private Data
• Current & Accurate: most recent information
must be available for meaningful answers
• Private data: internal, confidential, sensitive,
subject to privacy regulations
• Utilizing with LLMs:
• Improves accuracy (less hallucinations)
• Enhanced personalization (better user
experience)
• Richer data insights (documentation,
support tickets, legal documents)
4
The AI Dilemma: Fine-tuning vs RAG
• Advantages:
• Updates knowledge with domain-specific data
• More cost-effective than full model retraining
• Challenges:
• Still a need for frequent data update
• Static knowledge (overfitting risk)
• May still produce hallucinations
• Resource intensive
5
Retrieval Augmented Generation (RAG)
6
Benefits of RAG
• Advantages:
• Access control, knowledge not incorporated into the LLM
• Real-time data available
• Reduced training needs
• Flexibility when integrating with different data sources and formats
• Flexibility in choosing embedding algorithms and LLMs
• Challenges:
• Depends on the efficiency of the underlying search system
• Limitations on the amount of context LLMs can consider
• Hallucinations can be reduced, but still might happen
7
How is semantics of language captured? Vectors!
Vectors / Embeddings are numerical representation of data objects (like words, phrases, entire
documents, images, audio, etc.) in a high-dimensional space. They enable systematic access
to unstructured data like similarity search and therefore enable processing and understanding
text in a mathematical form.
Text
[0.2, 0.3, 0.1, …]
Images
[0.5, 0.3, 0.8, …]
[0.6, 0.7, 0.1, …]
Audio
Embedding Model Vectors / Vector
of your choice Embeddings Store
Video
8
Knowledge Assistants - Architecture
Data Chatbot
People
Web Frontend
Directory
API
Internal Wiki
API
Landing Processing Input Response Output
Knowledge Handler Formation Handler
APIs
Parse Data Source
Source 11 Request Handler Output Guardrails
Source Data Routing
Access
Control
Input Guardrails Retriever Context Parser
Source 2 Chunk Chunks
…
Query Improver Context
Enrichment
Source 3 Vectorize …
Embeddings
Intent
Recognition
Chatbot Response
Context Data Vector Store … Response Handler
Backend
LLMs
Embeddings Model
Glossary Prompt Conversation Feedback
Classifiers Responses History Database
Large Language
Configuration Stores Operational Stores
LLM
Model Service Gateway
Monitoring and Reporting
LLM Logging
Usage Cost Data
Reports Reports Reports
Quantum Black: [Link] 9
Knowledge Assistants – Open Source J
Data Chatbot
People
Web Frontend
Directory
API
Internal Wiki
API
Landing Processing Input Response Output
Knowledge Handler Formation Handler
APIs
Parse Data Source
Source 11 Request Handler Output Guardrails
Source Data Routing
Access
Control
Input Guardrails Retriever Context Parser
Source 2 Chunk Chunks
…
Query Improver Context
Enrichment
Source 3 Vectorize …
Embeddings
Intent
Recognition
Chatbot Response
Context Data Vector Store … Response Handler
Backend
LLMs
Embeddings Model
Glossary Prompt Conversation Feedback
Classifiers Responses History Database
Configuration Stores Operational Stores
Large Language LLM Potential for LangChain
Model Service Gateway
Potential for CrateDB
Monitoring and Reporting
LLM Logging
Usage Cost Data
Reports Reports Reports
Quantum Black: [Link] 10
Why CrateDB and LangChain?
• Data comes in many formats: structured,
semi-structured, unstructured; while typical
Robust data management: distributed,
databases can only cope with one type of data highly scalable database natively supporting
and come with custom APIs tables, time-series, geospatial, full-text,
vector; accessible via standard SQL
• 80% of data is unstructured (Gartner)
• Generative AI requires efficient data
management, especially contextualization
Comprehensive set of building blocks
• Foundational Models are only trained on public and swappable libraries to access models,
data vector stores, text splitters, output parsers,
and pre-built chains; covering development,
• (Too?) many alternatives regarding
embedding models and LLMs serving and observability; available in
Python and JavaScript
Demo: Chat for Support Knowledge Base
CrateDB + LangChain for RAG
[Link]
12
Demo
13
Get Started Today!
marija@[Link]
[Link]@[Link]
LangChain: [Link]
LangChain Docs: [Link]
CrateDB Cloud: [Link]
CrateDB Community: [Link]
CrateDB Docs: [Link]
