• 📝 Table of Contents
• 📙 Overview
• Get Started
  • Infernce Mode
  • Run Locally
• Methods
  • Inverted File Inedex (IVF)
  • Local Sensitive Hashing (LSH)
  • Product Qunatization (PQ)
  • PQ-LSH
• 🕴 Contributors
• 📃 License

Given the embedding of the search query we can efficent get the top matching k results form DB with 20M document.The objective of this project is to design and implement an indexing system for a semantic search database.

Check Final Notebook

https://github.com/ZiadSheriif/IntelliQuery/blob/main/Evaluate_ADB_Project.ipynb 

Clone Repo

git clone https://github.com/ZiadSheriif/IntelliQuery.git 

Install dependencies

pip install -r requirements.txt 

Run Indexer

$ python ./src/evaluation.py 

This is out final Approach with Some Enhancements

1. Changed MiniBatchKMeans to regular KMeans
2. We calculate initial centroids with just the first chunk of data
3. Introduced parallel processing for different regions

It Combines both LSH & PQ

Ziad Sherif

Zeyad Tarek

Abdalhameed Emad

Basma Elhoseny

This software is licensed under MIT License, See License for more information ©Ziad Sherif.