made various changes to cleanup retrieval #362
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
epinzur changed the title 
nicoloboschi suggested changes Apr 8, 2024
Contributor
nicoloboschi left a comment
nicoloboschi left a comment There was a problem hiding this comment.
LGTM in general
I think we should improve the error handling
epinzur force-pushed the colbert-retriever-improvements branch from 7342320 to 83e5f29 Compare epinzur changed the title epinzur changed the title 
Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
epinzur force-pushed the colbert-retriever-improvements branch from 6f8d18d to d632492 Compare epinzur force-pushed the colbert-retriever-improvements branch from e8e8534 to 4ea9f8c Compare Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
made a bunch of changes that generally make sure our ColBERT retrieval code interacts nicely with Cassandra (doesn't overload the database when making queries).
There is little change in retrieval latency with these changes running on CPU.
wrapped async calls to cassandra inside a
ThreadPoolExecutor.max_casandra_workersworks the best.automatically dropping down to half-precision when calculating chunk scores, if running on GPU and the GPU supports it.
is_fp16flag is getting set toTrueon boxes with CUDA support.