When you're creating a lot of vectors - for example, indexing a bunch of documents at once using OpenAI embeddings - you quickly run into IO-related performance issues. Your web requests will be throttled if you make too many parallel API requests, so OpenAI allows for batched requests via the OpenAI embeddings API. However, this API only allows for a maximum of 8,191 tokens per request: about 32,764 characters.
Solution: @instant.dev/vectors provides a simple VectorManager utility that performs automatic, efficient batch creation of vectors. It will automatically collect vector creation requests over a 100ms (configurable) timeframe and batch them to minimize web requests.
It is most useful in web server contexts where multiple user requests may be creating vectors at the same time. If you rely on the same VectorManager instance all of these disparate requests will be efficiently batched.
To use this library you'll need to also work with a vector creation tool, like OpenAI.
npm i @instant.dev/vectors --save # vector management npm i openai --save # openai for the engineCommonJS:
const { VectorManager } = require('@instant.dev/vectors'); const OpenAI = require('openai'); const openai = new OpenAI({apiKey: process.env.OPENAI_API_KEY}); const Vectors = new VectorManager();ESM:
import { VectorManager } from '@instant.dev/vectors'; import { Configuration, OpenAIApi } from "openai"; const configuration = new Configuration({ organization: "YOUR_ORG_ID", apiKey: process.env.OPENAI_API_KEY, }); const openai = new OpenAIApi(configuration); const Vectors = new VectorManager();Once you've imported and instantiated the package, it's easy to use.
// values will automatically be batched appropriately Vectors.setEngine(async (values) => { const embeddingResult = await openai.embeddings.create({ model: 'text-embedding-ada-002', input: values, }); return embeddingResult.data.map(entry => entry.embedding); });let vector = await Vectors.create(`Something to vectorize!`);Manually manage vector creation:
const myStrings = [ `Some string!`, `Can also be a lot longer`, `W`.repeat(1000), // ... ]; let vectors = await Promise.all(myStrings.map(str => Vectors.create(str)));Or create multiple vectors easily with the batchCreate utility:
const myStrings = [ `Some string!`, `Can also be a lot longer`, `W`.repeat(1000), // ... ]; let vectors = await Vectors.batchCreate(myStrings);You can configure the following parameters:
const Vectors = new VectorManager(); // these are the defaults Vectors.maximumBatchSize = 7168 * 4; // maximum size of a batch - for OpenAI, 4 tokens per word, estimated Vectors.maximumParallelRequests = 10; // 10 web requests simultaneously max Vectors.fastQueueTime = 10; // time to wait if no other entries are added Vectors.waitQueueTime = 100; // time to wait to collect entries if 1+ entries are addedSpecial thank you to Scott Gamble who helps run all of the front-of-house work for instant.dev 💜!
| Destination | Link |
|---|---|
| Home | instant.dev |
| GitHub | github.com/instant-dev |
| Discord | discord.gg/puVYgA7ZMh |
| X / instant.dev | x.com/instantdevs |
| X / Keith Horwood | x.com/keithwhor |
| X / Scott Gamble | x.com/threesided |