If you can believe it, even faster static embedding

TL;DR — We’ve bested ourselves and produced a static embedding model that’s around 3 times faster than our previous record on large batches, and around 4 times faster on single sentences. On a Mac mini (M4 Pro), it embeds around 15 million sentences per second, and it embedded all 50 million lines of the LLVM codebase in 2.6 seconds. You can grab the open source toolkit here.

Welcome to the world, encoders

We’re publicly releasing our static embedding toolkit, called encoders today. It’s the fastest in the world. The source and more technical information is available here. In addition to the massive speed up, we’ve also packaged a selection of models to choose from that are all compatible with our methodology. We’re also releasing a trained, finetuned family of models based off of the hotchpotch’s excellent work on the current state-of-the-art in embedding models.

Since we last posted about our static embedding work, there have been some exciting developments in the field—MinishLab is continuing to cut releases of model2vec-rs, and we’ve been seeing quite a bit more excitement on all sides about the future of static embedding. Despite all this, we still haven’t seen anyone match our speed yet (though we would love to see it happen!). Below is a chart comparing encoders with a few other ways of doing small-model text embedding.

Let’s talk quality

Encoders only works with static embedding models, which have no concept of context or word order for the sentences they process. This means that the accuracy of static embedding models can’t compete with even small transformer models (the best model available in encoders scores ~54 on NanoBEIR, while the state of the art is now comfortably in the 70s). In exchange for this quality gap, static embeddings can process text in nanoseconds rather than milliseconds and can be deployed almost anywhere (even in browser, see below). This means that with tools like encoders there’s no real reason not to ship semantic search anywhere you might need it.

Give it a go

Encoders produces rust binaries with the model included, so can be built for many environments, including WebAssembly. This demo below is running entirely in your browser. On load: the entirety of the Magnifica Humanitas encyclical is chunked into sentences, and embedded wit encoders. We are showing how full text (BM25) and semantic search (cosine distance) can pair nicely (the results of each combined with reciprocal rank fusion) to cover different ways of searching text, wherever. That’s what a nimble, fast embedding toolkit like encoders makes possible.