language 🤝 money 🤝 search
For most of computing history, language has been a cheap substrate: chunks of text shipped to different databases, pulled apart, combined, indexed. Now with the rise of LLMs, language has become discretely financialized. Not just priced, it’s a metered material that’s counted, budgeted and optimized against. We spend legible amounts with every inference call. Every prompt, piece of context, tool-call loop has a measurable size, performance impact, and sometimes price, if a platformed call is being made. Even if you have a subscription that covers your inference needs, the underlying economics are still measured in token costs. And if you are running local models, hardware and software decisions are downstream of the token performance you are after.
Few developers have an economic intuition for this shift, and the current set of tooling incentivizes building insight through superfluous token usage. Most of us and agents greedily add as much context as possible to any particular inference session because ‘it just works’. But as the compute crunch continues, prices will adjust, and concern for token efficiency will grow, creating incentives for a new kind of software.
That software will be built on this premise: not all tokens are equally valuable. The language we provide to an LLM influences its response, so obviously there is some subset of all tokens that when included, lead to better responses from a model. As LLMs are increasingly deployed, and efficiency becomes more integral to how these systems operate, focus will shift to surfacing the most valuable tokens for steering inference in a desired direction. At the moment, the pattern of uploading text chunks to models for summarization and compaction is grossly inefficient, and leads to wasteful round trips that burn even more tokens. We need new ways to determine which tokens are worth spending.
This is a search problem, and we should treat it like one.
Given any corpus (summaries, conversation history, docs, local files etc), a system should locate the smallest, most valuable set of tokens likely to improve the next inference call. Search tooling that condenses the valuable linkages between text chunks into performant indices already exists, and is a very valuable slice of the tech industry. Unfortunately, it often comes with a large runtime footprint (from dependencies to the management required), and can’t be deployed to arbitrary edge locations. The efficiency and power of these tools needs to be brought to where most of the corpus lives at the moment: locally, at the edge.
What we do have that’s edge friendly is rudimentary: grep, markdown files filled with summaries, sqlite with remote embeddings, etc. These tools encourage token bloat and lack the ability to richly query across abstract representations of importance (e.g. semantic distance, recency, provenance), which is key for surfacing the minimum valuable token set for any situation. And while these tools are functional though wasteful at the moment, they do not scale to a world that involves diverse forms of inference at every decision point.
The gap to fill is tooling that combines the deployability of SQLite with the abstract indexing and performance of a tuned search system. At Flower we think this is possible and are rapidly approaching a release of some basic building blocks for meeting this need.
If this way of thinking about language, inference and infrastructure resonates, we’d love to talk.