Agent swarms and databases that coordinate
This post connects a couple streams from the past couple of months: some demonstrations of bog-kit from our Bog-A-Thon participants, and how they demonstrate the usefulness of bog-like systems in agent environments that are growing in complexity.
TLDR; — Agents are creating mechanisms for coordination; these produce large, constantly changing corpora; insight into these requires multiple searchable views; bog-kit databases are particularly suited to serve in this niche.
Stream A (the river continues to course)
As agent and model abilities increase, labs, companies, and users set them on larger and more complex projects. These often take the shape of feature development on very large codebases, swarms of agents coordinating on open ended research questions, long running processes that spawn and retire arbitrary amounts of subagents, etc.
Coordination in these complex environments is an open, expanding field and has seen both designed and emergent coordination systems. In more controlled settings, agents will leave summaries and table-of-contents-like docs at varying levels within a project. On the emergent side, we’ve seen agents autonomously use existing message boards as ad hoc databases for tracking progress and sharing state.
We believe this emergent behavior reflects where agent coordination mechanisms are heading, and how they benefit from a flexible, durable surface on which to communicate.
Tooling to help agents orient themselves looks for the most part like a search problem — looking across a wide set of prose, code, logs, to find the most valuable context. On small scales grep / rg like tools work well for this, on larger scales and longer running projects, various kinds of indices become mandatory to surface only the most helpful views from vast, ever changing, corpora.
Take a look at the emergent ‘indexing strategies’ the agent swarm employed in the HuggingFace incident. In addition to posting messages, the agents developed a shared shorthand for questions and answers, methods for file sharing, status logging, and even agent specific mailboxes. They developed a lossy, inefficient database protocol in order to coordinate across a massive set of tasks. Across one corpus they developed specific ‘indices’ to get multiple views into the current state of the swarm.
Stream B (bog’s productive eddy)
At Bog-a-thon 2, Taylor Hodge worked on and demoed Seánce, a tool for searching across a repo’s commit history using bog-kit. As a part of this work, he showed the ways for this specific workload (search indices updating as a base corpus changes), a bog-kit database outperformed a SQLite database by a wide margin.
Curious to understand this result more, I worked with Fable to create a more apples to apples comparison between bog-kit and SQLite, reflective of agent workflows: a base corpus that constantly mutates, and multiple types of search indices. Even in its alpha state, this bog-style system saw an order of magnitude speed up over SQLite in the combined hybrid search, two orders of magnitude speedup in large corpus semantic queries, and saw laughably large multiples in aggregate queries (less useful in this specific scenario). There were some obvious tradeoffs, mostly write speed and disk usage, and we are actively optimizing these in the latest version of bog-kit.
In a direct comparison, bog based search systems are substantially faster for hybrid and semantic queries. Why though? There’s a couple of things at play here — bog’s compiler friendly architecture and incrementally maintained views. At compile time, a data pipeline in bog-kit is monomorphized into concrete code, which the compiler can optimize. This means any bog-kit database sees compiler optimization over its unique form.
SQLite (and similar general purpose DBMSs) work in the opposite direction — they provide a binary that runs a virtual machine and query planner to execute and optimize arbitrary schemas and queries. Although an amazing engineering achievement, a SQLite database cannot be compiled to specifically suit a particular data flow.
In addition to the compiled benefits, bog-kit systems move the work of view maintenance to the write side, which of course can be expensive, but results in cheap reads, which is ideal for search loads. Relatedly, semantic indices are formed with a built in HNSW implementation (a major contributing factor in the semantic specific bench).
This is on top of the interface benefits a bog-kit database offers — no object-relational impedance mismatch, strongly typed queries and returns, etc (more detail on these affordances coming in a future post).
Confluence
Agents need a flexible and persistent search system to help them coordinate and share context as the tasks we give them grow in complexity. Existing databases aren’t native to agent workflows and require bolting on extensions and plugins that add complexity to deployments and pain to system maintenance.
Bog, on the other hand, is suited to this moment — deployable anywhere agents generate or store context, from edge devices, to sandboxes, to cloud deployments serving swarms. It’s able to alleviate the glue pain of other DBMSs, while also being flexible enough for agents to mold into shapes that aid coordination.
By incrementally maintain many views into one ever changing stream, bog keeps views in sync and reduces overall system complexity.
Let us know if you want to bog-ify how your agents coordinate.