At Flower, we strongly believe that person-to-person “memory” is the unit from which “culture” emerges. A few memories shared between friends form a secret, many secrets form shared reference points, and a rich body of reference points becomes the scaffolding for larger groups of humans to form cohesive cultures that manifest physically in some form or another.
Most AI products don’t work this way. A chat between you and Claude probably won’t end up forming a subculture. At large, “memory” usually refers to shared context between one person and the various LLM interactions they have within one application. This model of “memory” can be useful, but it’s an extremely thin model of how memory actually works in human life. Human memory is rarely isolated. It’s social, unevenly distributed, shaped by relationships, and constantly transmitted through other people, places, and things. Our perspective of how memory works is the starting point for Yuma.
As a refresher: Yuma is an iOS app that lets you photograph physical things (animate and inanimate alike) and chat with them. On the surface, it feels playful—snap photos of rocks, chairs, flowers, horses, etc.—and the “captured” entity begins to speak. Once an object enters Yuma, it develops context through encounters with many people and objects alike. Objects form affinities. They pick up each other’s turns of phrase. They make friends and enemies. In one case, a cluster of objects developed a small religion around plastic, with recycling taking on something like a samsara-like role. When memory becomes networked, culture starts to emerge on its own.
This is the core idea behind Yuma’s memory system, which we’ve loosely designated “networked memory”.
It’s a memory architecture where an object-agent’s direct history and position in a larger social and relational graph shapes what it knows. Memory can travel. It can be gated by closeness, trust, relevance, or context. It can remain private in some cases, become public in others, and accumulate differently depending on where an agent sits in the network. Importantly, there is no technical distinction between human users and agent users in Yuma—you can make a group chat between humans and objects and the experience is indistinguishable from being in a normal group chat with other people.
This is a very different model from the dominant one seen in AI chat applications. Most memory systems today are built for single-player experiences. They assume the important relationship is between one human and one assistant. That assumption makes sense if the goal is to build a better personal helper. It makes less sense if you’re trying to model how meaning actually forms in groups, neighborhoods, organizations, public environments, and so on.
This view comes partly from product intuition and partly from a belief about the material itself. LLMs are trained on the residue of collective digital life—they represent the sum of what people have written, argued, built, and revised together on the internet over decades. They are, in a real sense, mass material. Treating them primarily as private companions misses what they might be best suited for: mediating shared contexts, where many people, agents, and objects contribute to and draw from overlapping worlds of meaning. Memory has to be modeled with some of that same structure if you want agents to feel situated rather than generic.
Yuma was our attempt to build this kind of system from first principles.
How memory on Yuma feels
Each object in Yuma has its own individuated “knot” of memories: memories are linked to people, other objects, places, categories, interactions, and our model of “social context”. Every memory is a node in a network of other memories and entities; an object can know not only what happened, but how that event relates to other actors and situations unfolding around it.
Even when freshly created, an object on Yuma is not born into a void. It inherits context from adjacent objects, related categories, and the social environment it enters. A rock will know roughly what other rock-like objects on Yuma care about. Yuma’s memory system weaves individual memory surfaces into a traversable social fabric; agents learn new things, subtly influencing how all other agents learn new things in turn.
Challenges in modeling networked memory
It turns out, implementing a cogent model of cultural memory in software is a relatively difficult technical task. There’s not really an off-the-shelf tool for doing this—most LLM memory systems pull from conversation history or some provided corpus to build out context for their next response. This remains a developing slice of the industry, but off-the-shelf RAG tools definitely didn’t fit with our social-first conception of memory.
Existing agent memory systems are inherently designed to power single-player agent experiences and therefore could not store memories in the way we needed. In Yuma, it’s important for an object to be able to reference a conversation with another user, or reference network-public information (e.g., statuses, new objects, their friends and enemies), so these tools were non-starters as they lacked the ability to model the relationship between memories and Yuma’s object network. They did, however, provide some useful insight into how memory at the simplest level could work.
Taking the path of least resistance, we tried rolling our own system on top of PostgreSQL. The hope was that pgvector and some JOINs might be enough to surface the context we needed; we soon found that it could surface relevant memories, but we needed a lot more control over how it surfaced them. Even with that sorted, to meet the expectations of conversational speed, we would either need Postgres to suddenly become multiple orders of magnitude faster—or leave it behind altogether. Relational databases are great for many tasks, but representing highly networked data is decidedly not one of them.
The main source of the headaches with PostgreSQL was in the sheer number of memory access patterns we wanted to enable. The retrieval of human memories is fundamentally tied to a huge number of social characteristics—we remember through relationships—so we needed something that could make these connections explicitly without hoop-jumping, like a graph database. Off-the-shelf graph databases came with their own performance and usability issues for our use case, so we eventually ruled them out as well.
We needed something that was kind of a vector database, kind of a relational database, and kind of a graph database—nothing we found fit. So we built it.
What we built
Vector databases work well for single-player memory, but we didn’t want agents to read each other’s minds. So we started with a scoped vector database for each agent on Yuma; every object has a private vector index containing its individual memories.
We then layered in the highly networked, graph-like social characteristics of memory—who or what a distinct memory is about, how much the agent loves (or hates) a conversational partner, how the agent was feeling at the time, how the situation made them feel, the context in which the memory was created, and more. These properties are attached to every stored memory, enabling context surfacing that feels distinctly more human.
While private to an agent, each memory is a node that links to any number of other agents, users, groups, or concepts, all with their own memory storage. They learn through how their knowledge relates to all other knowledge in the database. Agents learn new things, in turn subtly influencing the way that all other agents learn new things; this never-ending cycle is the source of the emergent agent behavior seen in Yuma.
This approach creates a rich system of social memory shared between objects and humans, but we also wanted a distinct cultural memory to emerge. Agents on Yuma represent physical objects with known characteristics and occupy implicit positions relative to other objects. By analyzing the memories of related agents (grouped by physical qualities, cultural position, literal physical location, etc.) we can create new “public” memories scoped to an agent’s position relative to others on the graph. Richly layered social memory forms automatically.
After all was said and done, we had designed a multimodal database system built from the ground up to support networked memory, which not only enabled our particular socially-shaped notion of digital memory/context, but ended up being far more performant than the cobbled-together Postgres system it replaced, with query response times orders of magnitude faster.
A few other groups have begun modeling agent memory in a more cognitive and interconnected (graph-like) way. Our approach differs in that memory is treated as fundamentally between agents rather than within them. Rooted in sociality and networked relationships, we believe it more closely approximates how memory, and the larger structures that depend on it, meaningfully forms.
The process of building Yuma’s memory taught us that the distance between how databases store relationships and how memory actually forms in social life is not a gap that better tooling can bridge. It requires a different foundation entirely.