Writing 13 November 2026
Memoria on three public memory benchmarks: what we measured and what we found
We ran Memoria through OmniMemEval, the open harness that tests memory systems with the same models. It placed fifth on LoCoMo, third on LongMemEval-S and second on BEAM-100K. Its context is small on one benchmark and large on another. Its knowledge graph lowered the LoCoMo score and made no difference on LongMemEval-S.
Memory systems for AI agents publish benchmark numbers, but most of those numbers do not compare. Each vendor picks its own answer model, its own judge model and its own prompt, and those choices move a score by ten points or more. The same system can score 58 in one write-up and 75 in another.
OmniMemEval, an open-source harness from MemTensor, runs many memory systems under the same answer model and the same judge. We added Memoria to it and ran three benchmarks of memory over long conversations. This post gives the method, the results, two results that went against our expectations, and the limits of what the results show.
What we ran
The benchmarks.
- LoCoMo: 10 long conversations between two people, with 1,540 scored questions. Like every published result, we left out the 446 adversarial questions.
- LongMemEval-S: 500 histories between a user and an assistant, each about 115,000 tokens long. The questions ask about facts, preferences, changed facts and dates.
- BEAM-100K: 20 conversations of about 128,000 tokens each, on topics from coding to legal help, with 400 questions in ten areas.
The harness. OmniMemEval at a fixed commit, with the same models as its published table: gpt-4.1-mini writes each answer from the memories that the system returns, and gpt-4o-mini judges each answer against the answer key. We used the harness code unchanged, apart from the few lines that register Memoria as a system.
Memoria, as we run it. Memoria used no language model when it stored data. On LoCoMo it stored each message as a memory. On the other two it stored each round, a user message and the assistant reply, as a memory. Search was semantic, with the top 20 results returned, using the same small embedding model (qwen3-embedding-0.6B) that we run in production. The knowledge graph was off. The store was Postgres; this is the version that moves Memoria’s vector store to Postgres, and we will repeat the runs on the release.
The results

| Benchmark | Memoria | 95% interval | Place | Leader | Mem0 |
|---|---|---|---|---|---|
| LoCoMo | 78.96 | 76.95 to 80.91 | 5 of 16 | MemOS 88.83 | 77.68 |
| LongMemEval-S | 81.40 | 77.80 to 84.60 | 3 of 14 | MemOS 89.20 | 56.00 |
| BEAM-100K | 71.39 | 67.97 to 74.79 | 2 of 13 | LCM-X 71.84 | 70.41 |
Differences of less than about 5 points are inside benchmark noise, so read close scores as a group, not a ranking.
- LoCoMo: only MemOS is more than 5 points ahead. Memoria sits in a group with Cognee, EverOS, Hindsight, LCM-X, Mem0 and Letta. A second pass of the answer and judge stages, on the same retrieved memories, gave 79.16%.
- LongMemEval-S: only MemOS and LCM-X score higher. Mem0 is 25 points behind. We also graded this run with LongMemEval’s own judge and
gpt-4oanswers, the setting in which many vendors publish. Memoria scored 75.4% there. Most of the drop is on questions about the user’s preferences. That pass changes both the answer model and the judge, so we cannot yet say which change caused the drop. - BEAM-100K: the top seven systems are within about 3 points of each other, so this benchmark does not separate them. Memoria is in that group.
No system leads on all three. MemOS leads LoCoMo and LongMemEval-S and is eighth on BEAM-100K.
Where Memoria is weak
Two kinds of question are weak. Questions that combine facts from several sessions are weak on all three benchmarks: LoCoMo multi-hop 72%, LongMemEval-S multi-session 70%, BEAM-100K multi-session reasoning 67 out of 100. Questions about the order of events are weak on LongMemEval-S (temporal reasoning 75%) and very weak on BEAM-100K (event ordering 25 out of 100, the lowest of its ten areas). On LoCoMo, temporal questions are a strength (81%).
We have not analysed these failures yet. A likely cause is that semantic search ranks memories by how similar they are to the question, not by time, so the top 20 can miss some of the events.
Context size: small on one benchmark, large on another

Context size is the amount of text that the answer model reads for each question. It sets the cost and the delay of every answer, so it matters as much as the score in daily use.
On LoCoMo, Memoria sent 1,369 tokens per question, the second smallest of 16 systems. Mem0 sent about 17,000 for a similar score. After the LoCoMo run, we expected this to be Memoria’s strong point.
The other two benchmarks show otherwise. On LongMemEval-S, Memoria sent 8,085 tokens, in the middle of the table. On BEAM-100K it sent 19,084, and only three of 12 other systems sent more. The cause is simple: Memoria returns 20 memories, and on these two benchmarks one memory is a full round with a long assistant reply. Mem0 shows the reverse pattern: about 17,000 tokens on LoCoMo and about 1,000 on the other two.
We have not tuned the number of memories or the memory unit yet. The next step is to measure how much accuracy a smaller context costs.
The graph did not help
Memoria can also build a knowledge graph. A language model extracts the people, places and relations in each memory, and search then gives a score boost to memories that the graph links to the question. We expected the graph to help multi-hop and temporal questions on LoCoMo by at least 3 points, and we wrote that prediction down before the run.
It did the opposite. With the graph on, Memoria scored 75.06% on LoCoMo, 3.9 points lower on the same 1,540 questions. Multi-hop fell 5.3 points and temporal 5.6. The graph made 126 answers worse and 66 better.
The cause is in the ranking. Search adds a fixed boost to every memory that the graph links to a name in the question. In a conversation between two people, almost every question names one of them. The graph then returns about 20 memories that mention that person, and the boost lifts them over memories that match the question better. Mem0’s own paper reported the same direction for multi-hop questions with its graph layer.
We then tested the graph on a 60-question sample of LongMemEval-S, where one user talks to an assistant about many topics. The graph changed the retrieved memories on 39 of the 60 questions, but only a little: on average, 94% of the top 20 stayed the same. On LoCoMo, about half stayed. No answer changed from right to wrong, or from wrong to right. So on this sample the graph did no harm, and no good. This fits the cause above. Some LongMemEval-S questions name a city, a brand or a person, but each name is specific to its question. In LoCoMo, the name in most questions is one of the two speakers, an entity that links to almost every memory.
The graph is not turned on for users today, and all the main results above have the graph off.
Limits of these results
- The answer keys have errors. An independent audit found that about 6.4% of LoCoMo’s answers are wrong (locomo-audit), so no system can score much above 93% on it. This affects every system equally.
- The judge is lenient. It accepts some vague answers. It is the same judge for every system in the table. LongMemEval’s official judge, with
gpt-4oanswers, gave Memoria 6 points less. - Who publishes the table. MemTensor publishes OmniMemEval, and its own product, MemOS, leads two of the three benchmarks. The harness and results are open; we used the numbers for other systems as published and did not re-run them.
- One setting each. We ran one memory unit and one result count per benchmark, and one run each of LongMemEval-S and BEAM-100K.
- Conversations, not teams. All three benchmarks test one person’s or two people’s conversations. Team and organisation memory has many more names and topics, so results there can differ.
What comes next
We will measure accuracy against context size on LongMemEval-S and BEAM-100K, and work on questions about the order of events. We will repeat all three benchmarks on the released Postgres store, and test the graph on team-shaped data before we change how the boost works. When those are in, the results move to a standing benchmarks page, so that the numbers stay current.