past.dev’s BEAM #1 claim: what it does and doesn’t show

past.dev says it ranks first on BEAM, a long-term memory benchmark for AI agents, at every size up to 10 million tokens. Every past.dev figure is the company's own, CTN found no independent reproduction, and one model both answers and judges in its harness. Databricks' new NEAREST BY join, a batch vector-search feature that never mentions agent memory, shows the vector index getting easier. The harder part for buyers is which fact is current, what it replaced and who may see it.

Agent memory looks like a search problem. A vector index finds text that resembles a question. That part is mostly solved, and it is cheap to build.

The hard part is a data problem. Which statement is current? What did it replace? Who is allowed to see it? A similarity score answers none of those.

That is the argument past.dev made when it launched on 30 September and again in a press release on 6 October. The San Francisco company sells a memory API for AI agents. It also says it is number one on BEAM, a public benchmark for long-term memory, at every history size from 100,000 to 10 million tokens. This story takes the claim seriously and checks what it does and doesn’t show.

The score is a number past.dev published. CTN found no one else who has reproduced it. Even if it holds, BEAM tests only part of what a buyer needs.

CTN also sets past.dev beside Databricks’ new NEAREST BY SQL join. That pairing is CTN’s framing. Databricks’ blog never mentions agent memory. It is about batch workloads: entity resolution, deduplication and record enrichment. The pairing has a narrow use. Databricks’ post shows the vector index getting faster and more standard. past.dev’s pitch points at where the unsolved work sits.

CTN has not yet asked past.dev, Databricks or the BEAM authors for comment.

What past.dev claims

BEAM is an academic benchmark. The paper, “Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs,” lists Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla, Hamed Zamani and J Ross Mitchell as authors. It went on arXiv on 31 October 2025. The current version is v2, dated 21 February 2026. The paper carries an ICLR 2026 conference footer.

BEAM has 100 conversations and 2,000 validated questions. Conversations run up to 10 million tokens. The questions probe ten abilities, including abstention, contradiction resolution, knowledge update, event ordering and multi-session reasoning.

past.dev reports these answer accuracies from a run on 29 September 2026:

History size	past.dev	Questions
100K tokens	92.08%	400
500K tokens	89.63%	700
1M tokens	90.65%	700
10M tokens	85.03%	200

CTN checked the arithmetic. Weighted by question count, the four splits average 90.02%, which matches the harness README. The drop from 100K to 10M is 7.05 points, as past.dev says.

On its benchmarks page, past.dev lists rivals: Exabase M-1, Hindsight, Honcho and mem0. It says they trail at every size. At 10M tokens, it puts Exabase M-1 next at 68.0%.

Those are the facts as published. Every one of them comes from past.dev.

Why the index is the easy part

Take the example past.dev uses in its launch post. On 3 February a call note records an Acme pilot budget of $32,000. On 28 July an email moves it to $40,000. Both records are true. Both mention the budget.

A similarity search ranks them by wording, not by time. The agent may answer with whichever reads better. past.dev says its API returns $40,000, cites the 28 July email and keeps the February note as the value it replaced.

That is database work. It needs supersession: a rule for when a new fact replaces an old one. It needs validity windows, so you can ask what was true on a given date. It needs entity resolution, so the same person in an email signature and a chat handle is one person. And it needs access control, so a support agent and a sales agent asking the same question get different answers.

past.dev’s methodology page names the same four operations: updates, identity, historical questions and unsupported questions. The launch post says the API takes an audience with each record and a reader identity with each recall. It also says deleting a source record removes the memory derived from it.

CTN has not tested any of that. These are product claims. They are the right things to claim, because they are where agent memory breaks in a company. But a benchmark score is a different kind of evidence from a working access-control model.

In the press release, CEO Mehdi Djabri says: “For two years, the industry has tried to solve memory with search. But search finds what looks similar, not what’s still true.” CTO Alex Ronse says: “Deciding that one fact replaced another, across sources, without leaking a private email, is the hard part.” Those are company statements from a press release. They are not independent comment.

What the claim shows

It shows that past.dev built a harness, ran it on all 2,000 BEAM questions and published per-question records. The repo holds a results.jsonl file for each split and a summary.json with the scores.

CTN opened the four summaries. Each reports zero failed questions and zero failed verdicts. Each names the dataset revision, the recall settings and the models used. The recall call asks for up to 100 results within 8,000 tokens of evidence.

It shows a consistent profile. On past.dev’s own data, six abilities stay above 90% at every size: abstention, event ordering, instruction following, preference following, summarization and temporal reasoning. Knowledge update runs from 85.71% to 92.50%.

And it shows past.dev is willing to publish its weak spots. The launch post says four abilities stay below 90%. It calls them “the work in front of us.”

What the claim doesn’t show

Start with who ran it. The scores are past.dev’s. CTN searched for an independent reproduction and found none. That is a soft hold, not proof that none exists.

Then the harness. The repo, pastdotdev/benchmarks, was created on 28 September and has one commit, dated 29 September. As of 9 October it shows one star and zero forks. That is not a verdict on quality. It does mean the code has had little outside scrutiny so far.

The harness README says its verification script checks scoring arithmetic. In its words, it “does not rerun retrieval, answer generation, or judging.” So verify.py tells you the published scores add up from the saved judge verdicts. It does not tell you the verdicts are right. Rerunning the benchmark needs a past.dev management key and an OpenRouter key. The README says the 10M split takes hours and the dataset is about 1 GiB.

Then the model. The README says one model answers and judges: openai/gpt-5.6-luna through OpenRouter, with reasoning off. CTN confirmed that name in the README and in all four summary.json files, where answerModel and judgeModel are identical. CTN’s concern is simple: a model grading its own answers can share its own blind spots. past.dev’s own methodology page says to use more than one judge and report each, “because judge choice affects scores.” The BEAM results CTN found use one.

That methodology page also lists items a result should publish: baselines, cost, latency, and failed requests counted rather than excluded. The summaries CTN opened carry accuracy and failure counts. They do not carry cost, latency or baseline scores. Those may exist elsewhere. CTN did not find them in the files and pages it checked, so it does not rely on them.

Then the prompts. The answer and judge prompts are ExaBase’s, reproduced unchanged and excluded from the repo’s MIT license. The README says that is so scores compare with other published BEAM results. It also means past.dev’s score depends on a rival’s prompt design.

One more detail matters for a story about “which fact is current.” The README says recall runs at a set time: the end of the conversation or, for knowledge-update questions, the update the dataset cites. The press release footnote says the scores use the prompts “including their guidance from the dataset’s question annotations.” CTN has not judged how much that helps. It is a reminder that a benchmark can hand the system the as-of date a real agent has to work out.

Two sets of “previous best”

The press release and the benchmarks page do not agree on the field.

The press release says the previous best published results were 86.2% at 100K, 80.1% at 500K, 79.1% at 1M and 68.0% at 10M. The benchmarks page says 76.9%, 71.1%, 75.0% and 68.0%, using Exabase M-1 and Hindsight.

Only the 10M figure matches. The release footnote defines “previous best” as the best competing score published as of 2 October 2026, counting only runs on every question with a public, reproducible harness. It does not name the system behind the three higher figures. CTN could not find them on any page it checked, so it does not use them as fact.

The gap matters. If the release’s figures are right, past.dev’s lead at 100K, 500K and 1M is about 5.9, 9.5 and 11.5 points by CTN’s arithmetic, not the 15.2, 18.5 and 15.7 the benchmarks page states. past.dev leads either way on its own numbers. The margin is unclear.

The rival numbers are not one test

past.dev says so itself. Its footnote states that competitor figures are “separately published runs rather than one harness, so models, judges and configurations differ between rows.”

CTN read each source.

Exabase’s post reports 76.9% at 100K, 75.0% at 1M and 68.0% at 10M. It says M-1 used Gemini 3 Flash while “every other system used Gemini 3 Pro.” It publishes no 500K figure. Its own table lists Hindsight at 73.4% at 100K, below the 75.0% Hindsight now shows.

The Hindsight page lists 75.0%, 71.1%, 73.9% and 64.1% for the four sizes. The page text CTN retrieved does not name the answer or judge model for those BEAM runs.

Plastic Labs posted Honcho’s results on 19 December 2025. Its post reports 0.630, 0.649, 0.631 and 0.406 on BEAM’s zero-to-one scale, which past.dev shows as percentages. The post says its runs used gemini-2.5-flash-lite for ingestion and claude-haiku-4-5 at the chat endpoint. Exabase’s post lists Honcho under Gemini 3 Pro. CTN has not resolved the difference.

mem0’s page reports 64.1% at 1M and 48.6% at 10M. It says those come from an updated algorithm dated April 2026. The page was updated on 22 September 2026.

The runs differ in models, judges and dates. The ordering may be right. The gaps are not reliable.

Plastic Labs’ post is blunt about benchmarks in general. It says each benchmark has “dozens of questions with ambiguous answers,” and adds: “These are the perils of synthetic data.”

BEAM is synthetic, and an LLM grades it

The BEAM paper is open about how it was built. An automated pipeline generates the conversations from plans. GPT-4.1 helps draft titles and themes. GPT-4.1-mini selects and writes probing questions. Human annotators review the candidates and validate the final set. The paper reports human ratings of the conversations at 4.53 for coherence, 4.57 for realism and 4.64 for complexity, each on a five-point scale.

Scoring uses nuggets. Annotators break each reference answer into small criteria. An LLM judge scores each at 0, 0.5 or 1. Event ordering uses a rank-correlation measure, Kendall tau-b, after an LLM aligns the events.

None of that is a flaw. It is how the benchmark reaches 10 million tokens at all. It also limits what a score means. A high BEAM number says a system recalled facts in generated conversations as judged by a model. It does not say the system handled your company’s messy email, your permission model or your retention rules.

The paper’s own tables show how much the setup matters. Its best method scored 0.358 on average at 100K, on a zero-to-one scale, across the four models it tested: Qwen2.5-32B, Llama-4-Maverick, Gemini 2.0 Flash and GPT-4.1-nano. The press release says models given the full history scored at most 25.9% at 1M. CTN checked Table 1: the highest full-context average at 1M is 0.259, for Llama-4-Maverick. That matches. But those are the paper’s four models under the paper’s setup. The vendor runs above used other models and, in past.dev’s case, a different judge setup. Scores can move a great deal with the model and judge. That is why a shared harness matters.

The weak spots are the ones that matter most

past.dev’s own ability table is the most useful part of its disclosure.

Contradiction resolution sits between 73.13% and 78.57% at every size. That is the ability closest to “which fact is current.” Knowledge update runs 85.71% to 92.50%. Information extraction falls from 95.00% at 100K to 68.75% at 10M. Multi-session reasoning, which combines facts from separate conversations, drops to 28.17% at 10M.

Two cautions. First, at 10M each ability has 20 questions. Each question can move a category by up to 5 points, by CTN’s arithmetic. A 28.17% score on 20 questions is a signal, not a measurement. Second, a headline of 85.03% is an average. It includes categories at or near 100%. It can sit above a category that fails.

For a buyer, the readable result is not the top line. It is that contradiction handling, the core of the pitch, scored about three in four on past.dev’s own run. past.dev says as much.

Databricks: a different job

Now the pairing. On 5 October, Databricks published a blog post on NEAREST BY, a SQL join for batch vector search. For every row on the left, it finds the k nearest rows on the right, by similarity or distance, exact or approximate.

The post is explicit about its target. It describes batch workloads: entity resolution, deduplication, semantic tagging, classification, record enrichment and batch recommendations. It gives three customer examples without names: a payments company matching more than 100 million daily transactions against 140 million merchant embeddings; a data firm enriching tens of millions of historical records nightly; and a quantitative fund running million-query batches against a 50-million-vector corpus. CTN could not verify those customer figures.

The post never mentions agent memory. CTN found no mention of agents or agent memory in it. Databricks is not claiming to solve the problem past.dev sells.

Three design choices show why the pairing still helps.

First, the contract. NEAREST BY takes APPROX or EXACT. EXACT guarantees the true top-k by exhaustive evaluation. APPROX lets the optimizer use an index. The post says creating or dropping an index can never silently change results, because only queries that say APPROX accept approximation. Databricks’ docs repeat the two modes. That is a data-management habit: say what the answer guarantees.

Second, the storage. The IVF vector index is an ordinary liquid-clustered Delta table. Vectors live in the same lakehouse as the rest of the data, with no separate store to sync. Databricks says that means no separate vector store to sync and no second system to operate.

Third, the numbers. Databricks says it tested on workloads observed from customers, with base tables from 100,000 to 5 billion vectors and up to 10 million queries, “targeting at least 96% recall@K.” It reports a 10-million-record self-join finishing “in minutes” and 1 million queries against 1 billion vectors finishing “in minutes.” These are Databricks’ own benchmarks. CTN has not seen them reproduced. A 96% recall target also means the approximate path may miss up to 4% of true neighbors, by CTN’s reading.

The docs add limits. The page says NEAREST BY applies to Databricks SQL and Databricks Runtime 18 LTS and above. It is not supported on streaming sources. The join is asymmetric: swapping the sides asks a different question. The docs page shows a last-updated date of 11 September 2026. Neither page CTN retrieved carries a preview or general-availability label. GA status is unconfirmed.

NEAREST BY makes the retrieval step a first-class, tunable SQL operation, at batch scale. That is the “easy” half, getting easier and more standard. What it does not do, and does not claim to do, is decide that a July email replaced a February call note, or that one reader may not see another’s thread. Those need a data model on top: timestamps, validity, provenance, audiences. The Databricks post says governance and autoscaling “came with the runtime engine.” It does not say how that applies to agent memory, and CTN will not guess.

What buyers should ask

Use the same questions with any memory vendor, past.dev included.

Reproducibility

Who has rerun the benchmark besides the vendor? Can you run it without the vendor’s key?
Which model answers and which models judge? Is there more than one judge, and are results shown for each?
Are baselines, cost, latency and failure counts published from the same run?
Which dataset revision and answer key were used? Do rival numbers use the same harness?

Access control and provenance

Is permission checked at recall time, per reader, or only at ingest?
Can you see the source record and excerpt behind every returned fact?
When a source record is deleted, what happens to the facts derived from it? How do you verify that?
Can you test a leak? Ask as reader A for something only reader B may see.

Fact versioning

When two sources disagree, what rule decides which one wins? Who can override it?
Is the superseded value kept as history? Can you ask what was true on a past date?
How does the system resolve one person across an email address, a chat handle and a transcript name?
What does it do when it has no answer? Does it abstain?

Then test with your own data. past.dev’s launch post invites that, with free credits for a trial. Use changed facts and unanswerable questions, and check the answers against records you can read yourself.

What this piece is not

This piece is not a review of past.dev, Exabase, Hindsight, Honcho, mem0 or Databricks. CTN has not run any of their products. It is not a ruling on who leads BEAM. It is not an endorsement of any score, and it is not investment or purchasing advice.

It does not claim past.dev’s numbers are wrong. It says they are self-reported and unreproduced, and that BEAM measures part of the job.

It does not say Databricks competes with, or plans to enter, agent memory. The pairing is CTN’s framing, built on one idea: the vector index is the easy part.

It contains no expert comment. CTN has not yet asked past.dev, Databricks or the BEAM authors for comment. If real comment arrives, CTN will add it and mark the update.

Disclosure: Lone Cowry / Cowry Tech News has no commercial relationship with past.dev, Databricks, the BEAM authors, Exabase, Vectorize, Plastic Labs or mem0.

Sources

Tavakoli, M., Salemi, A., Ye, C., Abdalla, M., Zamani, H., Mitchell, J. R., “Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs,” arXiv:2510.27246 (v1 31 Oct 2025; v2 21 Feb 2026): https://arxiv.org/abs/2510.27246
BEAM repository (mohammadtavakoli78/BEAM): https://github.com/mohammadtavakoli78/BEAM
past.dev, “Benchmarks”: https://past.dev/benchmarks
past.dev, “Introducing past.dev,” 30 September 2026: https://past.dev/blog/introducing-past-dev
past.dev, “How to benchmark agent memory”: https://past.dev/benchmarks/methodology
past.dev, “past.dev Introduces Frontier Memory for AI Agents: #1 on BEAM, the Largest Public Memory Benchmark,” PR Newswire, 6 October 2026: https://www.prnewswire.com/news-releases/pastdev-introduces-frontier-memory-for-ai-agents-1-on-beam-the-largest-public-memory-benchmark-302899770.html
past.dev BEAM harness and results (pastdotdev/benchmarks, beam/, including README, NOTICE.md and results/*/summary.json): https://github.com/pastdotdev/benchmarks/tree/main/beam
Exabase, “M-1 achieves state-of-the-art on BEAM at every scale, with a smaller model and fewer tokens”: https://exabase.io/blog/exabase-m1-achieves-state-of-the-art-on-beam-benchmark
Hindsight (Vectorize) benchmarks: https://benchmarks.hindsight.vectorize.io/
Plastic Labs, “Benchmarking Honcho,” 19 December 2025: https://plasticlabs.ai/blog/research/Benchmarking-Honcho
Mem0, “AI Memory Benchmarks: LoCoMo vs. LongMemEval vs. BEAM” (updated 22 September 2026): https://mem0.ai/blog/ai-memory-benchmarks-in-2026
Databricks, “NEAREST BY Join: Scaling Vector Search in Databricks Runtime,” 5 October 2026: https://www.databricks.com/blog/nearest-join-scaling-vector-search-databricks-runtime
Databricks documentation, “NEAREST BY clause”: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-select-nearest-by

Method: CTN retrieved each primary on 9 October 2026 and checked figures against the page text and, for the past.dev harness, against the repository files and GitHub’s public metadata (stars, forks, commits). The press-release URL supplied in the assignment returned a 404; the release was read at the PR Newswire URL listed above. Figures CTN could not trace are labeled in the text and are not used as fact.