Databricks (5 Oct engineering blog) describes NEAREST BY, a SQL join that returns the k nearest rows by vector distance for every query row, with an optional IVF index stored as an ordinary Delta table, so embeddings stay in the lakehouse with no separate vector store to sync.
Sources: Databricks: NEAREST BY join, scaling vector search (5 Oct) · Note: This is an engineering-blog description of the design. CTN has not verified any customer or benchmark claims, and availability or general-availability status is not confirmed.
Read the full story: Agent memory is a data problem. past.dev’s BEAM claim only partly tests it. (Cole Harlan)