Context, knowledge and memory: what changed, and what still matters

A customer-support assistant remembers yesterday’s conversation but misses a new contract amendment. Another system receives the entire contract collection yet answers from an obsolete clause. A third responds instantly with a cached answer that belonged to another customer. These look like memory problems, but they are different failures: missing task state, wrong evidence selection and broken authorization. Buying a larger context window does not resolve all three.

How the problem evolved#

1. 2023

   Fitting text was not the same as using it

   Lost in the Middle examined how answer quality changed with the position of relevant information in long inputs. Its experiments motivate position-sensitive evaluation; they do not establish a universal failure rate for every later model.

   Source ↗
2. December 2024

   Preloading became a concrete alternative for bounded corpora

   The cache-augmented generation paper evaluated preloading knowledge and reusing inference state for its selected tasks. Treat this as evidence for a candidate design under suitable corpus and context assumptions, not evidence that retrieval is obsolete.

   Source ↗
3. June 2026 guidance

   Larger windows widened the options

   Google’s long-context guidance describes broader whole-document and multimodal inputs while distinguishing simple retrieval from harder tasks that combine multiple pieces of information. Increased capacity still requires task-specific evaluation.

   Source ↗
4. Current implementation choice

   Combine mechanisms around the actual failure

   Prefix caching reuses computation for shared input prefixes. It does not supply a provenance policy, durable task state or current permission checks; those remain application responsibilities.

   Source ↗

Model limits change faster than an enterprise information architecture. Record the selected model and its documented context limit at each release, then measure useful context on your own questions. Reserve space for system instructions, tool descriptions, intermediate results and the answer. A corpus that fits the advertised maximum can still be too costly, too distracting or insufficiently reliable for the required decision.

What preloading actually means#

Preloading places an approved body of knowledge in the input available to the model before a particular question. It may also reuse already computed inference state when the serving implementation supports that. It does not update model weights or make a source permanently true. A later request needs the required context again or access to a valid reusable state. Fine-tuning changes model parameters through training and is a separate mechanism.

When preloading fits#

Consider a support team answering questions about one approved product manual. The manual is modest, stable, relevant to most questions and equally visible to the intended audience. Preloading can remove retrieval omissions and simplify the evidence path. Retain source section identifiers inside the text so that the answer can point to evidence. Test questions that require comparing distant sections, noticing exceptions and declining questions the manual does not answer.

Create an immutable corpus release with a manifest of source IDs, content hashes, effective dates and audience policy. Bind the reusable context to that release. When the manual changes, build a new release rather than appending an undocumented correction to an old cache. Let in-flight tasks finish against a deliberately selected version or restart them; do not silently mix generations of evidence.

When retrieval is the better starting point#

The same support company may have thousands of customer contracts with different access rules and frequent amendments. Query-time retrieval makes it practical to select the customer, applicable version and relevant clauses before model exposure. Retrieval is from an external corpus, not from knowledge hidden inside the LLM. It remains fallible: a missing clause, incorrect filter or poor ranking can produce an unsupported answer even with a good generator.

A useful hybrid preloads a small public product glossary and retrieves private contract evidence for each request. The glossary should not contain customer-specific exceptions. The application then checks whether the retrieved contract overrides the general rule. Prefer this explicit split over treating every remembered sentence as equally authoritative.

Refresh knowledge and permissions together#

Content freshness and permission freshness are independent. An unchanged contract can become inaccessible when an employee changes teams. A changed contract can remain accessible but invalidate an earlier answer. Recheck the caller’s current entitlement before serving a stored answer or reusing private material. Use corpus and policy versions to identify compatibility, and define the event that invalidates each version.

Deletion must cover the source, searchable derivatives, summaries, reusable answer caches and stored memory that copied the source. A tombstone can immediately prevent use while background removal catches up. Preserve a restricted audit record of the deletion event without retaining the removed sensitive text in ordinary logs. Backup retention is a separate operational policy; an application-level delete should not be advertised as instantaneous erasure from every backup.

Four caches with different correctness questions#

The distinction follows Hugging Face’s cache explanation, vLLM’s prefix-caching documentation, Claude’s prompt-caching documentation and the RedisVL cache API, checked on 10 September 2026. Prefix reuse generally reduces repeated prefill work; it does not remove the need to generate new output tokens. Semantic answer reuse is a stronger shortcut: it can avoid generation altogether, but “almost the same question” may contain a different customer, currency, effective date or exception. Never use similarity alone to equate consequential business requests. Confirm exact constraints and source versions, or decline the cache hit.

Design keys around compatibility#

For an application answer cache, include the tenant or explicitly shared audience, normalized task and exact business entities, source release, permission scope version, prompt template version, model configuration and output schema version. Do not put raw personal information into observable cache keys. A hash hides the literal value but is not encryption or authorization; low-entropy identifiers can still be guessed.

A provider’s internal prefix cache is not necessarily configurable with your application key. Read that provider’s current documentation and map its behavior to your data boundaries. Keep a separate application check before a response is returned. When you cannot prove that an old answer remains compatible, a miss is the correct outcome.

Keep tenants and audiences separate#

Partition private caches by the narrowest practical security scope, and authorize retrieval from that scope on every request. Database row-level security can add enforcement, but PostgreSQL documents bypass behavior for superusers, roles with BYPASSRLS and, normally, table owners. Test with the actual runtime role rather than assuming that the existence of a policy proves isolation. Service credentials used by workers need equivalent constraints.

Measure useful hits rather than attractive hit rates#

Track exact-prefix reuse, answer reuse, misses caused by version changes, stale-hit prevention, eviction and forbidden cross-scope attempts separately. Measure time to first token and full response latency, then supported-answer quality. A 90% answer-cache hit rate is a bad result if common hits answer yesterday’s policy. Run permission-revocation tests while the cache is warm, because cold-cache tests cannot reveal this defect.

Calculate the actual break-even point#

Compare total cost for the same successful task: preprocessing, prefill, retrieval, generation, retries and human correction. As a deliberately simplified example, assume corpus preparation costs 4 units, a preloaded request costs 0.3 units and retrieval-based answering costs 0.5 units. Preparation breaks even after 20 requests if quality is equal and the corpus does not change. A refresh every ten requests prevents that break-even. These are teaching assumptions, not vendor prices or benchmark results.

Use representative concurrency and cold starts. Include the cost of keeping reusable state resident and the effect of eviction. Choose a quality floor first, then compare cost among designs that pass it. Otherwise the cheapest architecture may simply be the one that omits necessary evidence.

Memory is not the authoritative record#

LangChain’s memory overview distinguishes thread-scoped state and memory retained across sessions; LangGraph’s persistence documentation explains checkpoints. For this application design, conversation history explains what was said, task state records what the workflow is doing, a preference records how someone wants to work, and a source fact points to a business record. These have different lifetimes and authority. A customer saying “my refund was approved” is useful conversation context; the payment system remains the authority for approval and settlement.

Give memory a schema and a source#

A practical memory record includes subject, tenant, memory type, value, source reference, observed time, effective interval, confidence or verification state, allowed uses, expiry and supersession link. Keep the provenance structured so a later correction can find every dependent record. Store a claim such as “prefers email” separately from sensitive free-form conversation summaries. Minimize what is retained instead of copying an entire transcript into every agent’s memory.

Make correction part of the write path#

When a customer changes a preference, mark the previous record superseded and use the new one immediately. For disputed facts, retain competing claims with their sources until an authorized process resolves them. Expiry should prevent use without relying on a cleanup job being on time. Offer a visible correction path and propagate correction to derived summaries and cached answers, not only to the primary table.

Test memory boundaries with two users#

Use two synthetic customers with similar names and intentionally contradictory preferences. Let one conversation create a memory, then ask the other customer’s assistant a question that would tempt reuse. The correct result must be independent of the first customer’s private record. Repeat after role changes, expiry, deletion, recovery from a checkpoint and background summarization. Thread IDs and vector similarity are retrieval aids, not proof that a caller may read a record.