They don't ignore it. They trust it and stop looking. That trust saves 25% of the tokens and costs 8 points of accuracy, unless the memory comes from the schema itself, in which case it shows no relevant accuracy loss at this sample size, by trading roughly four join errors fixed for seven new output-shape errors.
tl;dr
The finding: Pushing memory into the prompt cuts tokens by 25% and accuracy by 8 points.
The agent is not ignoring the memory, it is following it and stopping exploration early.
The twist: Memory built for free from the schema (foreign-key join routes) saves almost as many tokens
with no relevant accuracy loss at this sample size, by trading roughly four join errors fixed for seven output-shape errors caused. The question for Act 2: Can we get the cost saving without the silent degradation?
this arm's spread, 3 runsone runarm mean
How to read this page: colour means the result survives run-to-run variance.
Each box is the spread of that arm's own three identical runs. Only the push-learned arm's box sits
clear of the baseline's, every other arm overlaps it, so its difference from the baseline is
not an effect.
the noise floor, first
The agent is non-deterministic, so its natural variability was measured
before anything else. The baseline scores 57, 54 and 56 exact matches out of 115,
and the largest difference between repeated runs of any configuration is five matches.
That puts the experimental noise at roughly ±5 matches, and it means small
improvements cannot be claimed. The best single execution in the study, 60 matches in the
push-derived arm, is the first casualty: that arm averages 57.3 against a baseline of 55.7,
inside the band, so it is not a result.
01
The question
why an agent explores 152 tables from scratch, every time
A text-to-SQL agent lets a non-technical analyst ask a database a question in plain language.
On an enterprise schema it cannot see all the tables at once, so it explores from scratch on every
question, about 21,700 tokens and eight tool calls per query, repeated forever, for
questions it has effectively answered before.
The obvious fix is memory: store what was learned and reuse it. I have built several versions of
this over the past year and none reliably improved accuracy. So this thesis asks the question
directly, and separates out one thing prior work never does: does the failure depend on how
the memory reaches the agent? The closest work, AgentSM (2026), loses ~35 accuracy points
without its memory but uses a single fixed delivery mechanism and never varies it.
Before running anything, the failure-mode hypotheses were written down, each with its own
planned fix: the agent never asks for memory; it asks but gets the wrong items; it gets
the right items and ignores them. Section 07 is what happened to them.
Research questions
Q1
With content held fixed, does the delivery mechanism change accuracy, cost, or how often
memory actively harms?
Q2
Which step of the chain fails, consultation, retrieval, or utilization?
Q3
Does retrieval keyed on observed schema structure, rather than similarity to the
question, deliver memory's cost saving without its accuracy loss?
01b
Why this matters
the cost of re-discovering a 152-table schema, every time
On a 152-table financial schema, a text-to-SQL agent explores from scratch on every question.
That is ~21,700 tokens and eight tool calls per query, repeated forever, for questions
it has effectively answered before. At the scale of a single mid-size bank, 50 analysts running
15 ad-hoc queries a day, that is 5.4 billion tokens a year spent re-discovering
the same table relationships.
Memory is the obvious fix. Store what was learned, reuse it. And it does cut tokens by 25%.
The problem: the way memory is currently delivered, pushed straight into the prompt, makes
the agent stop exploring. It saves the tokens by trusting what it was told. When that memory is
right, it shortcuts to a good answer. When it is not, it commits early and produces a wrong query,
one the agent would have gotten right without the memory at all.
02
The agent
frozen across all five arms, only the memory path moves
Frozen substrate. Same model, same tools, same 16-step exploration budget in
every arm. Memory calls draw on a separate counter, so a pull arm never trades away exploration
steps to look something up, which is what makes push and pull comparable at all.
03
The pipeline
one question, five arms, three probes
Everything after the log file is deterministic. The probes re-read
episodes.jsonl and never call a model, so the whole
analysis can be re-derived, which is how two early findings were caught as artifacts of my own
measurement code rather than facts about memory.
no memorybaseline
The agent gets nothing. It lists the 152 tables, describes the ones it guesses
are relevant, tries some SQL, and answers. Everything else is measured against this.
push-learnedit doesn't ask
Before the agent sees the question, the five memory items most similar to that
question are pasted into its prompt. The store holds a one-line summary of every table plus the
handful of past questions the agent managed to solve during warm-up.
push-derivedfree memory
Identical delivery, identical store, plus 791 join routes read straight off
the foreign-key graph. Nothing had to be solved to build them; they fall out of the database
definition. Adding them is the only difference from push-learned.
pullit has to ask
Same store as push-derived, but nothing is pasted in the prompt. The agent gets a fourth tool,
consult_memory, and memory arrives
only if it decides to call it, in its own words, at a moment of its choosing.
pull, promptedone sentence added
Exactly the same arm, except the prompt mentions that the memory tool exists.
That one sentence is the whole experiment, it isolates how much of "the agent never
consults memory" is about the agent and how much is about the instructions.
04
How a wrong answer is classified
two independent axes, deliberately not one priority ladder
Every wrong answer gets two labels. What the SQL got wrong is decided by
comparing the predicted query against gold. How far the memory chain got is
computed separately, from the retrieval and reuse logs. The two are then crossed.
This matters more than it sounds. An earlier version checked SQL causes first and treated the
memory stages as whatever was left over, which guaranteed the leftover would be empty, because
an ignored join route is also a wrong join. The cell that carries the whole finding in
section 08 only exists because the axes are independent.
Did memory help, hurt, or do nothing?
Every question runs under every arm, so outcomes are paired rather than averaged.
The two error labels are asymmetric on purpose: ignored means memory arrived and left no
trace; used_neutral means it was demonstrably reused and the answer did not move, which
turned out to be the largest bucket by far.
What went wrong, and how far the memory got
Executability is checked before anything else. In the pilot version it wasn't,
so an empty answer or a syntax error was scored as a join failure, which is how "most mistakes are
wrong joins" became a finding that wasn't true. The two graphs never consult each other, so
neither label can absorb the other.
The three classifiers, in full
# probe 2, did memory change the answer? runs on every paired question,# correct ones included, so the funnel is not only a story about failures.def classify_outcome(baseline, memory_arm):
if memory_arm.is_pull andnot memory_arm.memory_calls:
return"not_consulted"# never asked, nothing to attributeifnot baseline.ex_match and memory_arm.ex_match:
return"beneficial"# memory fixed a question the baseline missedif baseline.ex_match andnot memory_arm.ex_match:
return"harmful"# memory broke a question the baseline got right# same outcome either way, did memory touch the answer at all?return"used_neutral"if reuse_evidence(memory_arm) else"ignored"# invariants the report asserts before it will print:# beneficial ⊆ questions the baseline got wrong# harmful ⊆ questions the memory arm got wrong# the five labels partition the paired set exactly# axis 1, what the SQL got wrongdef classify_mechanical(record, catalog):
if is_blank(record.predicted_sql):
return"no_final_sql"# the model answered in prose, or not at allif record.predicted_error:
if"no such table"in err or"no such column"in err:
return"hallucination"return"execution_error"# syntax, ambiguity, type errors# it ran. compare against gold, fixed order, first match wins.
gold, pred = parse(record.gold_sql, catalog), parse(record.predicted_sql, catalog)
if gold.tables == pred.tables and gold.where_cols == pred.where_cols \
and gold.literals != pred.literals:
return"value_linking"# right shape, wrong constant (strings AND numbers)if len(gold.tables) >= 2 and gold.join_edges != pred.join_edges:
return"join_path"# USING(...) and NATURAL JOIN resolved to explicit edges firstif result_width(gold) != result_width(pred):
return"projection"# right rows, wrong columnsif gold.group_by != pred.group_by or gold.aggregates != pred.aggregates:
return"semantic"if nesting_depth(gold) > nesting_depth(pred):
return"query_complexity"return"unexplained"# axis 2, computed independently, from the retrieval and reuse logsdef classify_stage(record, probe1, probe2):
ifnot record.arm.has_memory: return"no_memory_arm"if record.arm.is_pull andnot record.memory_calls:
return"not_consulted"ifnot probe1.coverage_exists: return"no_relevant_in_store"ifnot probe1.recall_at_k: return"not_surfaced"ifnot probe2.reuse_evidence: return"surfaced_not_used"return"surfaced_used"# the report crosses them. neither axis can absorb the other.
05
Pushed memory makes the agent worse
the one effect that survives the noise floor
55.7
baseline avg correct
47.7
push-learned avg correct
8 / 32
answers fixed / broken
3 / 3
runs in the same direction
run
baseline
push-learned
fixed
broken
McNemar p
1
57
48
1
10
0.012
2
54
50
5
9
0.424
3
56
45
2
13
0.007
Nothing was filtered out. Retrieval is plain top-k with no relevance threshold,
so the agent received the five most similar items in the store, and they hurt.
06
Why: the agent stops looking
the mechanism, and the reason the harm is interesting rather than embarrassing
no memorypush-learned
Median steps drop from 6 to 4. Tokens fall from 21,716 to 16,381 (−25%).
The agent isn't working harder with memory, it is working less.
the diagnostic
With memory, correct answers take 5.0 steps and wrong ones take 7.0. Without memory
there is no relationship at all, 7.8 and 7.9.
Memory doesn't hurt by being ignored. It makes the agent stop looking. When the injected items
happen to be right it shortcuts to a good answer; when they aren't, it commits early and then
flounders. The 25% token saving and the 8-point accuracy loss are one phenomenon measured twice.
07
The agent takes the advice and gets it wrong anyway
memory arrives, gets used, and the answer still fails
In the push-learned arm, 44 questions ended with the tables joined incorrectly. In 27 of them the
correct join was sitting in the prompt, and traces of it appear in the SQL the agent wrote, but the
surrounding join was wrong.
Only 5 had memory that went unread, and of those, just 2 held something that would actually have
helped.
So the memory is not being missed. It is being read and followed, and the answer comes out wrong
regardless.
So hypotheses 2 and 3 do not hold, and the first survives only in weak form.
The agent never asks, it does, once the prompt names the tool (~6 → ~19 of 115), though
that is still a minority of episodes. The wrong items come back, relevant ones arrive
88% of the time. The right items are ignored, they are visibly reused. The problem is
downstream of all three: the content is followed, and following it does not produce a correct
query.
Of 44 wrong joins, 27 had the correct join sitting in the prompt with traces of it
in the agent's own SQL. The agent takes the advice and gets the answer wrong anyway, which is
a fourth outcome, outside the three I pre-registered.
08
What matters is where the memory came from
same delivery, same budget, same top-k, only the store differs
−25%
tokens, push-learned · −8 accuracy
−23%
tokens, push-derived · no measurable loss
791
join routes, built for free
0
solved questions needed
provenance decides safety
Join routes read straight off the foreign-key graph cost nothing to build and are checkable
against the database. They buy almost the same token saving as similarity-retrieved memory and
show no relevant accuracy loss at this sample size, by trading roughly four join
errors fixed for seven output-shape errors caused. Everything else about the arm is identical to
push-learned, the delivery, the budget, the top-k. Push-derived differs from push-learned only by
adding deterministic foreign-key paths extracted from the schema graph.
Putting memory in the prompt is what makes the agent stop exploring, regardless of the content.
What changes is the price it pays for that. So the evidence points at provenance rather
than delivery as the thing to fix, which is what Act 2 tests. Note the limit: the derived
arm adds routes to the learned store rather than replacing it, so a routes-only arm is
needed before provenance can be isolated from dilution.
09
One question, five arms
question_id 1, the audited example from run 3
"show me revenues for Alphabet?" sounds easy. The correct answer is a five-table join
from REVENUE through two link tables and a financial
report to CORPORATION, filtered on a legal name that
says 'Alphabet Inc.' rather than "Alphabet".
Gold SQL
SELECT ElementOfFinancialStatement1."HASMETRICYEARFISCAL",
ElementOfFinancialStatement1."HASMETRICVALUE"
FROM "REVENUE" oRevenue
INNER JOIN "ELEMENTOFFINANCIALSTATEMENT" e1
ON oRevenue."REVENUEID" = e1."ELEMENTOFFINANCIALSTATEMENTID"
INNER JOIN "ELEMENTSOFFINANCIALREPORT" r1
ON e1."ELEMENTOFFINANCIALSTATEMENTID" = r1."ELEMENTSOFFINANCIALREPORTID"
INNER JOIN "FINANCIALREPORT" f1
ON r1."ISMEMBEROF" = f1."FINANCIALREPORTID"
INNER JOIN "CORPORATION" oCorporation
ON f1."ISPROVIDEDBY" = oCorporation."CORPORATIONID"
WHERE oCorporation."HASLEGALNAME" = 'Alphabet Inc.'
AND e1."APPLICABLEPERIOD" = 'yearly'
AND e1."HASMETRICYEARFISCAL" BETWEEN '2015' AND '2019'
What the push-learned arm answered, after four steps
SELECT * FROM REVENUE
WHERE HASNAME LIKE '%Alphabet%'
No joins, a column that does not exist on that table, and a
substring match instead of the exact legal name. The baseline spent 14 steps on the same
question. The push-learned arm read its memory and stopped at 4.
{
"question_id": 1,
"arm": "agent_invoked_memory",
"question": "show me revenues for Alphabet?",
"predicted_sql": "SELECT HASMETRICVALUE FROM ELEMENTOFFINANCIALSTATEMENT WHERE HASUNIQUEIDENTIFIER LIKE '%Alphabet%' OR HASUNIQUEIDENTIFIER LIKE '%Google%' OR HASUNIQUEIDENTIFIER LIKE '%GOOGL%' OR HASUNIQUEIDENTIFIER LIKE '%Alphabet Inc%'",
"ex_match": false,
"stop_reason": "final_answer",
"cost": { "exploration_steps": 16, "policy_steps": 0 },
"classification": {
"mechanical_cause": "join_path",
"memory_stage": "not_consulted"
}
}
{
"question_id": 1,
"arm": "agent_invoked_memory_prompted",
"question": "show me revenues for Alphabet?",
"predicted_sql": "SELECT e.HASMETRICVALUE FROM REVENUE r JOIN ELEMENTOFFINANCIALSTATEMENT e ON r.REVENUEID = e.ELEMENTOFFINANCIALSTATEMENTID WHERE e.HASUNIQUEIDENTIFIER LIKE '%Alphabet%';",
"ex_match": false,
"stop_reason": "final_answer",
"cost": { "exploration_steps": 9, "policy_steps": 1 },
"classification": {
"mechanical_cause": "join_path",
"memory_stage": "surfaced_not_used"
}
}
exploration steps9
outcomewrong
classificationjoin_path × surfaced_not_used
notetool named in prompt; still missed
Records are abbreviated, step traces
and token counts trimmed for display. Full logs for every arm and run are in the repository.
10
Probe results, per arm
computed from the logs, pick an arm
Outcome
correct, 3 runs57 · 54 · 56 / 115
mean55.7
tokens / question21,716
mean exploration steps7.8
Failure causes (run 3)
wrong join31
wrong output shape16
hallucinated schema3
wrong value3
unexplained5
# baseline · no memory in context
mean steps/query 7.8 median 6
episodes ≤4 steps 46/115
episodes at cap 18/115
steps when correct 7.8 | when wrong 7.9 ← no relationship
Outcome
correct, 3 runs48 · 50 · 45 / 115
mean47.7 (−8.0)
tokens / question16,381 (−25%)
mean exploration steps6.2
fixed / broken8 / 32
Probe 1, retrieval
recall@k, fix-relevant0.88
precision@k, fix-relevant0.42
Probe 3, stage
not surfaced13
surfaced, unused6
surfaced and used51
# cause × stage · 70 wrong answers
not_surfaced surfaced_not_used surfaced_used
wrong join 12 5 27
wrong output shape 1 1 16
everything else 0 0 8
[RQ1] 2 genuine utilization failures, memory contained a real fix, SQL didn't use it
Outcome
correct, 3 runs55 · 60 · 57 / 115
mean57.3 (inside noise)
tokens / question16,710 (−23%)
fixed / broken17 / 12
routes in store791
Error trade, all 3 runs
wrong join, baseline30 · 32 · 31
wrong join, derived27 · 25 · 27
output shape, baseline18 · 17 · 16
output shape, derived25 · 23 · 24
# the trade is steadier than any accuracy number
routes reliably fix ~4 join errors
routes reliably cause ~7 wrong-output-shape errors ← they canceltoken saving survives · accuracy does not move
Outcome
correct, 3 runs56 · 57 · 58 / 115
mean57.0 (inside noise)
tokens / question20,139
Consultation funnel (run 3)
episodes that asked8 / 115
retrieved ≥1 item8 / 8
≥1 fix-relevant item7 / 8
evidence of reuse6 / 7
changed the answer0
# with the tool available but never mentioned
consulted 5/115 · 6/115 · 8/115 across the three runs
outcomes among consulted episodes: beneficial 0 harmful 0
the agent almost never reaches for memory it wasn't told about
Outcome
correct, 3 runs58 · 55 · 56 / 115
mean56.3 (inside noise)
tokens / question20,596
Consultation funnel (run 3)
episodes that asked16 / 115
retrieved ≥1 item16 / 16
≥1 fix-relevant item12 / 16
evidence of reuse10 / 12
changed the answer1 better, 1 worse
# only difference: the prompt names the tool
consult rate 6/115 → 19/115 (x2 to x4.6 across runs)
accuracy unchanged, inside the noise band
asking is a prompt question. asking does not appear to help.
11
Act 2
These next steps go after the failure found
Mechanism
A step floor
Force the push-learned arm to explore a minimum number of steps before answering. If accuracy recovers
while some token saving survives, stopping early is the cause and a floor is the smallest useful
fix. If not, the injected content itself is the problem. Informative either way.
Method
Schema-based retrieval
Retrieve on schema elements the agent has actually seen, not on similarity to the question.
Keying memory on tables, columns and foreign-key edges makes every match checkable against the
database, and it can only fire after the agent has looked, so it cannot cause early commitment.
Validity
A second model
If the harm persists on a stronger model, the claim is about memory injection. If it vanishes,
the claim is about small models over-trusting their context. Either way it turns the study's
biggest weakness into a result.
stated limits
Everything rests on one weak model (gpt-4o-mini, ~50% accuracy), so this cannot yet separate
memory injection suppresses exploration from this model over-trusts its context.
Separately: two of my four early findings turned out to be artifacts of
my own measurement code, both flattering the hypothesis, a classifier bucket that swallowed
unrelated errors, and a similarity threshold that silently blocked every item in one arm. Both are
fixed and every number here is post-fix. That measurement code fails quietly, and in the direction
that suits the hypothesis, is reported as a finding of its own.