An investment assistant can pass an evidence check while answering the wrong question: the check accepted 31 of 35 misleading monthly SIP answers in exploratory agent runs reported by Yadav, Devadiga, Mishra and Singh. All 31 quoted a percentage present in their own tool output. Each percentage answered a question the user had never asked.
Anyone releasing or grading an investment assistant should care about that distinction. The paper's title, Right Numbers, Wrong Questions, captures its argument. A figure can come straight from the assistant's evidence and still answer the wrong question.
Two returns the investor could have meant
For HDFC Flexi Cap Fund, the three-year annual return to 30 September 2026 was 14.72% for the Direct plan and 13.97% for the Regular plan. Both figures are net of fund expenses, before exit load and tax. Both are correct. On Rs. 10 lakh, choosing between them changes the answer by about Rs. 29,300. Across the ten funds studied, that gap ranges from Rs. 21,800 to Rs. 48,300.
The authors call the failure frame collapse: the model chooses an interpretation of an underspecified question without establishing it or telling the user. As they point out, a support check accepts the answer because its number traces to the evidence.
Indian mutual funds give the authors formally defined plans, options, investing modes and tax rules. They can therefore calculate a reference for each reading. For Studies 2 and 3, those calculations use code, AMFI NAV records and the Income-tax Act, 2025. Each of the three studies varies one part of what the model sees.
Study 1 presents 22 questions in three wordings apiece. A lookup tool supplies no data, a thin record with one ambiguously named field, or labelled records naming every candidate; the scheme figures are illustrative. Study 2 collects 3,256 answers from 11 current models about ten real funds. Plan or mode is either left open or specified, and the Direct and Regular columns are counterbalanced. Study 3 supplies agents with profile, holdings and NAV tools for two constructed redemption cases. Separate tests have AI judges grade Study 1's answers and try a prompt that checks the data against the question.
Traceability does little for a mislabel
The three main Study 1 models got 96 of 126 answers wrong when a thin record supplied a figure other than the requested quantity. Labelled records reduced that to 7 of 126. Questions back to the user were scarce: only 1 of 162 thin-data answers asked anything. Even precise wording failed to protect the answer. In 26 of 42 thin answers, the model put the user's label on the wrong figure, compared with 0 of 42 labelled answers. GPT-4.1 called a Rs. 84,955 crore change in assets net inflow, reporting 2.9 times the actual Rs. 29,329 crore.
A mislabel could reverse the conclusion. Thin answers to one question said the large-cap category had beaten the Nifty 50's 7.6%; some described that benchmark as the total-return index (TRI). The figure belonged to the price index. At 8.2%, the category beat the price index and trailed the TRI's 8.9%.
Seven of eight models showed this pattern, with errors on 69 to 90% of those 126 answers. GPT-6.1 Sol got 38 wrong and mostly flagged the issue or asked.
Real fund data produced silent choices too. In Study 2, five models accounted for 140 of the 149 silent answers. The denominator was 713 open-item answers that stated a figure and on which both extractors agreed. Every one of the 81 silent answers about investing mode assumed a lump sum, despite the investor saying they had been investing since 2021.
Column order caused a milder failure in Study 2. These answers showed and labelled both plans, yet the layout determined which came first. The first-column plan led in 130 of 145 answers, and reversing the columns changed the lead in 61 of 84 pairs. The authors explicitly say they did not test whether readers act on the first figure.
Can the judge catch the wrong reading?
I would build a release process around the judging result. With the intended reading and the data in view, GPT-6.1 Sol identified 17 of 44 mislabels. A neutral dictionary explaining each field's measurement raised that to 40 of 44 (sign p = 0.004). False alarms on acceptable answers also increased, from 15 to 23 of 61.
On vague questions, supplying data alone made the judge worse: Sol accepted 28 of 70 wrong vague answers with data, against 4 without it. That comparison was post hoc. A deployed judge cannot know the user's intended reading, so the authors also tested it without that information. Given the dictionary, Sol still caught 40 of 44 mislabels, with 17 false alarms of 61. It also accepted 42 of 52 silent readings of vague questions.
The dictionary catches mislabels. Silent choices still get through.
The fact the agent failed to fetch
The authors selected Study 3 models because they had failed earlier. They present this study as evidence of a mechanism in some models, rather than an estimate of its frequency. When residency was removed, Gemini 3.1 Pro returned the resident figure in 20 of 20 runs. In the non-resident Indian (NRI) case, that overstated the bank credit by Rs. 16,484: Rs. 5,52,835 against the correct Rs. 5,36,350.
An explicit demand for the fact prompted retrieval in 10 of 10 runs; mentioning it in the tool description did so in 1 of 10. Retrieval alone was insufficient. Gemini fetched NRI status in 15 of 15 runs, yet passed along the compute tool's resident-based figure in 14. Under a checklist, GPT-4.1 fetched the facts 10 of 10 times and got none correct. For booked gains, it used the rule superseded in July 2024, calculating Rs. 23,516 of tax rather than the correct Rs. 26,145. Providing the current rule in prompt text corrected 5 of 5 runs, against 0 of 5. When a tool calculated under the old rule, the model relayed it 5 of 5.
GPT-6.1 Sol and Gemini 3.8 Flash, the frontier agents, retrieved the fact without prompting and were correct in 20 of 20 baseline runs. Even so, the 95% Wilson upper limit on failure per run remains 16% pooled and 28% per model. Those bounds concern these two cases only.
Choices left to the implementer
We did not rerun these experiments. Reproducing them exactly requires the authors' model versions, prompts, tool interfaces and scoring materials. Tests on models available to us would examine the same mechanisms without reproducing these counts.
The paper treats a change beyond the reporting tolerance as material. Choosing that tolerance requires judgment, and the authors' choice lies close to the edge. Study 2 allows 5% of the reference, with a minimum of 0.5 points. For HDFC Flexi Cap, the other plan's return falls only narrowly outside the tolerance, so the authors score the plan using the stated label rather than the figure. Study 3's tolerance permits an omitted cess, as they disclose. They also disclose that Study 1's codebook is stricter than their definition, pushing wrong-answer counts upward, and that Study 2's two extractors are themselves models being tested.
The cost assigned to a clarifying question is the largest judgment call. On gpt-oss-20b, the data-match prompt reduced wrong answers from 38 to 9 of 53. Questions to the user climbed from 1 to 33. Plainly correct answers dropped from 11 to 0, and the model asked a question on 21 of 53 labelled items where none was needed. The check outperforms both doing nothing and always asking only when an unnecessary question costs between half and about one wrong answer. Always asking earned the higher score for appropriate behaviour, 41 of 53.
The paper says it shows "how these failures arise, not how often," and I accept that limit. The authors recommend supplying a computed reference for every admissible reading and scoring whether the model established its reading. Failing every answer that leaves its reading unstated is an obvious next rubric to try; the paper did not test it. I would rank this concern lower if a prevalence count from real user queries showed that open dimensions are rare. Nothing here measures that.