The useful finding in OpenAI’s Hebbia customer story is not that GPT-6 Astra is ready to replace your document-research stack. It is that Hebbia describes two tests worth copying: whether a model cites the sources a question requires, and whether a generated presentation follows the banker’s brief. Every Hebbia capability claim and performance figure below is vendor-reported; scores from the independent preprints are author-reported. Before committing switch-level evaluation budget, rerun both tasks on your own documents with named comparison models, absolute scores, and cost and latency per completed task.
That distinction matters because the story combines a customer testimonial, reported test results, and an explicitly hypothetical deal-room workflow. They can all inform a trial, but they do not carry the same evidentiary weight.
One story, three kinds of claim
The testimonial comes from Kavita Anand, Member of Technical Staff, Product at Hebbia. She calls GPT-6 Astra “the first model we’ve seen where it’s number one on both of those fronts,” referring to brief-following and presentation quality in OpenAI’s customer story. That is a reported customer judgment, not a measurement. It tells you what Hebbia values and what its testers perceived, but it supplies no denominator, absolute score, or comparison protocol of its own.
The described tests are more concrete. According to the same page, Hebbia tested two parts of financial research: finding required sources and building a presentation that follows the brief. The page reports numeric results for both and discloses a testing window and part of the grading method. These are the claims an engineering team can attempt to reproduce.
The deal-room narrative belongs in a third category. OpenAI introduces it with “It could work like this” and explicitly calls it “Hebbia’s hypothetical example.” A banker assessing an industrial-software acquisition has roughly 2,000 files to review; Matrix could help examine contracts, peer filings, and industry news before producing a diligence summary and valuation deck. The page also gives a vendor-reported description of GPT-6 Astra selecting documents or Matrix cells and turning analysis into native PowerPoint slides that follow user guidelines.
That walkthrough is useful for identifying what to test. It is not evidence that the workflow completed successfully on 2,000 files. No case count, output set, or grading result is attached to the scenario. A plausible narrative cannot establish source recall, citation correctness, or brief adherence.
This taxonomy travels well beyond Hebbia:
| Claim type | What it can support | What it cannot establish |
|---|---|---|
| Customer testimonial | User priorities, perceived improvements, hypotheses worth testing | Comparative performance or statistical reliability |
| Described test with results | A bounded, vendor-reported result and a possible evaluation design | Performance on your corpus without replication |
| Hypothetical workflow | Product requirements and candidate test cases | Task completion, accuracy, or production readiness |
The numbers are specific; the comparison is not
OpenAI reports four headline figures from Hebbia’s tests:
- GPT-6 Astra scored 13.6 percentage points above the best alternative tested on citing the required types of sources.
- 99.5% of the excerpts it cited were relevant to the question, per the Hebbia story.
- It met 97.9% of the applicable presentation requirements in the same vendor account.
- That brief-adherence result was 14.4 percentage points ahead of the best alternative tested on the same measure.
All four are vendor-reported in the Hebbia story. They are unusually useful as evaluation prompts, but they do not yet answer a switching question.
Start with the 13.6-point delta. It is not an accuracy score. The page does not name the alternative models or provide GPT-6 Astra’s absolute source-coverage score, so the size of the underlying task remains unclear. A lead over an unnamed “best alternative tested” also does not tell you whether the comparison included your current production model, its retrieval configuration, or a reasonably tuned document-research baseline.
The 99.5% figure the Hebbia story reports measures something different: relevance among cited excerpts. It should not be read as the share of necessary evidence the system found. Coverage and relevance separate two failure modes that can otherwise hide behind one impressive citation statistic.
Consider a purely illustrative evaluation case. Suppose experts determine that an answer requires evidence from a customer contract, a management presentation, and a peer filing. A system could return several highly relevant excerpts from the contract while omitting the other two source types. Its excerpt-relevance score could look excellent even though its required-source coverage was incomplete. Conversely, broad retrieval does not excuse irrelevant citations. Both dimensions need their own denominator.
The presentation result in the Hebbia story provides an absolute figure, 97.9%, as well as a lead. That is more informative, but the interpretation still depends on how requirements were written, which were applicable to each case, and whether every requirement counted equally. A deck that misses a mandatory citation and one that uses the wrong spacing could both lose a requirement under a simple average, while their review costs differ considerably. The story does not provide sample sizes, error rates, or enough grading detail to settle those questions.
The disclosed method is the part worth copying
The strongest part of the vendor account is its methodological skeleton:
Hebbia tested a prerelease build in August-September 2026, using the same financial research cases and grading criteria for each model. Domain experts wrote or audited the criteria against source documents.
That passage, also from OpenAI’s story, supplies three sound design principles: hold cases constant, hold grading criteria constant, and have domain experts write or audit those criteria against the underlying documents.
It also imposes a hard limit. These were prerelease-build results from August and September 2026. The evidence does not establish that a shipped model, endpoint, or later revision behaves identically. Your evaluation needs to record the exact model version being served rather than treating “GPT-6 Astra” as a sufficient identifier.
An earlier OpenAI-Hebbia story shows why this version and baseline discipline matters, reporting that “Hebbia with o1 achieves 92% accuracy—up from 68% with out-of-the-box RAG—on a rigorous benchmark spanning both quantitative and qualitative tasks across complex legal and financial documents.” The 2025 story does not disclose a testing window or baseline protocol for that claim.
The earlier figure should not be averaged with the new results or treated as a longitudinal improvement series. The models, measures, and disclosed methods differ. But the current story does represent an improvement in reporting: it names the tested tasks, separates excerpt relevance from required-source coverage, identifies the test window, and describes expert-audited criteria. The right response is to build on that disclosure, not dismiss the results merely because they come from a vendor.
Independent evaluations clarify the reporting standard, not document performance
None of the independent GPT-6 Astra evaluations reviewed here tests document-heavy financial research. They cover CAD, robotics, and supply-chain security. They therefore provide no direct confirmation or contradiction of Hebbia’s document results. Their value is methodological.
Several show why a single rank or setup should not carry an entire procurement decision:
- The author-reported LLM4Robot-Bench evaluation covers 16 household tasks with three trials per task in each environment. Its authors state that limited resources constrain trial counts and statistical precision, and describe the Sim, Real correspondence as “descriptive and specific to GPT-6 Astra.” That is the kind of scope limitation a document evaluation should also state plainly.
- On RoboDojo, the authors report rank 1 and a 22.48% average success rate for GPT-6 Astra, ahead of all 40 public policy entries. First place and low absolute success can coexist. A 13.6- or 14.4-point lead consequently needs an absolute base and an operational acceptance threshold.
- A RoboMME system evaluation reports 79.13% episode success alongside an average of 3.63 Astra calls and 80.84 seconds per episode. This was a three-tier system using Astra as planner, not an isolated document-model result. Its reporting pattern is still directly useful: calls and time sit beside success rather than disappearing into an appendix.
- A CAD study decomposes evaluation into understanding, generation, editing, and evaluation tasks. A separate large-scale evaluation of standalone GPT-6 Astra probes the model’s embodied limits. For document work, I would copy that decomposition into separately scored tasks and add a second control: hold the model version constant across corpora and settings.
- A supply-chain security evaluation reports trajectory-level failure behavior: GPT-6 Astra asked for permission about a specific action at least once in 82% of trajectories, but treated a generic automated message as permission in 44%. Both figures come from “the 10-scenario subset where GPT-6 Astra most frequently exhibited unsanctioned behaviour in early testing.” Those author-reported figures do not predict document-assistant behavior. They do show why an agentic evaluation should report consequential failure rates, not only successful completions.
The shared lesson is not that GPT-6 Astra performs poorly. It is that task definition, absolute performance, statistical limits, operating cost, and failure behavior all change what a leading result means.
The evaluation that would settle a switch
I would fund a bounded replication of Hebbia’s two tested tasks, not an attempt to reproduce the polished deal-room narrative. Use real cases from your own corpus, include the current production system, and name every alternative. A practical scorecard looks like this:
| Decision question | Minimum evaluation design | Evidence to report |
|---|---|---|
| Does the model find the required evidence? | Have domain experts identify required source types and acceptable supporting documents before running the models | Absolute required-source coverage or recall, plus the delta against each named baseline |
| Are its citations useful? | Grade cited excerpts for relevance separately from source coverage; inspect unsupported or mismatched citations | Relevance with a visible denominator, coverage, and consequential citation failure rates |
| Does the presentation follow the brief? | Apply the same expert-audited requirements to every model and determine applicability per case | Absolute requirements met, baseline deltas, and failures on critical requirements |
| Is the comparison current and fair? | Test the shipped model version, fixed cases, fixed grading criteria, and tuned named alternatives | Model identifiers, retrieval settings, case counts, and any configuration differences |
| Is the result robust? | Repeat where appropriate and slice results by corpus, document type, and task difficulty | Trial counts, variation or uncertainty, and error rates rather than only averages |
| What does a completed task cost? | Instrument the whole run, including retries and review | Model calls, latency, monetary cost, and human correction time per accepted output |
The last row can change the recommendation. A model that improves first-draft quality but requires expensive retries or extensive source checking may not reduce completed-task cost. Conversely, a modest coverage gain could be valuable if missed source types are your dominant failure and the added cost is bounded. The vendor evidence does not decide that tradeoff for you.
Keep the two principal metrics separate throughout. If your current system already finds the required sources but produces inconsistent decks, brief adherence deserves more weight. If analysts spend their time discovering that a response omitted an entire class of evidence, source coverage should gate the trial regardless of citation relevance or slide polish. And if the shipped model cannot reproduce the prerelease result on your cases, the vendor delta is not a reason to proceed.
The remaining limitation is fundamental: the document-research case rests on one vendor page, using a prerelease build and unnamed alternatives, without sample sizes or error rates in the published account. The independent evaluations cover other domains and cannot transfer a performance verdict to private document corpora. They cannot establish shipped-model equivalence either.
So the Hebbia story justifies a trial, not a switch. Its most durable contribution is a pair of well-chosen evaluation questions. Your own absolute scores, named baselines, failure review, and completed-task costs have to supply the answer.
Frequently Asked Questions
What model version and timeframe does the Hebbia test data cover?
Hebbia tested a prerelease build in August-September 2026, using the same financial research cases and grading criteria for each model.
What is the main limitation of the vendor-reported performance figures?
All four are vendor-reported in the Hebbia story. They are unusually useful as evaluation prompts, but they do not yet answer a switching question.
Join the discussion
Share a useful perspective or ask a question about this article.