HiFi-KPI, a benchmark for hierarchical KPI extraction from earnings filings that resurfaced in the forward feed on 2026-08-22, gives finance data teams a public yardstick for the build-vs-buy decision: in-house LLM pipeline or an AlphaSense-style vendor. The findings reported here come from a single, unreplicated preprint (arXiv:2502.15411) read at abstract level, so model-specific scores should be treated as directional rather than settled. The durable result is the shape of the failure, not the decimals.
What does HiFi-KPI actually measure?
HiFi-KPI measures whether a model can extract a key performance indicator from an earnings filing as a structured, hierarchical object, not just as a number found near a label. The corpus contains 1.65 million paragraphs annotated with 198,000 unique, hierarchically organized labels, each linked to the iXBRL taxonomies that regulators already mandate for public financial filings, according to the paper’s abstract.
The hierarchical part is the entire point. A flat extractor answers “what is the revenue figure in this paragraph.” A hierarchical extractor has to chain four things together correctly: which segment the figure belongs to, which metric it is, which reporting period it covers, and what value (in which unit and currency) it carries. Get any link in that chain wrong and the top-line number can still look right while the record is garbage.
The benchmark splits the problem into three tasks of increasing depth. KPI classification asks a model to assign the correct taxonomy label to a paragraph. KPI extraction asks it to pull the KPI out of the text. Structured KPI extraction asks for the full hierarchical object: segment, metric, period, value, resolved against the taxonomy. That three-rung ladder is what makes the dataset useful as a decision tool rather than just a leaderboard, because each rung maps to a different production ambition.
One structural detail matters before any numbers get quoted. Alongside the full corpus, the authors release HiFi-KPI-Lite, a manually curated subset of roughly 8,000 paragraphs intended for rapid evaluation, and every headline baseline number in the paper is measured on Lite, not on the full 1.65-million-paragraph corpus, per the abstract. Manual curation cuts both ways: it makes the labels trustworthy, and it means the subset is cleaner than whatever a production pipeline will actually encounter. Scores on Lite are, if anything, an upper bound on what a team should expect against raw filings in the wild.
Where does performance actually fall off a cliff?
Performance collapses between the classification task and the structured extraction task. On HiFi-KPI-Lite, encoder-based models achieve above 0.906 macro-F1 on KPI classification, while the LLM baselines in the paper reach 0.440 F1 on structured extraction, according to the authors’ reported baselines.
Two caveats have to ride alongside that spread, because quoting it raw would mislead. First, the 0.906 and the 0.440 come from different model families evaluated on different tasks; this is not a like-for-like encoder-versus-LLM shootout on identical inputs. An encoder doing classification is picking a label from a closed taxonomy, a fundamentally easier problem than generating a structured record. Second, the LLM baselines date to roughly February 2025, when the preprint was posted. Current-generation models may close part of that gap, and no replication with newer models has been published as of 2026-08-25.
What survives those caveats is the gradient itself. The same underlying filing text supports near-ceiling performance when the task is “attach the right label” and drops to coin-flip-adjacent performance when the task is “resolve the full hierarchy.” The paper’s authors frame this spread as the benchmark’s core signal, and that framing holds even if every absolute number shifts with newer models. The difficulty is not in finding KPIs. It is in binding them to their context.
For a team scoping a filings pipeline, this reorders the usual vendor-evaluation questions. “Can your model read a 10-K?” is answered by the classification scores: yes, comfortably, and cheap encoder models have been able to do taxonomy classification for years. “Can your model produce a structured KPI record I can load into a warehouse without a human checking it?” is answered by the structured extraction scores: not reliably, and not by a margin that anyone’s error budget absorbs.
Why do pipelines die one level down?
The dominant failure mode in the paper’s error analysis is dates, not values. The authors’ qualitative analysis finds that extraction errors primarily relate to periods: matching a figure to the correct reporting period, rather than parsing the number itself, per the paper.
This should worry anyone whose acceptance test is “spot-check the headline revenue figure.” Top-line values are the easiest thing in a filing to extract: they appear in the press release, the summary table, and the MD&A, usually restated several times, often in the first three pages. A pipeline can be right on every spot-check a stakeholder runs and still be systematically corrupt underneath, because the errors live in the joins. Was this operating income for the fiscal quarter or the trailing twelve months? Is this segment figure reported under the old segment structure or the one the company adopted this year? Is the number in thousands, millions, or actual units, and in which currency?
Period errors are particularly vicious because they are silent. A wrong value often trips downstream sanity checks: revenue jumps 40x quarter over quarter, someone looks. A right value attached to the wrong period passes every magnitude check and poisons every trend calculation built on top of it. Year-over-year growth computed against a misattributed comparison period is not an approximation; it is a confidently wrong number with correct-looking formatting.
Segment attribution has the same profile. Companies restate segment structures, rename divisions, and move product lines between reporting units. The model sees “Cloud revenue: $X” and has to know which “Cloud” the taxonomy means this period. This is exactly the kind of context that a single-paragraph extraction pass does not carry and that a hierarchy-aware schema forces into the open.
There is a temptation to read “errors concentrate in dates” as a prompt-engineering problem. It is closer to a data-modeling problem. Dates in filings are expressed relative to the document (“the three months ended June 30”), inconsistently across sections, and sometimes only implicitly through table headers. Resolving them requires document-level context, a fiscal-calendar model per issuer, and a reconciliation step against known filing periods. That is engineering, not prompting.
What does the benchmark say to spend the budget on?
The practical reading of HiFi-KPI is that the marginal dollar belongs in schema design and validation infrastructure, not in a bigger model. If the failure mode is hierarchical binding and the dominant error class is period attribution, then the parts of the system that determine accuracy are the schema that explicitly encodes segment, metric, period, and unit as separate fields, and the validation layer that reconciles extracted periods against the issuer’s filing calendar.
This inverts how most LLM extraction pilots are run. The default pilot picks a frontier model, points it at a folder of 10-Ks, and eyeballs the output on a handful of familiar companies. The top-line numbers look right, because top-line numbers are easy, and the pilot graduates to production with the period-attribution bug intact. HiFi-KPI’s task split is essentially a warning that this acceptance procedure measures the easy rung of the ladder and calls it the whole climb.
A defensible in-house design, given what the benchmark shows, looks something like this. First, use iXBRL tags as ground truth wherever they exist rather than parsing rendered HTML or PDF; the taxonomy linkage is the one free lunch in this problem. Second, define the extraction target as a typed record with explicit period semantics (period type, start, end, fiscal-year mapping), not as a string field called “period.” Third, build a reconciliation layer that checks every extracted date against the issuer’s actual filing periods and flags mismatches for review, because that is where the paper says the errors are. Fourth, evaluate on a curated, hierarchy-aware test set before trusting any aggregate accuracy number, and report the score decomposed by task depth the way HiFi-KPI does, so that a 0.9 on classification cannot launder a 0.5 on structured extraction into a blended 0.7 that looks shippable.
None of that is exotic. All of it is unglamorous. The recurring cost is also easy to underestimate at pilot time: taxonomies get revised, issuers restate segments, and the validation layer needs its own maintenance budget. The build-vs-buy math has to price that ongoing spend, not just the initial pipeline.
Build or buy: what are you actually paying a vendor for?
What a vendor subscription buys, at AlphaSense-style price points, is not extraction accuracy so much as a decade of accumulated corpus and curation. AlphaSense claims access to over 500 million premium financial and business documents on its platform page, and in a Forbes interview pegged to the company surpassing $500 million in ARR the company says it spent ten years building and curating that library, including 200,000 expert-call transcripts, so that model outputs are trustworthy, cited, and explainable.
An in-house pipeline that reads only public filings is not competing with that library; it is competing with the subset of it that is public. Filings with mandated iXBRL are the friendly case: structured ground truth exists, and HiFi-KPI shows models still stumble on hierarchy. The vendor argument is strongest exactly where iXBRL is absent: broker research, expert calls, transcripts, private-company material. No amount of schema engineering conjures access to that.
The buy side has its own evidence problem, though, and it deserves the same skepticism applied to the benchmark. The Forbes piece is a contributor interview, effectively vendor-adjacent. Its headline anecdote has AlphaSense’s CEO describing a private-equity client whose internal team spent five weeks analyzing the U.S. mid-market banking sector while the company’s Deep Research agent “did better in ten minutes.” That is a single CEO-sourced case with no published methodology, and it concerns research synthesis, not KPI extraction; it says nothing about whether the underlying numbers were hierarchically correct. AlphaSense markets Deep Research on its platform page as an agent that autonomously generates investment-grade research and compresses weeks of analysis into minutes, which is a marketing claim about speed, not a benchmark claim about accuracy. HiFi-KPI’s lesson applies symmetrically here: a synthesized research note can be right about the headline and wrong one level down, and the buyer has no taxonomy to reconcile against.
The one independent datapoint in the Forbes piece is secondhand: MIT’s State of AI in Business 2025 report, cited through the interview, reportedly found that only 5% of embedded, task-specific GenAI projects reach successful implementation, while 40% of simpler general-purpose deployments deliver results. The statistic is filtered through a vendor-friendly article and should be read with that discount applied. But the direction of the finding rhymes with what HiFi-KPI measures directly: task complexity, not GenAI in the abstract, is what tracks with failure. Structured hierarchical extraction is about as task-specific and embedded as the problem gets.
Which teams should build, and which should buy?
The defensible answer depends on which rung of the HiFi-KPI ladder the team actually needs. Classification-grade needs justify building. Structured-extraction needs at production scale justify buying, unless the team is prepared to fund the schema and validation work as a first-class system.
| Task depth | Benchmark signal (HiFi-KPI-Lite, ~Feb 2025 baselines) | Dominant risk | Defensible choice |
|---|---|---|---|
| KPI classification (label paragraphs) | Encoders above 0.906 macro-F1 | Low; mature task | Build: cheap encoder, minimal validation |
| Flat KPI extraction (find the value) | Between the two poles; not separately headlined in the abstract | Unit and currency drift | Build with validation, or vendor search features |
| Structured hierarchical extraction (segment, metric, period, value) | LLMs at 0.440 F1 | Period misattribution; silent wrong-period values | Buy, unless schema plus reconciliation layer is funded |
A few boundary cases are worth naming. A team whose deliverable is screening and retrieval (find me every filing that discusses segment X metric Y) is doing classification-shaped work and can build cheaply on public filings. A team whose deliverable is a queryable KPI warehouse feeding downstream models is doing structured-extraction-shaped work, and per the benchmark should expect to spend most of its budget below the model layer. A team that needs non-public documents has no build option at any task depth; the corpus is the product.
The middle column of that table is also the most honest disclaimer in this article: the absolute scores are dated, measured on a curated 8K-paragraph subset, and drawn from a single unreplicated paper. The row structure, however, does not depend on the decimals. Whatever scores current models post, the ordering of difficulty across the three tasks is a property of the problem, and the gap between them is what the build-vs-buy decision prices.
What would change this answer?
Three things would move the conclusion. First, replication. Every number here traces to one preprint, and its per-model tables and full error breakdowns have not been independently re-run or even fully extracted at the time of writing. If replication shows the degradation-with-depth pattern is an artifact of the Lite subset’s curation, the schema-first argument weakens. Second, current-model baselines. The LLM scores date to roughly February 2025; if a current-generation model posts structured-extraction F1 well above 0.440 on HiFi-KPI-Lite with the gains concentrated in period attribution, the case for buying a vendor on accuracy grounds erodes, though the corpus-access argument stands regardless. Third, production telemetry. The benchmark measures paragraphs; production failures involve restatements, taxonomy revisions, and issuer-specific calendar quirks that no static dataset captures. The first team to publish honest error rates from a live filings pipeline will contribute more to this decision than another point of benchmark F1.
Until then, the working position for finance data teams is the one the benchmark’s structure dictates. Do not treat a correct top-line number as evidence of production readiness, because HiFi-KPI’s three-task split shows that correctness decays one level down, into segment attribution and period matching, where errors are silent and date-shaped. If you build, spend on the schema and the reconciliation layer before you spend on the model, and evaluate decomposed by task depth. If you cannot fund that engineering, a vendor with a decade-curated library remains the rational default, at subscription prices that reflect the corpus more than the extraction. And treat the weeks-into-minutes vendor anecdotes as what they are: marketing about synthesis speed, from an interview, with no methodology attached. The hierarchy problem does not care how fast the answer arrives.
Frequently Asked Questions
Why is the 0.440 F1 score on structured extraction not a fair comparison to the 0.906 macro-F1 on classification?
The scores measure different cognitive tasks: classification selects a label from a closed taxonomy, while structured extraction generates a free-form hierarchical record. Comparing them directly ignores that the latter requires resolving segment, period, and unit simultaneously, a task where LLMs currently struggle with binding context rather than parsing values.
What specific operational cost does an in-house pipeline incur that vendor subscriptions typically hide?
In-house teams must budget for ongoing taxonomy maintenance and reconciliation logic, as issuers restate segments and regulators update iXBRL standards. This recurring engineering spend, which vendors absorb into their subscription fee, is often omitted from initial build cost estimates, making the total cost of ownership higher than the initial pipeline setup suggests.
How does the MIT State of AI in Business 2025 finding relate to the HiFi-KPI benchmark results?
The MIT report found only 5% of embedded, task-specific GenAI projects succeed, a statistic that aligns with HiFi-KPI’s finding that structured extraction is a high-complexity task. This suggests that the failure rate for hierarchical KPI extraction is not an anomaly but a predictable outcome of applying general-purpose models to narrow, high-precision data modeling problems.
Why is the AlphaSense ‘10 minutes vs. 5 weeks’ anecdote insufficient evidence for KPI extraction accuracy?
The anecdote measures research synthesis speed, not data extraction precision. It does not verify whether the hierarchical KPI records generated were correct, nor does it provide a methodology for validating period attribution. Therefore, it serves as a marketing claim about workflow efficiency rather than a benchmark for the accuracy of the underlying financial data.
