Knowledge Graph RAG: How Constrained Decoding Stops Hallucinated Terms

The cheapest defensible way to keep a knowledge-graph RAG system from emitting hallucinated terms may be a filter that runs after the model has already spoken. The pro-team system at LLMs4OL 2026 (arXiv:2608.27101, submitted 27 August 2026) pairs retrieval-augmented few-shot prompting on Qwen2.5-14B with deterministic vocabulary-constrained filtering and reports 0.9200 Term-Typing F1 on the ontology extension task. Every number in this article is author-reported on a shared-task sample, unreplicated, with no baseline or ablation. Read accordingly.

What landed on arXiv on 27 August 2026, and how much weight does it carry?

The submission is a shared-task system paper describing one team’s pipeline for the LLMs4OL 2026 Flagship and Reuse tasks, and it carries exactly the evidentiary weight of a moderated arXiv preprint: some, but not much. arXiv’s own about page states plainly that material on the platform is not peer-reviewed, that contents are wholly the responsibility of the submitter, and that everything is presented as-is without warranty. What arXiv does provide is moderation: per its submission policy, a community of volunteer moderators verifies that material is appropriate and topical, and removes offensive, non-scientific, or plagiarized content. Moderation confirms a paper looks like science. It says nothing about whether the science is right.

That distinction matters more in 2026 than it used to. In November 2025, arXiv announced it would stop accepting computer science review articles and position papers that had not been vetted by a journal or conference, citing an increase in AI-generated research. The pro-team paper is a system description rather than a review article, so the policy does not touch it directly, but the policy shift is a signal about the platform’s signal-to-noise trajectory in CS. Three million hosted articles, volunteer moderation, and a flood of generated text is not a combination that rewards credulity.

So the correct posture toward this preprint is the one you would take toward a colleague’s unreviewed experiment writeup: the mechanism description is probably accurate (it is easy to check against your own implementation), while the scores are a single team’s result on a single shared-task sample. The abstract offers no baseline, no ablation isolating the filter’s contribution, no hallucinated-term rate, and no latency measurements. What it offers instead is a clean, inspectable design and one very instructive failure mode, which is where the operational value sits.

Is this actually constrained decoding?

No. Despite what the framing of “vocabulary-constrained” output suggests, the pro-team system applies its constraint after generation, not during it, and the difference is the whole implementation story. The paper’s Task B mechanism is deterministic filtering: a generated triple is retained when at least one of its endpoints belongs to the sample’s closed term and type vocabulary, and duplicates of the initial ontology are removed. The model generates freely. A separate pass decides what survives.

True constrained decoding operates at the logit level: at each generation step, tokens that would produce output outside a grammar, schema, or vocabulary get their probabilities zeroed, so invalid sequences are never sampled at all. Post-hoc filtering operates on finished strings: the model says whatever it says, and a validator discards what does not comply. The plain-English sense of “constrain” fits both, which is part of the confusion; Merriam-Webster’s primary definition, “to force by imposed stricture, restriction, or limitation,” describes a logit mask and a regex over completed output equally well. Vendors and paper titles will not draw the line for you. Draw it yourself.

The practical consequences of the distinction run through every operational property this article cares about. Decode-time constraint guarantees format validity by construction and wastes zero tokens on output you will throw away, but it couples you to a specific inference stack and, for large vocabularies, to nontrivial mask-construction cost per step. Post-hoc filtering, the paper’s approach, is deterministic, model-agnostic, and can be attached to any existing RAG pipeline without touching the sampler, but the model still burns tokens generating triples that the filter will discard, and nothing prevents malformed output from reaching the filter in the first place. Neither approach appeared in the abstract with latency numbers attached, so the cost comparison here is engineering reasoning rather than cited measurement.

How does the pro-team pipeline actually work?

The pipeline is an offline retrieval-augmented few-shot prompting system with a deterministic filter at the end, and each component is a conservative, well-understood choice rather than a novel one. According to the preprint, generation runs on Qwen2.5-14B-Instruct. Demonstrations for the few-shot prompt are retrieved with all-MiniLM-L6-v2, the standard lightweight sentence embedder, selecting the top-5 examples for Task A and the top-2 for Task B. A left-truncated context-windowing strategy drops the oldest content when prompts grow long while preserving the task instructions, which is the part of the prompt the model most cannot afford to lose.

Read the architecture as a list of bets. A 14B open-weights instruct model is large enough to produce structured triples reliably and small enough to self-host on modest hardware. MiniLM retrieval over demonstration examples is the cheapest few-shot selection mechanism available; no fine-tuning, no learned retriever, no task-specific training anywhere in the loop. Left-truncation windowing is an admission that real ontology-learning prompts, stuffed with retrieved demonstrations and vocabulary lists, will exceed comfortable context lengths, and that the instructions matter more than the earliest demonstrations when something has to go.

The Task B filter deserves a precise reading because its exact rule shapes both the results and the failure mode. A triple survives if at least one endpoint, subject or object, appears in the sample’s closed term/type vocabulary. Note the asymmetry: a triple with one known endpoint and one novel endpoint is kept, on the theory that extending an ontology means attaching new terms to existing anchors. A triple where both endpoints are outside the vocabulary is dropped entirely, and anything duplicating the initial ontology is removed as redundant. This is a grounding rule with an opinion: the model may propose new terms, but only when tethered to something the ontology already knows. Terms the model invents in free-floating clusters, unanchored to existing vocabulary, are treated as hallucinations and discarded regardless of whether a human might judge them valid.

What do the reported scores measure, and what do they miss?

On Task B, the ontology extension reuse task, the system reports Semantic Graph Similarity of 0.8692, Term-Typing F1 of 0.9200, and Taxonomy Discovery F1 of 0.8540; on Task A, the end-to-end flagship task, it reports Semantic Graph Similarity of 0.7416. All figures are from the abstract, author-reported, on the shared-task sample.

The strongest number, 0.9200 Term-Typing F1, is also the one most plausibly attributable to the filter’s design. Term typing assigns types to terms that, by construction of the filter, are anchored to known vocabulary, so the metric and the mechanism reinforce each other. Taxonomy Discovery at 0.8540 tracks the hierarchical relations the pipeline was oriented toward. Semantic Graph Similarity, a whole-graph measure of overlap between the extracted graph and the reference, sits at 0.8692 on Task B and drops to 0.7416 on Task A, which is a reminder that end-to-end extraction with no reused ontology to anchor against is substantially harder than extension. None of these scores can tell you what the filter bought, because the paper reports no ablation run without it.

What the scores do not measure is the longer list, and it is the list a practitioner actually needs. There is no hallucinated-term rate before and after filtering, so the headline benefit of the mechanism, suppressed hallucinations, is inferred from the design rather than quantified. There is no baseline of unconstrained RAG output on the same task, so the gap between “filtered” and “anything else” is unknown. There are no latency figures, no token counts, no discard-rate statistics showing what fraction of generated triples the filter threw away. A filter that keeps 0.9200 F1 while discarding the bulk of the model’s output is a very different operational proposition from one that discards a sliver, and the abstract does not say which regime this system occupies.

The paper’s own motivation, quoted in the abstract, is that LLMs “can hallucinate domain terms, produce inconsistent formats, and favor hierarchical over associative relations” in ontology learning. Two of those three failure modes are addressed by vocabulary filtering. The third is caused by it.

Why did the system extract zero non-taxonomic relations?

The system extracted zero non-taxonomic relations, a result the authors attribute to closed, taxonomy-oriented relation vocabularies, and this single line is the most operationally important finding in the preprint. The constraint did not only suppress hallucinated terms. It suppressed an entire class of valid output, silently and completely.

The mechanism is straightforward once you see the filter’s rule. When the allowed relation set contains hierarchical relations and little else, any associative relation the model proposes, the “related-to,” “influences,” “commonly-co-occurs-with” edges that carry much of a real knowledge graph’s value, has no valid label to attach to and no path through the filter. The model may well have generated such triples. The preprint cannot tell us, because a discarded triple leaves no trace in the scores. Zero extracted non-taxonomic relations is consistent with a model that proposed many and had them all filtered, and equally consistent with a model that proposed none. The paper’s own motivation admits LLMs “favor hierarchical over associative relations,” so some of the gap is upstream of the filter. How much is unknown, and unknowable from the reported numbers.

For teams evaluating this pattern, the lesson generalizes past ontology learning. Any closed-world filter on generative output inherits the coverage of its vocabulary as a hard ceiling. On the LLMs4OL shared-task sample, that ceiling was taxonomy-oriented relation vocabularies, and the visible symptom was a zero in one column of the results table. In an enterprise deployment, the ceiling is whatever your term list does not yet contain: the product name launched last month, the internal codename that entered Slack before it entered the wiki, the regulation published after the last vocabulary refresh. Each of those becomes not an error but an absence, and absences do not page anyone.

Should you use constrained decoding, post-hoc filtering, or plain vector RAG?

For most teams grounding LLM output in an enterprise knowledge graph, post-hoc vocabulary filtering is the right starting point: it is deterministic, model-agnostic, and bolts onto an existing RAG stack without touching the inference path, which is precisely the profile of the pro-team system. The preprint provides no direct comparison across the three mechanisms, so the table below is engineering reasoning from the mechanisms’ properties, not cited benchmark data. Treat it as a decision framework, not a results table.

The rows interact in ways worth spelling out. Decode-time constraint is the strongest guarantee and the heaviest lift: it moves the vocabulary problem into the sampler, where a stale constraint does not silently drop output but actively prevents it, which is arguably worse for observability. Post-hoc filtering, the paper’s approach, has the best deployability story and the worst waste story; every discarded triple was generated at full token cost, and as request volume grows the discard rate becomes an inference-budget line item. Plain vector RAG remains the right answer when the output space genuinely cannot be enumerated, which is true more often than vendors of either constraint mechanism like to admit.

The pro-team results slot into this framework as one data point for the middle column: on a closed-vocabulary ontology extension task, with a 14B model and cheap retrieval, filtering delivered 0.9200 Term-Typing F1 and zero coverage of non-taxonomic relations (arXiv:2608.27101). That is roughly the shape the table predicts. Strong within the vocabulary’s coverage, silent outside it.

What does term governance look like once you adopt a closed vocabulary?

Adopting a closed constraint vocabulary means inheriting a governance job that vector RAG never created, because the vocabulary becomes a production artifact with a freshness cadence, an owner, and a failure mode. Plain vector RAG degrades gracefully: when the corpus is stale, retrieval quality slides, and the symptom is vaguer answers. A closed vocabulary fails by absence: valid terms produced after the last refresh are excluded from output, and nothing in the system’s behavior announces the exclusion. The pro-team paper’s zero non-taxonomic relations is that failure mode caught in a benchmark, where a reference graph exists to make the absence measurable. Production systems mostly do not have reference graphs.

Running the governance job well comes down to three practices, all implied by the paper’s admitted limitation rather than stated in it. First, refresh the vocabulary on an ontology-change cadence rather than a calendar cadence; the trigger should be “the KG changed,” not “it has been a quarter.” Second, monitor the discard stream. The rate and content of filtered triples is the only signal that distinguishes a healthy filter (dropping genuine hallucinations) from a stale one (dropping valid new terms). A sudden rise in discards of one-endpoint-anchored triples is what vocabulary drift looks like in telemetry. Third, accept that associative relations need a different mechanism entirely. The pro-team result suggests closed relation vocabularies cannot capture non-taxonomic edges; if your KG needs them, plan for a separate extraction path, an open relation vocabulary, or human review of filter rejects rather than expecting the constraint to stretch.

None of this is free, and it should be priced into the build-versus-buy decision. The pitch for constrained output is that it converts a probabilistic correctness problem into a deterministic one. The fine print is that it converts a model-quality problem into a data-governance problem, and governance problems do not have F1 scores. Teams with mature KG operations already run the ingestion, versioning, and change-review machinery this requires; for them, a constraint list is one more governed artifact in an existing workflow. Teams adopting KG RAG precisely because they lack that machinery should treat the vocabulary as new operational surface area, not as a free accuracy win.

What is the verdict, and where does this evidence stop?

Post-hoc vocabulary filtering is the cheapest defensible grounding step for KG-backed LLM output: deterministic, model-agnostic, and bolt-on to an existing RAG stack, with the pro-team system’s author-reported 0.9200 Term-Typing F1 suggesting it works for term typing within a closed vocabulary’s coverage. The same result set carries the counterweight: zero non-taxonomic relations extracted, a failure the authors themselves attribute to closed, taxonomy-oriented relation vocabularies. A stale constraint list does not merely miss new terms; as this preprint demonstrates, it can silently ban entire classes of valid output. If you adopt the pattern, treat the vocabulary as a governed artifact: refresh it on ontology changes, log what the filter drops, and route associative relations through a separate mechanism.

Now the asterisks, and they are large. Every number cited here is author-reported on the LLMs4OL 2026 shared-task sample in a moderated-but-not-peer-reviewed arXiv preprint, unreplicated as of 30 August 2026. The abstract contains no baseline, no ablation isolating the filter’s contribution, no hallucinated-term rate, and no latency measurements, so the three-way mechanism comparison above is reasoned from mechanism properties, not evidenced by controlled experiment. The vocabulary in the shared task is closed per sample; how freshness behaves against a living enterprise ontology, over months of drift, is extrapolated from one admitted failure mode rather than measured. And the platform context matters: arXiv’s policies place responsibility for content entirely on submitters, and its November 2025 restrictions on unvetted CS review papers, prompted by rising AI-generated research, are a reminder that preprint scores in 2026 deserve more scrutiny, not less.

The mechanism is sound, the design is worth copying, and the failure mode is the most useful part of the paper. The numbers are a starting point for your own evaluation, not a substitute for one. Run the filter on your own ontology, measure your own discard rate, and decide with telemetry rather than with an abstract.

Frequently Asked Questions

How does the token cost of post-hoc filtering compare to decode-time constrained decoding?

Post-hoc filtering incurs full generation costs for every discarded triple, whereas decode-time constraints prevent invalid tokens from being sampled, resulting in zero wasted inference compute for rejected outputs.

What specific operational metric should teams monitor to detect vocabulary drift in a closed-constraint system?

Track the discard rate of one-endpoint-anchored triples; a sudden spike in this specific category indicates that valid new terms are being silently filtered out rather than genuine hallucinations being removed.

Why is the zero non-taxonomic relation result in the LLMs4OL 2026 preprint considered a stronger signal than the 0.9200 Term-Typing F1?

The F1 score confirms performance within the existing vocabulary, but the zero count reveals a hard coverage ceiling where the constraint actively suppresses valid associative edges, exposing a silent failure mode that high accuracy metrics hide.

What is the primary risk of adopting a closed vocabulary for an enterprise knowledge graph without a reference dataset?

Without a reference graph to measure against, the system cannot distinguish between filtering hallucinations and filtering valid new terms, making the absence of expected data invisible and unactionable in production telemetry.