How to Run a per-Prompt Diversity Audit of Generative AI Outputs

If your model sign-off rests on a single pooled diversity number, the audit has a structural hole: a model can score well on overall variety while collapsing on specific prompt types, and the pooled figure will not tell you. A prompt-aware metric now exists that makes the per-condition gap measurable. The Conditional Vendi Score (arXiv 2411.02817) quantifies “model-induced diversity by analyzing variability across prompt types,” and the audit redesign it enables is the subject of this article: report diversity per prompt type or per class, pre-register the similarity function, disclose the embedding, flag small sample cells, and write down what the score cannot certify.

Two cautions up front, because they shape everything below. First, the anchor method is a single preprint with author-reported results and no independent replication; its adoption status is unknown as of October 2026 [unverified]. Second, a diversity score certifies variety, a distributional property. It does not certify representational correctness, and it does not detect stereotypes. The audit structure below is built to make those limits visible rather than paper over them.

Why pooled diversity numbers pass models that fail on specific prompts

The case against pooled reporting is already in the literature. A 2025 paper on synthetic persona experiments in LLM research states the problem directly: “unified dataset diversity scores might miss the essential specifications of task and target population” (Whose Personae?, arXiv 2512.00461). An audit that reports one number for “the model” cannot express the question a reviewer actually needs answered: does the model produce varied outputs for this prompt type, aimed at this population?

The mechanism is easy to state, though the cited evidence establishes a risk rather than a documented audit failure. If an image generator produces wide demographic variety across its whole output distribution but defaults to a narrow slice for prompts about executives, the pooled score averages the collapse away. The inference that follows, and it is an inference from documented limitations rather than a measured benchmark result, is that the unit of audit reporting should be the condition: a prompt type, a class, a target population. The pooled figure survives, but as a summary row, not as the pass/fail gate.

The Vendi Score in one paragraph

The base metric comes from The Vendi Score paper (arXiv 2210.02410): “The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of a similarity matrix. This matrix is induced by a user-defined similarity function applied to the sample to be evaluated for diversity.” Two properties matter for audit design. First, it needs no reference dataset: it is “applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined.” Second, the user-defined similarity function is a load-bearing choice. An overly sensitive similarity function makes every collection look diverse; an insensitive one makes everything score low. Which similarity function you chose is therefore part of the audit result, and an undisclosed one is an undisclosed degree of freedom.

Conditional Vendi: conditioning on prompt types without repeated generations

The practical objection to per-prompt auditing has been cost: measuring per-prompt diversity the naive way means generating many outputs per prompt, for many prompts. The Conditional Vendi Score paper (arXiv 2411.02817) addresses this directly: “Conditional-Vendi does not require repeated generations; instead, it quantifies model-induced diversity by analyzing variability across prompt types,” and it “can be decomposed across different prompt modes to evaluate diversity conditioned on text categories.” The method builds kernel matrices over prompt, output pairs to separate variation the model introduces from variation the prompts themselves introduce. The authors report theoretical finite-sample convergence guarantees and experiments showing the scores “capture intuitive aspects of diversity that unconditional metrics overlook.”

The claimed convergence guarantees and benchmark behavior are author-reported; no independent evaluation of Conditional-Vendi is cited here.

The authors also disclose the limitation that matters most for an audit context: the approach relies “on high-quality embeddings, which may introduce biases for an arbitrarily selected embedding.” If the embedding used to compute similarity carries its own skews, the diversity estimate inherits them. An audit checklist has to treat embedding choice as a disclosed, justified artifact, not an implementation detail.

The per-prompt audit checklist

The checklist below is a synthesis of the limitations documented across the Vendi literature, organized as audit steps. It is not a validated procedure; no cited study measures audit outcomes under this structure. Each step exists because a specific published failure mode would otherwise stay invisible.

1. Define conditions before generating. Enumerate prompt types or classes and the target population each represents. This is the fix for the unified-score gap identified in arXiv 2512.00461: the condition taxonomy is the specification of task and target population.
2. Pre-register the similarity function. Write down the function and its sensitivity rationale before looking at scores. Sensitivity can invert conclusions, so choosing it after seeing results is tuning, not auditing.
3. Disclose the embedding. Name it, justify it, and record the known-bias caveat from the Conditional Vendi authors.
4. Set the metric order. For per-class diversity in image generative models, an extensive evaluation by Stein et al. (2023) “used the Vendi score of order q=1 and found it to work more effectively as a measure of per-class diversity,” as summarized in the Cousins of the Vendi Score survey (arXiv 2310.12952). Absent contrary evidence for your domain, q=1 is the documented default for per-class work.
5. Report per-condition scores with per-cell sample counts. Every cell reports its n. Counts well below 20,000 get flagged, per the convergence results in arXiv 2410.21719; the next section explains why.
6. Check prevalence balance across conditions. The survey notes the Vendi Score “treats each item in a given collection with a level of sensitivity proportional to the item’s prevalence. This is undesirable in settings where there is a significant imbalance in item prevalence.” Unevenly sampled prompt types distort the score exactly where audits care most.
7. Attach the boundary statement. Every report states, in writing, that the scores certify variety only.

Failure modes: sensitivity, imbalance, and the sub-20,000-sample trap

The strongest counter-evidence in this literature lands directly on the audit design above. Truncated Vendi Score work (arXiv 2410.21719) demonstrates numerically that “the Vendi score computed with standard sample sizes below 20,000 may not converge to its asymptotic value under infinite sampling” (arXiv 2410.21719). Worse for practitioners, the cost of eigendecomposition often limits the number of samples used to fewer than 20,000 (arXiv 2410.21719). Per-prompt audit cells are small by construction, which places them among the sample sizes where the score may not converge. The risk is also kernel-dependent: in the same experiments, a cosine-similarity kernel on ImageNet samples embedded with DINOv2 had almost converged by 5,000 samples (arXiv 2410.21719), while the infinite-feature Gaussian kernel was still climbing at a significant rate at 20,000 samples, with larger evaluations computationally prohibitive.

This does not kill the design; it changes what the numbers mean. A per-cell score with n of a few thousand is a flagged estimate, not a settled measurement. The honest options are to grow cells where generation cost allows, to treat small-cell scores as directional evidence requiring corroboration, and to disclose the convergence risk next to every flagged number. The Conditional-Vendi authors report a partial mitigation inside their own experiments: the truncated-spectrum variant of their score converged near 15,000 samples (arXiv 2411.02817), still far above a typical per-cell audit sample. What you cannot do is report a 2,000-sample per-prompt score with the same confidence as the pooled figure.

Layer on the other two documented failure modes. Similarity-function sensitivity means two auditors can reach opposite conclusions about the same model from the same outputs. Prevalence imbalance means a model that over-serves one condition drags the metric’s attention with it. All three failure modes share a property that argues for the checklist format: none of them is visible in the final number.

There is also a bound on what embedding-based scoring can capture for LLM persona audits. In Localizing Persona Representations in LLMs (arXiv 2505.24539), “the analyzed personas show large differences in representation space only within the final third of the decoder layers.” That finding concerns internal decoder activations, not the external embeddings an audit would compute over, so the bridge is analogical. Still, if persona differences live in a narrow slice of representation space, the embedding you pick determines whether the audit can see persona variation at all.

What statistical diversity cannot certify

A diversity score is a distributional summary. It can tell you that outputs conditioned on a prompt type are varied or collapsed. It cannot tell you whether the variety is correct: whether the demographics produced match the target population, whether the variation maps onto real groups respectfully, or whether a nominally diverse output set encodes stereotypes in every sample. A model can produce ten visibly different personas, all built on the same caricature, and score well.

No study cited here connects Vendi-family scores to concrete harm detection. The persona-transparency critique in arXiv 2512.00461 pushes in the same direction from the other side: even specifying task and target population correctly is a transparency and governance problem, not something a scalar score settles. The practical consequence is that the boundary statement in step 7 is not boilerplate. A reviewer who signs off on a diversity number has certified variety. If the organization needs a claim about representational correctness or stereotype absence, that claim needs a different instrument: human evaluation, harm taxonomies, community review. I would treat any audit report that presents a Vendi-family score as evidence of fair representation as a finding in itself, regardless of how the model scored.

Governance context: who is asking for this

The closest external pressure comes from governance indices rather than model audits. The Global Index on Responsible AI 2026 report assesses national responsible-AI commitments “across five dimensions: Inclusion and Diversity, Ethics and Sustainability, Labour and Skills, Trust and Safety, and AI Use in Public Service.” GIRAI scores countries, not models, so any connection to a per-prompt audit is this article’s inference rather than the index’s design. The inference is still worth making: an inclusion measure that ignored who is included would be incoherent, which is why per-condition evidence answers an inclusion question better than a single pooled number does.

As of the cited August 2023 analysis, the DMA route is closed. An analysis of the EU Digital Markets Act (arXiv 2308.02033) notes that the Article 2(2) list of core platform services “covers ten established digital services, such as operating systems, web browsers, and social networking. It does not explicitly cover generative AI services.” The obligations the DMA attaches to core platform services therefore did not reach generative AI offerings as such. No reviewed regulation mandates per-condition representation audits.

The decision

Should you restructure your audit around per-condition reporting? The comparison:

My recommendation, within the evidence: move the pass/fail decision to per-condition reporting, per prompt type or per class, using q=1 for per-class diversity, with a pre-registered similarity function, a disclosed embedding, per-cell sample counts flagged against the convergence risk, and the signed boundary statement. Keep the pooled figure as a summary row. This structure is an inference from documented limitations of the metrics, not a measured result, and no cited study establishes that it improves real audit outcomes.

The strongest reason for caution is the anchor itself. Conditional Vendi is one preprint, author-evaluated, unreplicated as of October 2026, with an authors-acknowledged dependence on embedding quality and a documented convergence risk at the small sample sizes per-prompt auditing creates. If you adopt this structure, adopt it as a disclosure regime: the value over pooled reporting is that it surfaces conditions, sample sizes, and design choices a reviewer can interrogate, not that the per-condition numbers are yet trustworthy enough to stand alone. Teams that cannot generate enough samples per condition to flag credibly, or that need claims about stereotype absence, should treat the diversity score as one input to a review that also includes human evaluation, and say so in the report.

Frequently Asked Questions

What is the recommended metric order for per-class diversity in image generative models?

For per-class diversity in image generative models, an extensive evaluation by Stein et al. (2023) “used the Vendi score of order q=1 and found it to work more effectively as a measure of per-class diversity,” as summarized in the Cousins of the Vendi Score survey (arXiv 2310.12952). Absent contrary evidence for your domain, q=1 is the documented default for per-class work.

At what sample size might the Vendi score fail to converge?

Truncated Vendi Score work (arXiv 2410.21719) demonstrates numerically that “the Vendi score computed with standard sample sizes below 20,000 may not converge to its asymptotic value under infinite sampling” (arXiv 2410.21719).