groundy
Industry & Business

Reusable Prompt Skills vs Ad-Hoc Tweaking: Kling's Text-to-Video Method

SkillPE shows evolved prompt libraries beat static seeds in controlled tests, but production teams need curation owners and transfer tests before adopting the method.

Published 4 references
On this page12 sections

A reusable cinematic prompt library is worth building when your team produces enough recurring work to reuse the craft and can assign someone to curate the references guiding its evolution. Without that curation capacity, ad-hoc tweaking remains the cheaper choice. The strongest evidence for building comes from SkillPE, whose authors report that evolved skill libraries outperform static seeds and external prompt-engineering baselines in their own controlled evaluation, with a small human study pointing in the same direction.

That is not field evidence from production video teams. The evaluation ran on two open-source generators, MiniMax-H3 and LTX-2.5, and the paper reports that libraries selected on one backbone still beat the seed skills on the other. What it does not report is independent replication, or any result covering Sora, Runway, Veo, or non-English prompting. The practical finding is therefore narrower than “libraries always win”: SkillPE makes a credible case for evolving reusable skills, but its method moves work from rewriting prompts into selecting references, reviewing changes, and checking that the original request survives.

Ad-hoc tweaking leaves little behind

The maintenance problem is documented most directly outside video production. Prompt-with-Me, a study of prompt practice in LLM-driven software engineering, reports that prompts are typically written ad hoc and, unlike source code, are rarely versioned, reviewed, or maintained across the development lifecycle. Its cited examination of 243 GitHub repositories found that prompt modifications were sparsely documented and frequently introduced logical inconsistencies.

That does not establish that video teams modify prompts at the same rate or for the same reasons. It is an analogy, not a video-production user study. The relevant shared risk is structural: when a revision is made only to fix the output in front of you, the reasoning behind it can disappear into the next wording change. A later artist or developer inherits the prompt, but not the distinction between an intentional creative choice and an accidental side effect.

Copy-paste prompt packs preserve wording more reliably, but preservation alone is not evolution. They can provide a useful starting vocabulary for composition, lighting, or shot logic while still leaving each team to decide which examples fit a new brief. SkillPE’s comparison with static seed skills is relevant here, although it does not establish that every published prompt pack is equivalent to its seeds. The measured comparison is between the paper’s own static seeds, evolved libraries, and selected external baselines.

What SkillPE makes reusable

SkillPE comes from authors affiliated with HKUST, KlingAI Research, and the University of Hong Kong; the paper notes that the work was conducted during an author’s internship at KlingAI Research. Despite the affiliation, the evaluation does not use Kling’s own generator: it runs on MiniMax-H3 and LTX-2.5, two open-source backbones. The framework represents shot logic, composition, lighting, sound design, and other filmmaking cues in a fine-grained format, starting with expert-authored seeds and evolving them into reusable cinematic skills.

The important distinction is not prompt length. A long, detailed prompt can still be disposable. A reusable skill is intended to carry filmmaking knowledge into later requests, so the team does not reconstruct its camera and lighting language whenever the subject or plot changes.

That reuse creates a tension. A system that only reproduces past choices can become rigid, while one that pursues novelty can wander away from the brief. The SkillPE authors name the gap directly: “systematically evolving reusable cinematic strategies to explore diverse creative realizations while preserving user intent remains largely underexplored.” Intent preservation is therefore not a finishing detail. It is one of the problems a team adopting the method still needs to manage.

Reference triage does the real work

SkillPE’s central curation device is a three-way reference taxonomy:

  • Resonators are good matches.
  • Dissonants are weak matches.
  • Divergents are creatively useful near-misses.

In operational terms, these categories ask different questions of a reviewer. A resonator helps identify what should carry forward. A dissonant supplies a contrast against which a weakness can be recognized. A divergent requires a more difficult judgment: the result departs from the target, but some part of that departure may be worth retaining.

That third category is especially useful for creative work because it prevents every near-miss from being treated as interchangeable failure. It also creates the clearest intent risk. A visually interesting departure is not automatically an acceptable answer to the original request. Someone still has to distinguish productive variation from a changed brief.

This is where the economics of the library become uncertain. Writing fewer prompts does not eliminate expert labor; it redirects that labor toward building reference sets, assigning categories, and reviewing what evolution produces. The paper measures construction compute but not curation labor: Appendix C reports roughly 68.2K model calls and about 625.6 GPU-hours across 2,000 development video rollouts per backbone to build a library, plus two model calls per prompt at inference, or three with MiniMax-H3’s prompt-schema adaptation. No staffing-hours or maintenance-frequency figure appears, so a payback-period calculation remains out of reach. The claim that reference curation becomes the cost center stays an inference from the method’s design, not a separately measured result.

What the evaluation establishes

The SkillPE evaluation uses four dimensions: prompt fidelity, cinematic quality, narrative appeal, and creativity. On its seven-point protocol, the authors report final-library improvements of up to 1.10 points over static seed skills and 1.02 points over the strongest external prompt-engineering baseline. The ten-annotator human study reports that the best SkillPE variant improved the overall score by 0.81 points over the seed skill and 0.71 points over the strongest external baseline, also on a seven-point scale.

These results support a specific conclusion: within the authors’ setup, evolving the skills was better than leaving the seeds static, and the best variants also beat the selected external comparison. “Up to” matters. The figures describe the strongest reported improvements, not a guaranteed gain for every skill, request, or team.

Several limits restrict how far that conclusion travels:

  • All of the numbers come from the framework’s authors; the paper reports no independent replication.
  • Ten annotators provide human confirmation, but not evidence of consistent preferences across larger or more varied audiences.
  • The evaluation runs on two open-source backbones, MiniMax-H3 and LTX-2.5; Sora, Runway, and Veo do not appear in it.
  • The scores do not measure curation effort, production turnaround, or long-term maintenance.
  • The cross-backbone transfer test covers only those two generators, and no result covers non-English prompting.

One of those limits is narrower than it looks, and it cuts in SkillPE’s favor. Appendix F.4 evaluates libraries selected on one backbone against the other: every transferred library outperformed the seed skills on both StoryEval and VBench, with overall gains between 0.42 and 0.98 points, leading the authors to conclude that “SkillPE transfers with positive gain over seed.” That is real evidence the skills are reusable across generators. It is also evidence from exactly two open-source models, both handled inside the authors’ pipeline, so it bounds what a team can assume about Sora, Runway, or Veo.

The most defensible reading is that SkillPE has passed its authors’ own comparative evaluation convincingly enough to justify a controlled team trial. It has not established a universal improvement that can be budgeted into a Sora, Runway, or Veo workflow without testing.

Kling’s own pipeline treats prompting as a staged process

A second Kling-affiliated paper provides useful context, but not another SkillPE benchmark. In Kling’s video-generation pipeline, an agentic prompt-engineering stage runs at inference before basic prompt engineering. It performs story refinement to make the requested plot coherent and retrieves captions of compelling videos as creative evidence. Basic prompt engineering then always writes the final condition, preserving the learned caption dialect regardless of what the agentic stage added.

This supports the broader idea that prompting can be managed as a process with intermediate evidence and explicit responsibilities, rather than treated as one undifferentiated text box. Retrieval of creative evidence also resembles SkillPE’s reference orientation at a high level.

The two mechanisms should not be conflated. Retrieving captions in Kling’s inference pipeline is not the same operation as curating resonators, dissonants, and divergents for skill evolution, and the pipeline paper does not supply another measurement of SkillPE’s effectiveness. Its practical value is to show why prompt wording and supporting evidence may need separate ownership: a system can refine the story or retrieve creative material while still requiring a final stage to express the condition in the generator’s expected dialect.

A text library is not the only route to control

Video-As-Prompt, or VAP, takes a different approach. It uses a reference video as a direct semantic prompt, guiding a frozen Video Diffusion Transformer through a plug-and-play Mixture-of-Transformers expert. Its authors report a 38.7% user preference rate, which they describe as a new state of the art for open-source methods and as rivaling leading condition-specific commercial models.

That figure cannot be compared numerically with SkillPE’s seven-point scores. The studies measure different things, and neither paper reports a head-to-head evaluation. VAP matters to the decision because it changes the options: a team whose main problem is communicating motion or semantics through an existing reference video may benefit more from testing reference-video control than from encoding every intention in text.

A text skill library remains a better candidate when the team needs an inspectable, reusable vocabulary across briefs that do not share a reference video. The choice depends on the available control artifact, not on a claim that one approach has universally displaced the other.

Build or skip?

The following recommendations are an operational interpretation of the evidence, not outcomes measured by the papers.

Your situationRecommended choiceReason
Recurring projects reuse the same cinematic vocabulary, and a reviewer can own reference curationBuild a small SkillPE-style pilotThe authors’ comparisons favor evolved libraries, and recurring work gives the curation effort opportunities to pay off
Work is mostly one-off, or nobody can maintain the reference setStay with ad-hoc prompting or static seeds for nowThe method depends on triage, and its labor cost is unknown
Creative variations increasingly fail to match the original briefPause library evolution and add an intent reviewThe paper identifies intent preservation as underexplored; creative near-misses need a separate acceptance decision
The workflow uses Sora, Runway, Veo, or non-English promptsRequire a transfer test before adoptionThe paper’s cross-backbone result covers only its own two generators; those settings are untested
Existing reference videos express the desired control better than proseCompare a reference-video route with the text libraryVAP demonstrates a competing control method, though not superiority over SkillPE

A useful pilot should include five gates:

  1. Keep a fixed baseline. Compare the evolved library with both the static seed and the team’s current ad-hoc wording on the same representative requests. This separates the value of evolution from the value of simply having a better starting prompt.
  2. Record the triage decision. For each reference, retain its resonator, dissonant, or divergent label and the reviewer’s reason. Without that record, the library can reproduce the same undocumented drift as ad-hoc prompting.
  3. Check intent separately from appeal. Review which parts of the original request must remain unchanged, then inspect whether an evolved skill or its output alters them. A visually stronger result should not silently pass if it answers a different brief.
  4. Version skills and references together. A later reviewer needs to know which reference set accompanied a skill change. This is a direct response to the maintenance failure documented by Prompt-with-Me, adapted here as a preventive practice rather than a measured video-team intervention.
  5. Test portability independently. Run the transfer trial on each generator and prompt language that matters to production; Appendix F.4 covers only the paper’s own two backbones, so anything beyond them is untested. Treat failure to transfer as a local result requiring revision, not as proof that the entire framework fails elsewhere.

Where adoption is likely to break

The first failure mode is curation debt. A team can approve the library while leaving reference selection as unassigned work. Reviews then become inconsistent, stale examples remain influential, and the nominal skill library turns into another collection of prompts whose history nobody can reconstruct. The evidence does not establish how often this happens; it follows from the amount of judgment the taxonomy requires.

The second is intent drift. Divergents are valuable precisely because a near-miss can contain something creatively useful. That makes them poor candidates for automatic acceptance. If novelty is allowed to override the brief without review, the library can become better at producing interesting variations while becoming less reliable at fulfilling requests. The authors’ own ablation gives the trade-off its shape: as mutations scale from Bold to Wilder to Extreme, creativity rises monotonically on StoryEval (4.91 to 5.50 to 5.88) while prompt fidelity falls, and the paper states that assessed selection “mitigates this trade-off but does not eliminate it.” Prompt-fidelity scores in their study therefore do not remove the need for this check in a different workflow.

The third is assumed portability. Kling’s separate pipeline preserves a learned caption dialect when writing the final condition, which is a reminder that generators can have their own prompting conventions; even inside SkillPE’s own setup, MiniMax-H3 required an extra prompt-schema adaptation call that LTX-2.5 did not. Appendix F.4 shows transfer between the paper’s two backbones, which is encouraging, but it covers one narrow pair. Neither fact is direct evidence about Sora, Runway, or Veo, so testing still beats assuming that the same library will behave identically elsewhere.

I would build the library for recurring cinematic work only with a named curation owner and an intent-review gate. Start with a bounded pilot, retain static seeds and ad-hoc prompts as baselines, and judge the maintenance record alongside output quality. If nobody can sustain reference triage, skipping the library is the more honest decision. For generators beyond the paper’s two backbones, other languages, or workflows centered on reference footage, transfer tests and a reference-video comparison should come before a broader commitment.

Frequently Asked Questions

What are the three categories in SkillPE’s reference taxonomy?

  • Resonators are good matches.
  • Dissonants are weak matches.
  • Divergents are creatively useful near-misses.

How much compute did SkillPE use to build a library?

Appendix C reports roughly 68.2K model calls and about 625.6 GPU-hours across 2,000 development video rollouts per backbone to build a library, plus two model calls per prompt at inference, or three with MiniMax-H3’s prompt-schema adaptation.

Stay with ad-hoc prompting or static seeds for now

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. SkillPEarxiv.orgAccessed
  2. Prompt-with-Mearxiv.orgAccessed
  3. Kling's video-generation pipelinearxiv.orgAccessed
  4. Video-As-Prompt, or VAParxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy