Fine-Tuning vs RAG for Internal APIs: StarCoder2 Constraints

Fine-tuning an open coding model on your own repositories is the only one of the three standard fixes for internal-API hallucination that puts proprietary function signatures into the model’s weights, and it converts a context-management problem into a dataset-curation problem with a refresh loop your team owns indefinitely. RAG over the codebase is cheaper to start but caps out at the base model’s attention budget. Routing to a shared frontier API outsources both problems along with your source code.

Why does a private coding assistant hallucinate internal APIs in the first place?

A coding assistant hallucinates internal APIs because every open code model and every frontier model was pretrained on public code, so a call to your internal SDK is statistically indistinguishable from a call to any of the thousands of public SDKs the model has actually seen: it completes the shape of the call and invents the arguments.

The failure is not random noise. Completion models are pattern matchers over a training distribution, and the distribution contains no trace of your billing.settle_invoice() or your internal deployment client. When the model encounters payments.client., the nearest public neighbor fills the gap. The result is a call that looks right in review and fails at runtime, which is the worst possible failure mode for an autocomplete tool because it burns reviewer trust in exactly the senior engineers the tool was supposed to help.

Teams converge on three responses. Fine-tune an open model on the organization’s repositories so internal APIs live in the weights. Retrieve codebase context at inference time and prepend it to the prompt, the RAG path. Or route to a shared frontier API and lean on a large hosted context window. Each moves the cost somewhere different, and the rest of this piece prices each move.

The fine-tune path has a reference methodology in Hugging Face’s personal-copilot recipe for continuing an open model’s training on a team’s own code. The specifics of that recipe, its dataset-curation pipeline and its measured outcomes, are not covered by the sources verified for this article [unverified], so the analysis below works from the base model’s documented constraints instead. Those constraints turn out to be more decisive than any recipe.

What do StarCoder2’s context limits actually allow?

StarCoder2 accepts prompts up to 16,384 tokens but applies a sliding attention window of only 4,096 tokens, which means the model can ingest a large prompt while only attending across a 4,096-token neighborhood at generation time, and that gap is the hard bound on how much retrieved internal code a RAG prompt can exploit.

The architecture, documented in the StarCoder2 repository, uses Grouped Query Attention across the family. Sliding-window attention is the load-bearing detail: a token generated late in the sequence attends back roughly 4,096 positions, not the full 16,384. Context placed at the head of a long prompt is, for practical purposes, invisible by the time the model is completing your function call. Retrieval pipelines built on the assumption of full attention over the whole window silently degrade on this base.

The family ships in three size tiers, and the pretraining budget scales with each. According to the StarCoder2 announcement, the 3B and 7B variants were trained on 17 programming languages drawn from The Stack v2, on 3+ trillion and 3.5+ trillion tokens respectively. The 15B variant, per its model card, was trained with a Fill-in-the-Middle objective on 4+ trillion tokens.

On capability, the announcement claims the 15B is best in its size class and matches 33B+ models on many evaluations, and that the 3B matches the original StarCoder-15B. Both claims come from the project’s own blog post and both are measured on public code benchmarks. Neither measures the thing a private assistant buyer cares about, which is hallucination rate against APIs the model has never seen. Treat the parity claims as evidence that the base is competent at public code, not as evidence about your codebase.

The organizational provenance is worth one line because it anchors the license discussion below: BigCode is an open scientific collaboration jointly led by Hugging Face and ServiceNow, and Hugging Face was founded in 2016 in New York City by Clément Delangue, Julien Chaumond, and Thomas Wolf, per Wikipedia’s entry. This is a decade-old organization with a documented governance posture, not an anonymous model drop.

What does fine-tuning on your own code actually require?

Fine-tuning for an editor assistant requires preserving the Fill-in-the-Middle objective the base model was trained with, curating an internal dataset that teaches your APIs in that format, and staffing a refresh loop that retrains every time the codebase drifts.

The FIM detail is easy to skip and expensive to get wrong. StarCoder2-15B was trained with Fill-in-the-Middle on 4+ trillion tokens, per its model card: training documents are split into prefix, middle, and suffix, and the model learns to generate the middle conditioned on both sides. That objective is what makes the model useful inside an editor, where the cursor sits mid-file with code below it. A fine-tuning set built as plain left-to-right completions of your internal files teaches the model your APIs while quietly eroding the infill behavior you needed. The dataset has to reproduce the FIM structure on your own code, which means a curation pipeline that splits files into prefix-suffix-middle triples, samples them sensibly, and filters out generated code, vendored dependencies, and secrets.

That pipeline is the real cost of the fine-tune path, and it is a permanent cost. Weights are a snapshot. Internal APIs change weekly, and every deprecation the model memorized becomes a fresh hallucination source, this time one you installed yourself. The refresh cadence question, how often you retrain and how you decide the codebase has drifted enough to justify it, has no vendor to outsource it to. When you routed to a frontier API, the vendor absorbed model refreshes and you complained about the price. Fine-tuning inverts that arrangement: the price drops, the refresh becomes your job.

The size tiers set the hardware floor for that loop. The 3B matching StarCoder-15B, per the project’s announcement, means the smallest tier is a legitimate inference target for in-house serving, with the 7B and 15B available when the internal-API coverage of the small model proves thin. Which tier you fine-tune is an empirical question about your codebase, and answering it requires an evaluation set built from your own repositories, because no public benchmark in the verified sources measures internal-API accuracy on any of these models.

Where does RAG over the codebase break?

RAG over the codebase breaks at the base model’s attention budget: StarCoder2 accepts 16,384 tokens of prompt, but the 4,096-token sliding attention window means retrieved chunks placed far from the generation point contribute weakly, so retrieval alone cannot guarantee internal-API accuracy.

The mechanics follow directly from the documented architecture. A naive retrieval pipeline prepends the top-k relevant files to the prompt. On a full-attention model with a long window, that works. On this base, the chunks at the top of a 16K-token prompt fall outside the attention reach of the tokens generated at the bottom. The model saw your internal SDK’s source and cannot use it.

Mitigations exist and they all cost something. Placing the single most relevant chunk adjacent to the generation point keeps it in reach, but then you are showing the model one file’s worth of internal context, not the codebase. Keeping the whole prompt inside roughly the 4,096-token effective window preserves attention quality by shrinking retrieved context to a couple of files. Better re-ranking improves which chunk lands in reach without changing how many can. Every mitigation trades away exactly the internal-API coverage the retrieval layer was supposed to provide.

RAG keeps two real advantages. Refresh is nearly free: re-index the repository and the next completion sees today’s API, no training run required. And there is no dataset-curation pipeline to staff. On the available evidence, though, the cheap RAG-only path is unproven for the failure mode that motivates this whole exercise. No source verified for this article measures internal-API hallucination rates under RAG on StarCoder2, and the sliding-attention arithmetic points the wrong direction. Claims that retrieval alone solves internal APIs on this base are, as of 2026-07-28, assertions without published measurements behind them.

What license and provenance exposure comes with StarCoder2?

StarCoder2-15B ships under a use-based license, per its model card, rather than a permissive MIT-style one, so commercial deployment of a fine-tuned private assistant is gated by acceptable-use restrictions that a counsel review has to clear before anything ships.

The distinction matters operationally. Permissive licenses impose conditions on redistribution. Use-based licenses impose conditions on what you do with the model, full stop, which reaches deployments that never redistribute anything. A team that fine-tunes StarCoder2 and serves it internally is making a licensed use, and whether that use is permitted is a legal question, not an engineering one.

Provenance is the quieter exposure. The pretraining corpus, The Stack v2, is derived from the Software Heritage archive, the largest public archive of source code and development history, according to the StarCoder2 announcement. Two consequences follow. First, the base model’s upstream license posture is unusually well documented: 17 languages, named archive, published token counts per tier. A compliance team can audit what went in. Second, your internal code is definitionally absent from that corpus, because Software Heritage archives public code. That absence is the entire reason the fine-tune path exists, and it also means nothing in the base model’s provenance chain covers anything your assistant learns from your repositories. That data’s provenance is your responsibility from the first training example.

The routing option rearranges these exposures rather than eliminating them. Send code to a frontier API and the training-data provenance problem becomes the vendor’s, accepted sight unseen under their terms, while a new problem appears: your proprietary source leaves your perimeter with every request. Fine-tuning keeps source in-house and hands you an auditable base, at the price of owning the license review and the corpus hygiene. There is no arrangement where nobody owns the legal surface.

Which approach should a team pick?

Fine-tuning is the only approach with a working mechanism for internal-API accuracy on this base model, RAG is the right choice when refresh cadence dominates and coverage requirements are modest, and routing to a frontier API is the choice that trades source-code custody for zero operational ownership.

The verdict rests on the constraints, not on benchmark theater. The 4,096-token sliding attention window documented in the StarCoder2 repository caps what retrieval can deliver, and no verified source shows RAG closing the internal-API gap on this base. Fine-tuning has a mechanism that matches the failure mode: the APIs the model hallucinates are the ones missing from its training data, so put them in its training data. The cost of doing so is real and permanent: a FIM-preserving curation pipeline, an internal evaluation harness, a retraining cadence, and a license review. If your team cannot staff that loop, the honest options are RAG with calibrated expectations or a frontier API with a data-handling agreement, not a fine-tune you will let rot.

The base is deployable and bounded. The 15B’s claimed parity with 33B+ models on public evaluations, the 3B’s claimed parity with the previous generation’s 15B, and the 4+ trillion tokens of FIM training all say the raw capability is there. What the public record does not say is whether your codebase, your refresh cadence, and your license posture fit inside those bounds. That part is yours to measure, which is the actual price of owning the model.

Frequently Asked Questions

Can I use StarCoder2-3B for a private assistant if my team lacks GPU budget?

Yes. The 3B variant matches the original StarCoder-15B on public benchmarks, which lowers the hardware floor for in-house serving. This allows teams to run a private assistant on consumer-grade GPUs or smaller cloud instances, though the 7B or 15B tiers may be needed if the smaller model lacks coverage for complex internal APIs.

Does the BigCode OpenRAIL-M v1 license allow me to sell access to my fine-tuned assistant?

The license is use-based, not permissive, meaning it imposes restrictions on how you deploy the model commercially. You must review the specific acceptable-use clauses before shipping, as the license gates the deployment itself rather than just redistribution. Legal counsel should clear the commercial use case before the fine-tuning run begins.

How often should I retrain my fine-tuned model to prevent hallucinations?

Retrain whenever the internal codebase drifts enough to introduce new APIs or deprecate old ones that the model still memorizes. Since internal APIs change weekly, the refresh cadence is a permanent operational cost you own. There is no vendor to absorb this; you must staff the curation pipeline and evaluation loop to keep the weights aligned with current code.

Why does RAG fail to retrieve internal context effectively on StarCoder2?

StarCoder2 uses a 4,096-token sliding attention window within its 16,384-token context limit. Chunks placed at the start of a long prompt fall outside the attention reach of tokens generated at the end. This means retrieved code often cannot influence the completion, making retrieval alone insufficient for internal API accuracy without aggressive chunking or re-ranking.