Fish-Speech: The Open-Source TTS Model That's Threatening ElevenLabs

Fish Audio’s open-weights Fish-Speech S2 model posts the strongest reported numbers on the Audio Turing Test (0.515 with instruction rewriting) and EmergentTTS-Eval (an 81.88% overall win rate), serves time-to-first-audio under 100ms, and covers roughly 80 languages in a single model. For practitioners considering ElevenLabs at current pricing, S2 resets the evaluation. The catch is the license: free for research, paid for commercial deployment.

What Is Fish-Speech?

Fish-Speech is the open-source TTS project maintained by Fish Audio, an AI audio startup. The company’s comparison pages put its GitHub repository at more than 22k stars (Fish Audio), among the most-watched open speech synthesis repos.

Fish Audio announced S2 on March 10, 2026 (UBOS). Unlike prior releases, S2 ships with model weights, fine-tuning code, and the production inference engine, plus a technical report on arXiv. The company is blunt about what this is: “S2 is open weights, not open source by the OSI definition” (Fish Audio). The hosted platform has since moved on to a newer S2.1 Pro, which Fish Audio advertises as the model powering its product (Fish Audio); nothing cited here indicates S2.1 Pro has been released as open weights. The open release, and every benchmark below, concern S2, distributed as fishaudio/s2-pro on Hugging Face.

The company also runs a commercial API alongside the open weights, and it is explicit about why: “Commercial licensing is the way we fund continued development” (Fish Audio). Its licensing post measures the S2 release against Llama 4, DeepSeek R1, Mistral Large 3, and GPT-OSS, which tells you which cohort Fish Audio puts itself in. The dual track, open weights plus paid products, is the same balancing act those labs run: give researchers everything, charge the companies that ship.

How Does Fish-Speech Work?

Architecture

Fish-Speech S2 is built on a Dual-AR (Dual-Autoregressive) design with three primary components (Fish Audio, “Fish Audio S2 Technical Report”):

• A Slow AR backbone (Qwen3-4B) that processes interleaved text and reference audio tokens, autoregressively generating the primary semantic codebook
• A Fast AR decoder, specified in the report as a 4-layer Transformer with independent weights and embedding tables, that generates the remaining nine residual acoustic codebooks at each time step. The report stops at the layer count; the roughly 400M parameters usually attached to this stage come from BentoML’s roundup and a launch write-up (BentoML; DEV Community)
• A DAC-based audio tokenizer with an EVA-GAN decoder (446M parameters in total, per the report) that reconstructs waveforms from the full discrete token sequence

For production serving, S2 integrates with SGLang and inherits LLM-native optimizations including continuous batching, paged KV cache, CUDA graph replay, and RadixAttention-based prefix caching. Because reference audio is prepended to the system prompt, repeated requests with the same voice hit the prefix cache heavily: Fish Audio reports an average hit rate of 86.4%, peaking above 90% (Fish Audio, “Fish Audio S2 Technical Report”).

Zero-Shot Voice Cloning

The standout capability is zero-shot voice cloning: provide a short reference clip and S2 replicates that voice across any script, in any supported language, without fine-tuning. Third-party launch coverage puts the useful range at roughly 10-30 seconds (DEV Community); Fish Audio’s own site says “as little as 10 seconds” (Fish Audio). The model was trained on over 10 million hours of audio across approximately 80 languages and dialects, which enables cross-lingual transfer: clone an English voice and read Japanese copy (Fish Audio, “Fish Audio S2 Technical Report”).

Local deployment follows the repo’s documented workflow (DEV Community):

git clone https://github.com/fishaudio/fish-speech.git
cd fish-speech && uv sync

python -m fish_speech.text_to_speech \
  --text "The market shifted overnight." \
  --reference_audio speaker_sample.wav \
  --output output.wav

Emotion Control: Two Different Rulebooks

S2’s most differentiated feature is inline emotion control, and it matters that the rules differ by surface. The open model accepts free-form natural-language descriptions embedded at specific word positions, such as [whisper in small voice] or [pitch up], rather than a fixed tag library; one launch write-up counts 15,000+ recognized tags covering emotion, tone, volume, and rhythm (DEV Community), and BentoML’s roundup describes the same open-ended, word-position control (BentoML).

The hosted API is stricter. Fish Audio’s own best-practices doc requires emotion tags in parentheses at the beginning of sentences, tone controls and sound effects anywhere, and spelling that exactly matches the supported list; it explicitly advises against creating custom tags (Fish Audio Docs). A script that works against the open weights may not behave the same through the API:

(whispering) The acquisition closed this morning.
Nobody outside this room knows yet.
(urgent) Keep it that way.

The benchmark backing the expressivity claim is real, with caveats worth knowing. On EmergentTTS-Eval, judged by Gemini 2.5 Pro in side-by-side comparisons against a gpt-4o-mini-tts baseline, S2 achieves an 81.88% overall win rate and leads specifically on paralinguistic tasks at 91.61% (Fish Audio, “Fish Audio S2 Technical Report”). Those are LLM-as-judge numbers against one baseline, not a human panel.

Fish-Speech vs. ElevenLabs: The Benchmark Picture

ElevenLabs is the benchmark target the community uses, so here is what the numbers actually show.

These benchmarks deserve nuance in both directions. An independent hands-on review of both platforms concluded that ElevenLabs “wins on pure voice quality” while Fish.audio “wins on cost efficiency and speed” (SaaS24 Reviews); note the vintage, though, because that reviewer tested Fish’s previous S1 model against ElevenLabs v3, so the verdict predates both companies’ current flagships. Fish Audio’s own technical report concedes that MiniMax-Speech and ElevenLabs “still maintain an advantage in certain low-resource languages,” typically those with under 1,000 hours of training data, even as S2 leads on Seed-TTS-Eval and the CV3-Eval multilingual subset (Fish Audio, “Fish Audio S2 Technical Report”). S2’s edge is strongest in paralinguistic expressivity, breadth of language coverage, and price.

Why Does Fish-Speech Matter?

The Economics Squeeze ElevenLabs From Below

ElevenLabs ended 2025 with $350 million in ARR (ElevenLabs). By May 2026 it had crossed $500M ARR (ElevenLabs), announcing a third close of its Series D that added investors including BlackRock, Wellington, NVIDIA (via NVentures), and Santander, alongside a $100 million tender offer and a team of 530 across more than 50 countries (ElevenLabs). That is a legitimate business, growing fast. But BentoML’s roundup prices Fish Audio’s hosted API at roughly $15 per 1M characters against roughly $165 for ElevenLabs (BentoML), and the research license explicitly invites teams to download the weights and benchmark on their own infrastructure before paying anyone (Fish Audio). Pricing power built on a quality premium does not survive a competitor that matches enough of the quality at a fraction of the price. What premium remains has to come from the layer open weights cannot replicate: compliance, SLAs, integrations, and a customer list that increasingly overlaps with the investor list.

What Open-Source TTS Competition Looks Like Now

Fish-Speech is not the only pressure point, and the licensing landscape is more varied than a simple open-versus-closed split (BentoML):

• Kokoro (Apache 2.0): 82M parameters, fast on modest hardware. Its decoder-only design trades away some expressive control for speed.
• XTTS-v2 (Coqui Public Model License): zero-shot voice cloning from a 6-second clip across 17 languages, restricted to non-commercial use. The original company shut down (BentoML dates it to early 2024), but the toolkit did not die with it: Idiap Research Institute maintains an active fork at idiap/coqui-ai-TTS, republished as the coqui-tts PyPI package, with releases through 0.27.x adding cached cloned voices and OpenVoice/kNN-VC voice conversion (Idiap/coqui-ai-TTS).
• Dia2 (Nari Labs, Apache 2.0): dialogue-first, English-only, streaming from the first tokens, with voice cloning via an audio prompt and [S1]/[S2] speaker tags.
• Chatterbox-Turbo (Resemble AI, MIT): a 350M-parameter model with a distilled one-step decoder, sub-200ms latency, an emotion exaggeration control, native paralinguistic tags, and PerTh watermarking on all output. English-only, with a separate Chatterbox-Multilingual model.
• Fish-Speech S2: the quality leader on the benchmarks above with the broadest language coverage, but research-licensed for anything commercial.

The pattern worth noticing: permissive licenses are common in open TTS, but rare at the top of the voice-cloning quality table, where both S2 and XTTS-v2 restrict commercial use. Chatterbox and Dia2 are the closest permissive options with cloning, each with narrower language coverage. Fish Audio’s own component table makes the trade explicit: S2 releases weights, fine-tuning code, the inference engine, and a full technical report, but withholds free commercial use, where the same table lists DeepSeek R1 as MIT-licensed and Mistral Large 3 as Apache 2.0 (Fish Audio).

Where the Latency Numbers Apply

S2’s published serving numbers are server numbers: on a single NVIDIA H200, RTF of 0.195, time-to-first-audio under 100ms, and 3,000+ acoustic tokens per second under load (Fish Audio, “Fish Audio S2 Technical Report”). BentoML’s caveat is the one to plan around: self-hosted performance varies with hardware and configuration, and the ~100ms TTFA should not be assumed on smaller GPUs (BentoML). The previous Fish-Speech generation demonstrated consumer-GPU viability with real-time factors around 1:5 on an RTX 4060 mobile and 1:15 on an RTX 4090 (FISH-SPEECH, arXiv), but S2’s team has not published equivalent consumer figures, so treat any specific self-hosted latency claim as untested until benchmarked on your hardware.

For developers who do self-host, the economics are increasingly compelling relative to per-character API billing. For those who prefer the managed route, Fish Audio’s API is listed at roughly $15 per 1M characters (BentoML).

Security matters here, not just cost. Voice cloning at this quality level (a short reference clip, zero fine-tuning) raises immediate concerns for deepfake audio, identity fraud, and social engineering. Fish Audio does not carry the institutional trust and compliance infrastructure an enterprise SaaS provider maintains, so deployers own consent and verification controls. For a concrete illustration of the threat model, the Mercor breach handed voice-clone attackers 40,000 pre-verified targets, each pairing a verified voice sample with the government ID used in screening. That is exactly the raw material zero-shot cloning pipelines like S2 consume.

What ElevenLabs Still Does Better

Credit where it is due: ElevenLabs keeps real advantages.

Voice cloning fidelity for studio-quality input. Independent reviewers describe ElevenLabs output as more natural for narration, with Fish.audio stronger on cost and speed (SaaS24 Reviews). Fish Audio’s own multilingual results show it trailing in some low-resource languages (Fish Audio, “Fish Audio S2 Technical Report”).

Enterprise infrastructure. Compliance certifications, SLAs, and organizational trust are not things you can open-source. The May 2026 investor update names enterprises like NVIDIA, Salesforce, Santander, KPN, and Deutsche Telekom both using the platform and, in several cases, investing in the company (ElevenLabs). That is the moat open weights do not erode.

Velocity and stability. ElevenLabs’ site now promotes Eleven v4, its most emotive model yet (ElevenLabs), while commercial APIs keep endpoints stable. Open-source models iterate rapidly, which is a feature for researchers and a risk for production pipelines that cannot absorb breaking changes.

The honest summary: on published benchmarks S2 leads; in subjective narration quality and enterprise readiness, ElevenLabs still does. What changed in 2026 is that the open side of the ledger is no longer a generation behind, and the license terms, not the model quality, are now the main gate on commercial adoption.