MakeSubs

MakeSubs turns uploaded videos into subtitles automatically. Drop an MKV file, watch it play back in the browser, and receive a timed caption file when processing finishes — with separate speakers labeled when the audio supports it.

The product is still a prototype, but the pipeline is clear: do as much as possible on the client, send only the audio that matters to the server, and run a speech stack tuned for messy real-world footage rather than clean podcasts.

The problem

Subtitle workflows usually assume you already have a clean audio file and a desktop tool like Subtitle Edit or Aegisub. For most people that is too much friction — and for MKV rips, TV captures, or clips with background music, the audio is nowhere near clean enough for off-the-shelf transcription.

Whisper is excellent at multilingual speech, but it hallucinates when music and effects leak into the input. Long files need chunking, speakers need separating, and you still want to see the video while you wait. MakeSubs is built around that full chain, not just “run Whisper on the upload.”

What it does

The intended flow is straightforward from the outside:

1. Drag and drop an MKV video file
2. The browser analyzes the container locally — tracks, codecs, duration, cluster layout
3. The video track is extracted and played back in-page with mp4box.js
4. The audio track is extracted and sent to the backend
5. The server runs voice separation, cleanup, and Whisper transcription
6. Finished SRT subtitles return to the browser for preview and download

Behind that UI sits speaker diarization, voice-activity chunking, and per-chunk language detection so captions stay readable on dialogue-heavy material with multiple voices and languages.

Client-side processing

Browsers cannot assume every upload is a simple MP4 with AAC audio. The prototype focuses on Matroska because that is what shows up in the wild — multiple tracks, AC-3 audio, long runtimes.

Container inspection

On drop, the client streams the file through an EBML parser and builds a map of tracks and audio clusters without uploading the whole video first. You see container metadata, codec details, and progress while the file is read in chunks.

Video playback

The video track is remuxed for playback with mp4box.js, so the user can scrub the source footage while subtitles are generated. Keeping playback client-side avoids storing full video files on the server when only audio is needed for ASR.

Audio extraction

Audio is pulled out of the container in the browser — including AC-3 payloads decoded through a WebAssembly build of liba52. The prototype proved that cluster-level extraction works; the product path sends a normalized audio stream to the backend rather than the raw MKV.

Backend pipeline

Heavy ML stays on the server. The processing chain is sequential, with each stage feeding the next:

Voice separation

Incoming audio passes through Mel-Band RoFormer vocal separation (via audio-separator). TV rips and clips with background music leave music in the stem that Whisper will confidently mishear; isolating vocals first materially reduces garbage captions.

Voice activity detection

Before separation and transcription, Silero VAD finds speech regions and splits on natural pauses. Chunks are merged to a minimum duration so the separator and ASR models always receive enough context, but never more than a safe maximum per pass. This keeps GPU memory predictable on hour-long files.

Transcription

Each chunk is transcribed with faster-whisper using multilingual mode and automatic language detection per segment. Timestamps from chunk boundaries are shifted back onto the full timeline so the final SRT is continuous.

Diarization

pyannote speaker diarization runs on the cleaned vocal stem to identify who spoke when. Segments are aligned with Whisper output so subtitles can prefix lines with speaker labels — useful for interviews, dubbing sessions, and multi-character scenes where a single block of text is not enough.

Cleanup and delivery

A light post-processing pass trims silence, merges overlapping cues, and normalizes line breaks. The result is returned to the client as standard SRT, ready to preview alongside the mp4box.js player or export for editing elsewhere.

Architecture

Browser                         Backend
───────                         ───────
MKV drop
  → probe tracks (EBML)
  → extract + play video (mp4box.js)
  → extract audio        ──────→  voice separation (RoFormer)
                                 VAD chunking (Silero)
                                 diarization (pyannote)
                                 transcribe (Whisper)
                           ←────  SRT + speaker labels
preview / download

The split is intentional: the client owns container parsing and preview; the server owns GPU inference. Video never has to touch the backend unless you later add features that require it.

What the prototype proved

Early experiments in the repo validated the hardest client-side pieces — parsing multi-gigabyte MKV files in the browser, walking cluster boundaries, and decoding AC-3 audio through WASM without sending the full file upstream. Separate Python scripts exercised the server-side chain: RoFormer separation with VAD-guided chunking, then faster-whisper with per-chunk language detection.

The remaining work is wiring those pieces into one product: a job API, progress streaming back to the UI, mp4box.js remux for playback, and pyannote integrated into the same pipeline as Whisper.

What I learned

Subtitle generation is not a single model problem. The quality ceiling is set long before Whisper runs — by how you split the file, how cleanly you isolate speech, and whether you know there are multiple speakers.

Client-side container parsing is worth the complexity. Users already have the file; reading it locally gives instant feedback and keeps privacy-sensitive footage off a server until they explicitly submit audio for processing.

Stack

Client

• React, Astro
• ebml-stream — MKV container parsing
• mp4box.js — in-browser video remux and playback
• WebAssembly — AC-3 decode (liba52)

Backend

• faster-whisper — multilingual ASR
• Silero VAD — speech chunking
• Mel-Band RoFormer — vocal separation
• pyannote — speaker diarization
• Python, PyTorch