How to Merge Chunked Whisper Transcripts Without Duplicate or Lost Words

Short answer: split the audio with an overlap, then join the chunks by aligning the words in the overlap, not by cutting on timestamps. Use a local sequence alignment (Smith-Waterman) on normalized words, splice at the middle of the matched region, and fall back to a time cut only when too few words match. If you use diarization, map each chunk’s speaker labels onto the previous chunk’s labels by majority vote over the aligned words. And when an overlap has no matching words at all, stop and report an error, because that almost always means a chunk offset is wrong.

The rest of this post explains why each step is there, with TypeScript you can adapt.

Why transcripts end up in chunks

Whisper models process audio in 30-second windows. Libraries hide that for a single file, but most real pipelines still split long recordings themselves:

• to transcribe several parts of a recording in parallel
• to stay under request size or time limits on hosted APIs and serverless jobs
• to retry one failed part instead of a whole three-hour recording
• to run diarization (WhisperX, pyannote) on pieces that fit in memory

The chunks almost always overlap by a few seconds. Without overlap, a cut can land in the middle of a word, and words right at the edge of a chunk get transcribed with no context after them, so they are the least accurate words in the chunk.

So you end up with something like this: chunk 1 covers 0:00 to 10:05, chunk 2 covers 10:00 to 20:05, and five seconds of speech exist in both.

Three ways that look right and aren’t

Joining end to end. Every overlap appears twice. A five-second overlap every ten minutes adds about a dozen duplicated words per seam, which is easy to miss in testing and very obvious to a reader.

Cutting at a timestamp. “Keep chunk 1 up to 10:02.5, take chunk 2 from there.” The problem is that the two chunks don’t agree on timing, or even on the words. Chunk 1 might hear “we’re going to ship it” where chunk 2 hears “we’re gonna ship it”, with word boundaries a few hundred milliseconds apart. A hard time cut will drop a word or keep half of each version.

Removing duplicates by matching text. “Find the last sentence of chunk 1 in chunk 2 and cut there.” This breaks the moment one side mishears a word, adds a comma, or capitalizes differently, which in an overlap happens all the time.

What you need is a match that tolerates small differences and still finds where the two chunks agree.

Step 0: put every chunk on one timeline

Whisper reports timestamps relative to the start of the audio it was given. Before comparing anything, shift each chunk by its start offset in the original recording:

type Word = { text: string; start: number; end: number; speaker?: string };

function toAbsolute(words: Word[], offsetSeconds: number): Word[] {
    return words.map((w) => ({ ...w, start: w.start + offsetSeconds, end: w.end + offsetSeconds }));
}

Also flatten whatever Whisper dialect you have into this one word list. openai-whisper and faster-whisper put words under segments[].words[] with probability, WhisperX uses score and adds speaker, and the OpenAI API’s verbose_json has a top-level words[]. If a segment has no word timings, spread its duration across its words so every word has a start and an end.

Step 1: compare normalized words

Two words should count as “the same” when they differ only in case, punctuation or accents:

const norm = (s: string) =>
    s
        .normalize("NFKD")
        .replace(/\p{M}/gu, "") // accents
        .toLowerCase()
        .replace(/[^\p{L}\p{N}']/gu, ""); // punctuation

Add a time check too. Common words like “the” and “and” repeat constantly, so two words should only match if their start times are close, for example within 1.5 seconds once both chunks are on the same timeline.

Step 2: align the overlap

Take the last few hundred words of the transcript so far (A) and the first few hundred words of the next chunk (B). Then find the best local alignment between them. Smith-Waterman is the standard algorithm for this. It comes from DNA sequence matching, and it fits this problem well: it finds the region where the two sequences agree and tolerates a few mismatches, insertions and deletions inside that region.

const MATCH = 2;
const MISMATCH = -1;
const GAP = -1;

function align(a: Word[], b: Word[], toleranceSeconds = 1.5): [number, number][] {
    const same = (i: number, j: number) =>
        norm(a[i].text) === norm(b[j].text) && Math.abs(a[i].start - b[j].start) <= toleranceSeconds;

    const n = a.length;
    const m = b.length;
    const H = Array.from({ length: n + 1 }, () => new Int32Array(m + 1));
    let best = 0;
    let bi = 0;
    let bj = 0;

    for (let i = 1; i <= n; i++) {
        for (let j = 1; j <= m; j++) {
            const diag = H[i - 1][j - 1] + (same(i - 1, j - 1) ? MATCH : MISMATCH);
            H[i][j] = Math.max(0, diag, H[i - 1][j] + GAP, H[i][j - 1] + GAP);
            if (H[i][j] > best) {
                best = H[i][j];
                bi = i;
                bj = j;
            }
        }
    }

    // Walk back from the best cell and collect the matched word pairs.
    const pairs: [number, number][] = [];
    let i = bi;
    let j = bj;
    while (i > 0 && j > 0 && H[i][j] > 0) {
        const isSame = same(i - 1, j - 1);
        if (H[i][j] === H[i - 1][j - 1] + (isSame ? MATCH : MISMATCH)) {
            if (isSame) {
                pairs.unshift([i - 1, j - 1]);
            }
            i--;
            j--;
        } else if (H[i][j] === H[i - 1][j] + GAP) {
            i--;
        } else {
            j--;
        }
    }
    return pairs;
}

This is O(n × m), so limit the window. A few hundred words per side covers any sensible overlap, and the cost stays in the low milliseconds.

Step 3: splice at the middle of the match

Once you have the matched pairs, you need to pick one place to switch from A to B. Pick the middle matched pair:

function splice(a: Word[], b: Word[], pairs: [number, number][]): Word[] {
    const [ai, bj] = pairs[Math.floor(pairs.length / 2)];
    return [...a.slice(0, ai + 1), ...b.slice(bj + 1)];
}

Why the middle and not the first or last match? Words near the end of chunk A were transcribed with little audio after them. Words near the start of chunk B were transcribed with little audio before them. Both are at their least reliable at their own edge. Splicing in the middle means each chunk contributes the half of the overlap it heard with context on both sides.

Every misheard word and every punctuation difference inside the overlap is resolved by this choice. Words before the splice come from A, words after it come from B, and nothing appears twice.

Step 4: decide what to do when the match is weak

Real data will hit these cases, so handle them on purpose:

Situation	What to do
No time overlap at all	Append B as it is
Overlap, but fewer than ~3 matched words	Cut at the midpoint of the overlap in time, and log that this seam was cut by time instead of aligned
B falls entirely inside A	If its words match A, it is a duplicate chunk: drop it. If they don’t match, stop with an error
Long overlap with zero matched words	Stop with an error. This almost always means a wrong offset, and guessing produces a transcript that is quietly wrong

The last rule matters most. A merge that fails loudly gets fixed in five minutes. A merge that silently shifts half a transcript by ten minutes gets found by a customer.

Step 5: keep speaker labels consistent

WhisperX and pyannote label speakers per chunk, and the labels mean nothing across chunks. SPEAKER_00 in chunk 1 can easily be SPEAKER_01 in chunk 2. Simply concatenating gives you a transcript where people swap identities every ten minutes.

The alignment from step 2 gives you a way to fix this. Every matched word pair is the same spoken word, heard by both chunks, so its two speaker labels should refer to the same person. Count the pairs and assign each label in B to the label in A it most often lines up with:

function mapSpeakers(a: Word[], b: Word[], pairs: [number, number][]): Map<string, string> {
    const votes = new Map<string, number>(); // "bLabel|aLabel" -> count
    for (const [i, j] of pairs) {
        const sa = a[i].speaker;
        const sb = b[j].speaker;
        if (sa && sb) {
            const key = `${sb}|${sa}`;
            votes.set(key, (votes.get(key) ?? 0) + 1);
        }
    }

    // Strongest agreements first, one to one, so two people never share a label.
    const map = new Map<string, string>();
    const taken = new Set<string>();
    for (const [key] of [...votes].sort((x, y) => y[1] - x[1])) {
        const [sb, sa] = key.split("|");
        if (!map.has(sb) && !taken.has(sa)) {
            map.set(sb, sa);
            taken.add(sa);
        }
    }
    return map;
}

Two cases are left over:

• A speaker who didn’t talk during the overlap. If exactly one label is unmatched on each side, they are almost certainly the same person, so match them by elimination.
• Anything still unmatched. Give it a new, distinct label rather than guessing. Two speakers incorrectly merged into one is much harder for a reader to notice and fix than one speaker who appears under two labels.

Apply the map to every word in B before splicing, and carry the result forward: the next seam compares chunk C against the already-merged transcript, so labels stay stable across the whole recording.

A checklist for production

• Shift every chunk to absolute time before comparing anything.
• Normalize words for comparison only. Keep the original text in the output.
• Align a bounded window, not the whole transcript.
• Splice at the middle of the matched region.
• Record how every seam was joined: aligned or time cut, words dropped from each side, speaker mapping. When someone asks why a sentence looks odd at 1:42:10, you can answer from the record instead of re-running anything.
• Treat a long overlap with zero matches as an error, never as a time cut.
• Keep it deterministic. The same chunks should always produce the same transcript, which makes it testable and auditable. None of this needs an LLM.

Where this ends up

A merged word list is rarely the final product. People want a readable document with speakers and timestamps, or subtitles that follow broadcast rules. We cover the subtitle side in From Whisper transcript to broadcast-quality SRT and VTT subtitles.

We built all of this into Splice, our service that merges chunked Whisper and WhisperX output from Google Cloud Storage or S3 and renders DOCX, PDF, SRT, VTT and CSV. On one CPU core it merges a 40,000 word transcript (about 5.4 hours of speech in 35 overlapping chunks) and renders all formats in about 1.6 seconds. If you are dealing with this in your own pipeline, we’re happy to talk.