My Mac Has Two Ways to Listen. I Chose the Wrong One.

For a long time I didn't think about where my words went after I said them.

I'd talk, and a moment later the text would appear. Usually the moment was short. Sometimes, on a long sentence, it stretched, and I'd catch myself waiting for the screen to catch up. I assumed that was just what dictation felt like.

Part of that delay came down to where the model was running. My Mac has two places it can run a speech model, and for months Yapivo had been sending my voice down the longer road.

Two copies of the same model

One evening I set up a small experiment. I took the speech model Yapivo uses, Parakeet TDT 0.6B v3, and ran two copies of it side by side on my Mac. Same model. Same audio. Every word reached both of them at exactly the same moment.

The difference was where they ran.

One ran on the CPU, the general-purpose processor that runs everything else on the computer: the browser, the music, the window you're reading this in. The other ran through Core ML on the Neural Engine, a smaller part of the same chip that Apple built for the kind of math models like this are made of.

To be precise about it, the two copies aren't byte-for-byte twins. The CPU copy is an int8 export that runs in ONNX Runtime. The other is a Core ML conversion that asks for the CPU and the Neural Engine, and Core ML decides which part runs each operation. I haven't profiled that placement, so I'll call it what it is: Neural Engine-backed.

Then I just talked, and watched.

The same words, at different speeds

Both copies wrote nearly the same thing, sentence after sentence. When one misheard a name, usually the other did too. Nearly isn't identical, though, and I'll come back to that.

What was different was the timing. With 14 of my own recordings fed in at real speed, the live words came back with a median inference latency of 45 milliseconds on the Neural Engine-backed path and 152 on the CPU. Both are quicker than a blink. But the difference adds up when it happens every second, all day.

As each inference request contained more audio, the CPU took longer to process it.

Why long sentences were harder

While you're speaking, Yapivo shows your words as you go. To do that it hands your audio to the model a little more than once a second, and once more when you pause. Each of those is an inference pass, and each pass reads everything you've said since your last pause, not just the newest part, because the model needs the whole thought to get the words right.

A longer sentence means more audio in every pass.

In the replay above, there's one long sentence on purpose. Through it, the CPU's passes climbed from about 150 milliseconds to 560. The Neural Engine-backed passes stayed between 50 and 106. At the pause that ended the sentence, the CPU took 467 milliseconds to finish the phrase and the Neural Engine-backed path took 74.

I'd felt that before without having a name for it: the screen trailing a step behind me. Now I could see where it came from.

A sound I hadn't connected

Weeks earlier, my MacBook had started making a faint, high whine whenever I dictated for a while. Quiet enough that I wasn't sure I'd heard it.

My best explanation, and it is an explanation rather than a diagnosis, is that it came from the chip's power circuitry reacting to bursts of work. Every time the model ran on the CPU, several cores woke up at once, worked hard for a fraction of a second, and went quiet again. Spreading that work over fewer cores made the sound quieter, which is why I believe it. I never measured the sound itself.

What I did measure is CPU time. Over the same 14 recordings, the CPU copy used 37% of one core on average while listening. The Neural Engine-backed copy used 6%. That's less work on the CPU, with the potential to reduce energy use and thermal load. I haven't measured either of those.

I also ran 139 of my own recordings through both, one after another, at background priority. The CPU needed 505 seconds; the Neural Engine-backed path needed 51. That's a throughput number for a batch, under a low-priority setting that slows both paths. It isn't the latency you feel while talking, which is why it's ten times instead of three.

The energy cost of one second of speech

The question I actually care about for a laptop is how much energy it takes to understand one second of speech:

Faster isn't automatically cheaper.

A path that finishes in a quarter of the time but draws twice the power uses half the energy, not a quarter. To fill in that equation you need power, in watts, sampled while the inference runs, with the machine's idle power subtracted.

I haven't measured power yet. So here's only the shape of the answer: if the Neural Engine-backed path drew half the active power and finished in a quarter of the time, it would use 0.5 × 0.25 = 0.125 of the energy, about 87.5% less, per second of speech. That is arithmetic on an assumption, not a result, and it says nothing about the battery life of the whole machine, where the screen, the radios and everything else are also drawing power.

What didn't change

The model itself didn't change. What changed was how efficiently it could run.

The transcripts were close, not identical. Across 139 recordings, 4.6% of words differed between the two paths, and 58 recordings had at least one difference. Different precision and different runtimes can do that. I don't have human-checked transcripts of those recordings, so I can't yet say which path was right more often. That needs a controlled accuracy comparison against reference transcripts, and it's on my list.

The pause after you finish a sentence didn't go away either. From my own logs that day, the time from my last word to the text landing in the app was about 630 milliseconds before the switch and 640 after. Most of that wait isn't the model. It's the app deciding that you're actually done (a breath in the middle of a thought shouldn't end it) and then putting the text where it belongs. A faster chip doesn't fix that. It's a different problem, for another day. The first live words did show up sooner in those logs, about 800 milliseconds after I started talking instead of 1,100, though those were different sentences, so take that loosely.

Nothing about privacy changed. Both copies ran entirely on my Mac. Your voice still never leaves your Mac.

People sometimes ask about the GPU, the third option. It can run models like this well too. Which part suits a model best depends on the operations in it, the precision, how big each piece of work is and how often it arrives. I measured the two paths Yapivo actually uses, not the GPU.

See the measurements and how they were taken

Metric	Workload	CPU	Neural Engine	Relative
Live-words inference latency, median	Live replay, 14 recordings (n = 204 / 230)	152 ms	45 ms	3.3× faster
Live-words inference latency, 95th percentile	Live replay, 14 recordings (n = 204 / 230)	275 ms	74 ms	3.7× faster
Live-words inference latency, slowest	Live replay, 14 recordings (n = 204 / 230)	712 ms	89 ms	8.0× faster
Inference latency at a pause, median	Live replay, 14 recordings (n = 59 / 59)	154 ms	43 ms	3.6× faster
Inference latency at a pause, 95th percentile	Live replay, 14 recordings (n = 59 / 59)	380 ms	68 ms	5.6× faster
CPU time while listening (share of one core)	Live replay, 14 recordings, 219.8 s of audio	37.2%	6.2%	6.0× less
Total inference time	Batch, 139 recordings, background priority	504.9 s	51.3 s	9.8× faster
Per-recording latency, median	Batch, 139 recordings, background priority	2452 ms	235 ms	10.4× faster

Transcripts: on 139 recordings, 4.58% of words differed between the two paths (58 recordings had at least one difference). There are no human reference transcripts, so this is disagreement between the two paths, not word error rate.

End to end, from the app log (preliminary): first live words 1143 ms on the CPU path and 803 ms Neural Engine-backed; last word to text landing 630 ms and 640 ms. Medians from the app log on the same day, before and after the switch: different sentences in different modes, so rougher than the controlled tests.

How this was measured

Benchmark date

2026-10-23

Mac

Mac16,7 (MacBook Pro), Apple M4 Pro, 10 performance + 4 efficiency CPU cores, 48 GB

macOS

26.6.2 (25G83)

Software

Yapivo 1.0.838 speech helper (yapivo-stt)

Model

Parakeet TDT 0.6B v3. Same model and version on both paths, exported two ways: the numerics are not identical.

CPU path

ONNX Runtime, parakeet-tdt-0.6b-v3-int8 (int8 quantized). CPU only: 3 threads for the encoder, 2 for the decoder and preprocessor.

Neural Engine path

Core ML through FluidAudio 0.17.7, parakeet-tdt-0.6b-v3-coreml (encoder 6-bit palettized, float16 compute). Core ML compute units cpuAndNeuralEngine for encoder, decoder and joint; preprocessor cpuOnly. Core ML chooses where each operation runs; that placement was not profiled.

Live replay

14 of the author's own dictations (219.8 s of audio) fed to the speech helper at real time in 682-sample pushes, with a live-words request every 0.8 s, once per engine.

Live timing

Live words: request sent to the speech helper → transcript returned. At a pause: the helper reports the pause → the finished phrase returned. Both include audio preprocessing, decoding and the helper’s own messaging; they exclude model loading, endpoint waiting and placing text in the app. The model was loaded before timing; there was no separate warm-up pass. Requests answered in under 5 ms had no audio to decode and are counted but not timed. (17 CPU, 19 Neural Engine). The replay waits for each live-words reply before asking again, so the slower path made fewer requests.

Batch

139 recordings (34.1 min of audio, 0.9–76.9 s each, median 10.4 s) transcribed whole, one after another, at background priority. Per recording: whole-file transcribe request → transcript. Background priority slows both engines; treat the totals as throughput under that condition, not live latency.

Excluded

Recordings one path could not open: 1. Totals compare only recordings both paths finished.

Instrumentation

Wall-clock timing in the Python harness around each request to the speech helper; CPU time from ps for the helper and its child processes.

Not measured

power (watts), energy, temperature, fan speed, battery drain, word error rate against human transcripts, per-operation hardware placement

Source

ane-lab/bench/export_benchmark.py, with the raw results beside it (live_cpu.json, live_ane.json, cpu_v3.jsonl, ane_v3.jsonl).

Where the difference won't show

A Mac with an Intel processor has no Neural Engine to move to. Everything here is about Apple silicon, and specifically one M4 Pro MacBook Pro.

If you only dictate a short sentence now and then, you may never notice. On one short sentence, 150 milliseconds and 45 milliseconds both feel instant. The difference shows up on long thoughts, and on a machine that listens all day.

Why it matters to me

Voice only works if it keeps up with you. A transcript that arrives a beat late is fine for one email. It isn't fine when you talk to your computer all day and expect it to follow along.

I think a lot of what makes a tool feel right is like that. Small delays you stop noticing. Warmth you blame on something else. A sound you aren't sure you heard. You never name them. You just trust the tool a little less each time.

Moving speech to the Neural Engine didn't change what Yapivo hears. It changed how quickly it listens, and how lightly it leans on the rest of your Mac while it does.

That's the path Yapivo runs on now in its newest builds.