Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe.
However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
Also, this test is English-only, while a strong point of other models is to understand different languages without first having to say which one (so you don't need 3 different keyboard shortcuts if you wanna dictate in 3 languages day-to-day)
Reminds me of the time my neighbours must have wondered if I was having some kind of a breakdown when trying out really basic MacOS voice recognition back in the early 2000s. There was a keyboard shortcut and you could say something like "phone number for firstname lastname" and it would theoretically show you that phone number. Thing is it didn't seem to like a British accent, so I spent a good hour trying out different accents, rotating through various US accents, Australian, South African, Canadian and so on. It seemed to respond best to some kind of a melange of Californian / Australian.
Not too far off what happened, although thankfully I wasn't actually trying to do anything other than test it. Going to take the opportunity afforded by Scottish TV comedy here, and make a very tenuous link to intercultural exchange so I can post my favourite Rab C Nesbitt scene, hands across the sea indeed: https://www.youtube.com/watch?v=uKxPH_QH940
Interesting - I don’t think I’ve ever seen anyone from the UK refer to talking in a “British accent” before since we are normally aware of the wild regional variations.
Fair point! I think it's a tic from being English and having lived in Scotland for quite a while so I autocorrect "English" to "British", but I've over-corrected here. (Also perhaps something to do with "British English").
As an Australian, Apples voice models have always sucked. I've tried using stt (again) more recently and its improved, but i'm so tired of having to Americanize my voice to get it to figure out what the hell i'm saying.
As a Texan first, American second, I sympathize with this statement. Siri can't understand me probably 25% of the time. I use STT for iMessage while in the car, and half the time it will take 3+ times to either get it right or me give up, and hope to remember to text them by hand when I next stop.
The Two Yoots problem. Do you use d's in place of t's such as dees/dems/dose/dere? I have a heavy queens accent so you'll hear me say things like "deres tree uh dem ova dere."
> there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia
Is parakeet state of the art? It always transcribes speech fragments for me, like if I stutter and say "m-m-m-map" parakeet will dutifully transcribe "m m m map". Which I guess could be a good thing or a bad thing depending on what you want. Whisper does not do that however.
I think that's parakeet doing its job there. That is a closer reflection of what you've actually said. The trick is then throwing that output through some additional deterministic and non-deterministic steps to tidy it up however you prefer. It's exactly what I do with my free and open source dictation app (dictator.robgough.net) for Mac+iOS. And of course, everything stays entirely on-device. Gemma E4B is really wonderful for that second step, it's great at language – but takes up 6-7GB RAM.
Parakeet is certainly faster on my machine (m3 max), but I can't stand using it over Whisper for dictating my prompts. It makes a lot more mistakes, possibly because (like you mentioned) large portions of the speech will pause / stutter as I think about what to include.
With whisper v3 turbo, I can almost always live with the few mistakes because the overall stream-of-thought word-salad I provide is still clear at a high level. The bits and pieces of context seem to help, that I might leave out if typing and focused more on traditional conciseness / clean writing. With parakeet, I needed to do frequent editing even for shorter bits of speech.
I realize some applications prioritize the latency.
It sounds like post processing should be the job of an LLM. I would like the voice model to be faithful to what was said and then that output can be smoothed over or postprocessed as needed for the use case
To be clear, I'm talking about high word error rate with parakeet vs whisper, not post processing and cleaning up my speech. Re: being faithful to what was said, one small example, Whisper will often put ellipses when I pause.
From my experience, Speech-to-text falls way short of Wispr flow and I would assume the ones that are said to be better than that. It lacks context awareness and formatting
Yeah, apple will be optimizing a model to work on ANE and then turn it into a native app. My only hope is that it has a reasonable api so that I can use that as a generic input source across iOS / macOS that’s equivalent to the ubiquity of the keyboard.
I don’t know how Apple divides computation between the GPU and the Neural Engine, but one major benefit, especially for real-time transcription on laptops, is the improved power and thermal efficiency. I noticed better accuracy after switching my app to SpeechAnalyzer, and I suspect part of that improvement for me came from the microphone no longer having to compete with jet-engine fan noise.
> RIP to a lot of the paid apps that simply wrap Whisper
I started using a few open source apps for transcription and eventually subscribed to a paid one...
On paper, it's not hard to compete, but for this use case, a few rough edges make it really frustrating to use. Like a keyboard that sometimes doubles the letter "e"
Automatic dictionary, seamless language switch, no issues with accents, etc... Putting the effort in the last mile makes a world of difference.
If anyone has better options, I'm willing to have a look. The best open source solution I found was Handy, and I currently use Wispr Flow
I built my own because I was frustrated with a lot of the free options. Largely because a lot of them had an upsell to be able to do the secondary post-processing step with an LLM. And it wouldn't pick up things like emojis properly or say numbers. Because of that, I left quite a lot of options in there for customising and adding additional steps, etc. Feel free to take a look: dictator.robgough.net
My initial Mac version actually had three additional steps that you could toggle, obviously at the cost of some speed. That is what the website talks about, although nowadays for my own use I've reduced that to just one step and found that it's pretty great. I've got a new version in test to tidy that up, but still lets you define as many steps as you want.
Thanks, was looking at a better diarization model.
Even for those sorts of apps, MacParakeet which I've been using is FOSS so no payment needed. In reality these days with AI the ability to spin up a free and/or OSS competitor falls to zero.
Thanks, I didn't see vanch007 version at first (only ~30 downloads), I usually look at mlx-community. For the size I was looking at the wrong model (TTS not transcribe-diarize), thanks for the corrections.
> However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
What's insane to me is that you have all of these low-quality me-too apps, and literally no one could bother to read the damn Human Interface Guidelines or follow iOS design conventions.
Doing so is literally LESS WORK than trying to make your own custom awful iOS UI.
Not if your app is a Web wrapper, which so many of these are.
If you use SwiftUI (the native recommendation by Apple), it severely penalizes you, if you want to paint outside the lines (which is a big reason that I don't use SwiftUI for shipping apps). It's insanely easy to write a native app that is 100% in line with HIG.
This particular product used Whisper, so that was obviously the right model to compare it against. Further this is explicitly on device, and Nemotron 3.5, as one example, is 2.5GB for the model.
And if someone were broadly comparing all on-device models (instead of just looking at how this new on-device ones compares to what a specific product uses), Nemotron 3.5's WER are actually a bit higher than what they report for SpeechAnalyzer, for both tests.
I will plug Willow for mac recording. IMO it's basically to me a "better than perfect transcription" as it cleans things up and is almost instant. I liked Superwhisper but switched to Willow as it was a big difference.
Its so good that I'm not sure that it's possible to get any better. Speech to text seems like basically a solved problem, if not now then definitely in 5 years. I don't know if any of these speech to text businesses will work in the long run, but for consumers they are great. My guess is the 2030 version of Apple's SpeechAnalyzer will be so good that nobody will need to use 3rd party software.
Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.
Been using it for a podcast app I have been developing for half a year lol (I hope I publish it by version 27) and I can confirm it’s real fast.
Splitting the audio in multiple segments and firing it up without hitting the maximum limit of concurrent decoding streams makes it blazing fast. Fair enough you loose the cut, but it’s good enough for just podcast. In one minute it chews through one hour of audio. This on an iPhone 17 Pro.
You could perhaps run over the segment splitting points (plus a few seconds back and forward) in a second batch then merge the results in the end so you don't miss anything.
Nothing really, except that I get to play with SpeechAnalyzer APIs, foundation models, translations. It’s basically my playground where to try all things. Been listening a lot of Chinese podcasts lately, transcribed and translated by local models.
Edit: all that said, the app is irrelevant. What I want to say is that live transcripts on iOS using Apples frameworks works very well. Only thing I miss is diarization support.
Vs Voxtral would be a better comparison. No other model, open or closed, has been able to hit such a low AER (Acronym Error Rate ;)) for my meeting transcripts. Seems to understand/infer all the technobabble I use at work. Never have to edit anything. Whisper was catastrophically bad.
I typically disable autocorrect on Apple products because of this, cautiously optimistic about their improved speech models, but definitely worried that it's going to 'correct' technical jargon to more common words.
Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?
I have tried everything (that will run on a 12GB RTX 4070) and I have yet to find anything with better accuracy than Whisper V2 Large for my dataset (discord audio from TTRPG sessions, isolated per-speaker, mostly non-American accents)
Any chance you can benchmark against whisper large and large v3 turbo? These run comfortably on older Macbooks and are still far more accurate in real life dictation compared to even the parakeet models( despite ASR leaderboards) with an RTF < 1.
Try MOSS-Transcribe-Diarize from a few days ago. I’m getting better results than those whisper models. And it’s very fast and small. Better suited to noisy audio too.
Was intrigued by the Inscribe product so I downloaded it to test it out. I uploaded 1 voice file about 4 minutes long and was promptly given a message that I've reached the free plan limits.
Kind of a bait and switch. How can we test the product with such short time limits and what, exactly are you offering if all the processing is done on device by Apple?
Finally. I‘d be delighted though if they actually implemented language autodetection (like everywhere else) though. There’s little more frustrating in my day to day than having dictated half a page to find that it‘s complete gibberish because Apple forces you to select the right language first…
Same with the keyboard. Apple is completely incapable of taking context into account for the input mechanisms of the operating system.
If I start typing and the existing text is in Spanish, then a sensible default is to select the Spanish keyboard I have installed and let me adjust otherwise.
App developers should also be allowed to supply mini-dictionaries within a context to allow autocorrect to work correctly in that context, so for example in this thread [SpeechAnalyzer, API, Whisper, Parakeet, Nemotron] should be supplied so that these terms are autocorrected.
I don’t like it written that way either, and it always seems like the type of number you put on a slide for a head of sales or something. It rankles because:
- it implies that error could be increased n-times, but a 15x _increase_ in 9% error would be an error rate of 135%, which is nonsensical.
- a reduction from 90% error to 20% error is clearly a bigger improvement in rightness to a reduction from 9% to 2%. One is “almost all wrong to almost all right”, the other is “more right”, but they are both a 4.5x reduction in error which means that the 4.5 quantity doesn’t have a constant meaning.
The answer is something like log odds ratios, but that introduces the additional need for a reader to know what that is, and that would be unusual.
For my current purposes, I need a speech-to-text model/API to also emit word-level timestamps - for now, that makes ElevenLabs's Scribe v2 the best multiplatform, multi-language choice though it does look like this SpeechAnalyzer API provides them (although only for English).
Cloud is so cheap and quick. I use local too but my api bill is like 3 quid a month. You would have to be very cheap or have compliance needs to tolerate the error gap
Every single asr model I tested so far did not support timestamps properly though. Some use external aligner to create timestamp, but the accuracy is still much inferior than whipser in case the audio is noisy.
i wish they had benchmarked it against parakeet-unified-en-0.6b and cohere-transcribe-03-2026. i am using parakeet with https://handy.computer daily and it's amazing.
I stopped reading after seeing they compared only with Whisper Small, Base, Tiny
This is useless test and benchmark when you have these day Whisper-V3-Large and Whisper V3-Turbo that you can faster than realtime on 5 years old macbook on apple sillicon (ANE). They didn't even compared to parakeet v2 or parakeet v3. And only english language...
this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.
This hasn't been tested in court. But there's a high chance that model weights are not copyrightable, only the code to generate them is.
Cloud models are usually protected by trade secret laws, leaking them would get you in trouble. However if the model is made available publicly, as long as you don't break the law to get them, anything after that would be fair game unless Apple can prove that humans have significant authorship over the weights, which hasn't been tested and is a significant burden to prove/disprove.
Copyright protects original forms of expression, not arbitrary data. It is very arguable whether it applies to model weights. However, it would likely constitute a license violation.
Aside from the legality of it, I think you are underestimating how complex it can be to do that. It is possible in theory but not something that will be a fun side quest like you are making it seem.
The Jedi Hand Wave-y nature of the way people talk about AI these days is going to make reigning in the AI superpowers nearly impossible. Because there are people out here who believe models of this quality are easily replicated or reverse engineered. Neither is really doable on any reasonable timeline by people who are not AI experts. Real AI experts. Not TF/PyTorch monkeys or Agent Slop Slingers.
And those people are already highly incentivized to not make anything performing better than SOTA models open source.
Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15.
Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking all the extra layers of privacy and security might be slowing them down?
Overall, Apple who is suing Open AI should just buy them and let me have the best conversational AI out there baked into my old ass iPhone. Because as so far the new Siri on my old phone (tho again GPT works great talking to it and for years) doesnt come close. It's the same old "Could you try that again," Siri. BOO!!!
Yeah, ChatGPT voice is great experience vs. Siri on that phone. In case you haven't done something like this already:
1. In Shortcuts app, make shortcut named "AI Voice Mode" (or whatever you want, YMMV)
2. Set it to run the ChatGPT action "Voice Mode" (requires at least the minimum paid tier, I think)
3. To trigger, say "Hey Siri, AI Voice Mode" (or whatever you called the shortcut)
This is a pretty slick integration, but yeah, if it were baked in that would be all the better.
Ridiculous that for past many years we can talk to GPT on our iPhones without any hiccups and this new Siri is still the same old horse crap (at least for me and this latest beta). Buy them already Apple or possibly be replaced by them as their path & trajectory (working on Ai devices now and they are stealing like Jobs did with Xerox) mirrors yours in the late 1970s.
Thanks for the tip and if Im not mistaken it's similar to asking Siri to ask chatGPT to ask XYZ?
> similar to asking Siri to ask chatGPT to ask XYZ
Effectively, it sort of does that, but really it just listens to the wakeword and opens/switches to the requested app & modality.
FWIW, I get a very different functional result using the Shortcut method vs. asking Siri to delegate natively. To compare, I asked Siri (non-beta here) now to "ask ChatGPT <x>" and I got a top-card with some fairly low quality SEO-ranked weblinks.
New Siri is impressive in that it answers satisfactorily now 80% of the time vs 10% with old Siri.
But it’s slow as shit. GPT, Claude, and Gemini can answer me in 5-10 seconds. Google AI Mode can answer in 2 seconds.
New Siri usually takes 25 seconds to respond to me. This morning it timed out (with strong network connection) when asked a simple multiplication question.
Author here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro.
Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has been guesswork. Short version: the new API cuts WER 3.5-4x vs the old one (2.12% vs 9.02% on test-clean), and it also beat Whisper Small on both splits at about 3x the speed. The old API came in last on clean speech, behind even Whisper Tiny.
On "why should I trust a vendor benchmark": the Whisper column reproduces OpenAI's published LibriSpeech WERs within +0.11 to +0.42 on all six measurements (same corpus, same normalizer, same scorer for every engine), and the raw per-utterance transcripts are downloadable from the article if anyone wants to rescore with their own normalizer.
Limitations worth stating up front: English only, read speech rather than meeting audio, one machine. Precise per-engine timing isn't in the article yet because the accuracy runs shared the machine with a dev workload; WER is load-independent, timing isn't.
Two things that might interest people migrating: SFSpeechRecognizer sends audio to Apple's servers unless you set requiresOnDeviceRecognition, and with SpeechAnalyzer, finishing your input stream is not enough to end a session. If you never call finalizeAndFinishThroughEndOfInput(), the results sequence never terminates and your await hangs forever. I found that one because it was shipping in my own app.
Happy to answer questions about the harness or the normalizer.
At this point, I would not recommend ignoring Parakeet TDT 0.6b v2/v3 (english-only versus multilingual). Those models have been out for a year, give or take, and they are both accurate and fast. I would choose Parakeet over Whisper in almost all situations these days. Parakeet works great even on my several year old iPhone 15 Pro Max, so if an app is going to ship a dedicated model, I strongly recommend investigating Parakeet.
On the more cutting edge front, Granite Speech 4.1 has proven to be a reliable workhorse for me, but it is larger than Parakeet. Cohere Transcribe is interesting, but how strong it is seems to vary more from task to task.
Parakeet Unified 0.6B came out a few months ago, combining both online streaming and offline transcription into one model, and that is one that I need to test more, but it seems promising.
As others have mentioned macOS 27/iOS 27 is supposed to have a new model, particularly on devices with 12GB of RAM or more. I have not actually seen the option to enable that new model yet, though, despite being on the beta on a device that meets the requirements. Maybe a benchmark would reveal that it is already active?
You can't reasonably expect generic ASR to infer tmux from "tee-mucks". "tee-em-you-ex" works reliably if you're ok with capitalisation for your use case.
If this isn't open source/weights and can't run locally, I don't see how this is a replacement for Whisper or other open models, e.g. within Home Assistant.
It's a local model so it's essentially open weight such that you could feasibly export it somehow since it's already on the laptop somewhere. Apfel is a wrapper app like ChatGPT but using Apple Foundation Models, I assume something similar will happen with this transcription model.
It's not open weight, but the point is to be an on device (and thus local, privacy preserving) option. The article mentions that as the caveat
> What this means if you just want good transcription
> If you are on a current iPhone or Mac, the best on-device transcription engine for English is already in the operating system, and the private option is no longer the compromise option
Presumably if you don't trust apple you wouldn't purchase their products and even if you were for example forced to use it via work or something you wouldn't use this feature ... so it doesn't really change the calculus as presented by this article - IF you ALREADY HAVE a MODERN Mac (and trust apple) this is your best option
You are being naive. An Apple device makes dozens of network requests every minute or so to Apple. It is neigh on impossible to verify what is being requested or sent. Also unplugging the Internet and verifying that something still works does not mean the app won't phone home behind your back when it can. These things are designed to fail silently.
The appeal is that users only have to download it once across all apps that use it. Instead of convincing a user to give a couple gigs for just your one app
Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe.
However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
Also, this test is English-only, while a strong point of other models is to understand different languages without first having to say which one (so you don't need 3 different keyboard shortcuts if you wanna dictate in 3 languages day-to-day)
Reminds me of the time my neighbours must have wondered if I was having some kind of a breakdown when trying out really basic MacOS voice recognition back in the early 2000s. There was a keyboard shortcut and you could say something like "phone number for firstname lastname" and it would theoretically show you that phone number. Thing is it didn't seem to like a British accent, so I spent a good hour trying out different accents, rotating through various US accents, Australian, South African, Canadian and so on. It seemed to respond best to some kind of a melange of Californian / Australian.
Scottish Elevator - Voice Recognition - https://www.youtube.com/watch?v=NMS2VnDveP8
Not too far off what happened, although thankfully I wasn't actually trying to do anything other than test it. Going to take the opportunity afforded by Scottish TV comedy here, and make a very tenuous link to intercultural exchange so I can post my favourite Rab C Nesbitt scene, hands across the sea indeed: https://www.youtube.com/watch?v=uKxPH_QH940
Interesting - I don’t think I’ve ever seen anyone from the UK refer to talking in a “British accent” before since we are normally aware of the wild regional variations.
Fair point! I think it's a tic from being English and having lived in Scotland for quite a while so I autocorrect "English" to "British", but I've over-corrected here. (Also perhaps something to do with "British English").
Does anyone have any experience with Mandarin STT? What's a good model for this? The use-case I have is subtitling of Mandarin speech.
As an Australian, Apples voice models have always sucked. I've tried using stt (again) more recently and its improved, but i'm so tired of having to Americanize my voice to get it to figure out what the hell i'm saying.
As a Texan first, American second, I sympathize with this statement. Siri can't understand me probably 25% of the time. I use STT for iMessage while in the car, and half the time it will take 3+ times to either get it right or me give up, and hope to remember to text them by hand when I next stop.
As a Brit, I concur.
That checks out.
It also struggles with my NYC-area accent, which is only medium thick.
The Two Yoots problem. Do you use d's in place of t's such as dees/dems/dose/dere? I have a heavy queens accent so you'll hear me say things like "deres tree uh dem ova dere."
> there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia
Is parakeet state of the art? It always transcribes speech fragments for me, like if I stutter and say "m-m-m-map" parakeet will dutifully transcribe "m m m map". Which I guess could be a good thing or a bad thing depending on what you want. Whisper does not do that however.
I do like cohere transcribe a lot.
I use Parakeet V3 via this tool and it is actually quite reliable for me (in English): https://github.com/cjpais/Handy
If you are using Parakeet for English only then you should be using V2. V3 is for several languages and is worse at English only.
I think that's parakeet doing its job there. That is a closer reflection of what you've actually said. The trick is then throwing that output through some additional deterministic and non-deterministic steps to tidy it up however you prefer. It's exactly what I do with my free and open source dictation app (dictator.robgough.net) for Mac+iOS. And of course, everything stays entirely on-device. Gemma E4B is really wonderful for that second step, it's great at language – but takes up 6-7GB RAM.
Agree on this point. Recent anecdotal testing I did found Whisper is still better than Parakeet.
Parakeet is certainly faster on my machine (m3 max), but I can't stand using it over Whisper for dictating my prompts. It makes a lot more mistakes, possibly because (like you mentioned) large portions of the speech will pause / stutter as I think about what to include.
With whisper v3 turbo, I can almost always live with the few mistakes because the overall stream-of-thought word-salad I provide is still clear at a high level. The bits and pieces of context seem to help, that I might leave out if typing and focused more on traditional conciseness / clean writing. With parakeet, I needed to do frequent editing even for shorter bits of speech.
I realize some applications prioritize the latency.
It sounds like post processing should be the job of an LLM. I would like the voice model to be faithful to what was said and then that output can be smoothed over or postprocessed as needed for the use case
To be clear, I'm talking about high word error rate with parakeet vs whisper, not post processing and cleaning up my speech. Re: being faithful to what was said, one small example, Whisper will often put ellipses when I pause.
Apple's own Voice Memos app already does automatic transcription since macOS 15 / iOS 18.
Speech-to-text is also already built into the keyboard as well, so it can be used in any app where a user would type.
From my experience, Speech-to-text falls way short of Wispr flow and I would assume the ones that are said to be better than that. It lacks context awareness and formatting
I am curious, what are the use cases people are using voice transcription for?
I find voice memos really annoying, so I’ve been dropping them into a Whisper frontend for a while. Something built into the OS would be nicer though.
Of these only Parakeet is <1B, it looks better than Apple's model, however it's not builtin. I wonder how it compares on latency and efficiency.
Parakeet is incredibly fast and accurate even on CPU, and it supports streaming now also in TDT3.
Apple likely needed a model that ran on their NPU natively.
- parakeet usually runs on Bfloat16. NPU doesn't support that
- CPU is not as fast as the NPU for these ops on A-series, and even on modern CPUs, there's a latency delay
- Parakeet latency is fine but "fine" may not be good enough for Apple's UX team.
- CPU increases power consumption over dedicated float blocks
So I would say that Parakeet was a non-option for Apple to ship, although it should be in the benchmarks anyways!
Fluidaudio implements Parakeet on ANE. I'd like to know how SpeechAnalyzer compares in speed.
https://github.com/FluidInference/FluidAudio
Just tried test using yap on a single ~1hr mp3: yap/Speechanalyzer is about 50% slower than fluidaudio on M1. yap interface is nicer though.
https://github.com/finnvoor/yap
Yeah, apple will be optimizing a model to work on ANE and then turn it into a native app. My only hope is that it has a reasonable api so that I can use that as a generic input source across iOS / macOS that’s equivalent to the ubiquity of the keyboard.
Presumably the existing transcribe button on the keyboard will route through this on iOS 26?
I hope they replace their awful voice to text on their keyboard. I can't stand that terrible bit of software.
I don’t know how Apple divides computation between the GPU and the Neural Engine, but one major benefit, especially for real-time transcription on laptops, is the improved power and thermal efficiency. I noticed better accuracy after switching my app to SpeechAnalyzer, and I suspect part of that improvement for me came from the microphone no longer having to compete with jet-engine fan noise.
> RIP to a lot of the paid apps that simply wrap Whisper
I started using a few open source apps for transcription and eventually subscribed to a paid one...
On paper, it's not hard to compete, but for this use case, a few rough edges make it really frustrating to use. Like a keyboard that sometimes doubles the letter "e"
Automatic dictionary, seamless language switch, no issues with accents, etc... Putting the effort in the last mile makes a world of difference.
If anyone has better options, I'm willing to have a look. The best open source solution I found was Handy, and I currently use Wispr Flow
I built my own because I was frustrated with a lot of the free options. Largely because a lot of them had an upsell to be able to do the secondary post-processing step with an LLM. And it wouldn't pick up things like emojis properly or say numbers. Because of that, I left quite a lot of options in there for customising and adding additional steps, etc. Feel free to take a look: dictator.robgough.net
My initial Mac version actually had three additional steps that you could toggle, obviously at the cost of some speed. That is what the website talks about, although nowadays for my own use I've reduced that to just one step and found that it's pretty great. I've got a new version in test to tidy that up, but still lets you define as many steps as you want.
I’m not sure I agree. There may be better models, but the comparison is still useful so long as whisper is so widely used.
For multilingual and noisy audio the best right now is MOSS-Transcribe-Diarize which was released just a few days ago
Superwhisper does a lot more than just provide a whisper/parakeet UI so I’m not sure Apple will destroy them so easily
Thanks, was looking at a better diarization model.
Even for those sorts of apps, MacParakeet which I've been using is FOSS so no payment needed. In reality these days with AI the ability to spin up a free and/or OSS competitor falls to zero.
I’m not even using it for diarisation just transcription and it’s amazing. It also doesn’t need a VAD
A new VAD I found though is FireRedVAD which has better benchmark results than TEN and Silero by far
Interesting! And what would you say are MTD top competitors?
I'm only dealing with Japanese audio so for me it's Anime-Whisper, a Japanese-specific fine-tuning of Qwen3 ASR, and Apple SpeechTranscriber.
16GB! (edit: this is wrong, I was looking at TTS, the transcribe model is 1.7GB). Compared to Parakeet 2.3GB (but no diarization).
Also doesn't seem to be tailored to Apple hardware (i.e. no MLX or ANE variant/implementation)
I am running the MLX fork https://huggingface.co/vanch007/mlx-MOSS-Transcribe-Diarize
Generally labs don't release MLX or ANE versions and we must rely on finding someone who's converted it
Parakeet is not multilingual so not directly comparable
Where do you see 16GB? MOSS is smaller than Parakeet at 1.82GB
Thanks, I didn't see vanch007 version at first (only ~30 downloads), I usually look at mlx-community. For the size I was looking at the wrong model (TTS not transcribe-diarize), thanks for the corrections.
I use handy.computer and it is pretty much everything I want from a transcribing app.
Hey. Yes. I did vive code one as an exercise yo learn how to publish to apple store.
Listen and transcribe felt like the easiest thing to do.
Distavo.com
The source is open for anyone to use, and the builds are in github.
I found quite interesting that claude didn't help too much on how to publish to SetApp until Fable.
How many of the Whisper competitors will work at a reasonable speed using only CPU (on Linux, not Apple)?
(Genuine question - I'm a happy Whisper user but am always looking for improvements).
> However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
What's insane to me is that you have all of these low-quality me-too apps, and literally no one could bother to read the damn Human Interface Guidelines or follow iOS design conventions.
Doing so is literally LESS WORK than trying to make your own custom awful iOS UI.
Not if your app is a Web wrapper, which so many of these are.
If you use SwiftUI (the native recommendation by Apple), it severely penalizes you, if you want to paint outside the lines (which is a big reason that I don't use SwiftUI for shipping apps). It's insanely easy to write a native app that is 100% in line with HIG.
This particular product used Whisper, so that was obviously the right model to compare it against. Further this is explicitly on device, and Nemotron 3.5, as one example, is 2.5GB for the model.
And if someone were broadly comparing all on-device models (instead of just looking at how this new on-device ones compares to what a specific product uses), Nemotron 3.5's WER are actually a bit higher than what they report for SpeechAnalyzer, for both tests.
Came here to post this. I use handy on my own machine and it's perfect with parakeet. If I switch to whisper it makes lots of mistakes
I will plug Willow for mac recording. IMO it's basically to me a "better than perfect transcription" as it cleans things up and is almost instant. I liked Superwhisper but switched to Willow as it was a big difference.
Its so good that I'm not sure that it's possible to get any better. Speech to text seems like basically a solved problem, if not now then definitely in 5 years. I don't know if any of these speech to text businesses will work in the long run, but for consumers they are great. My guess is the 2030 version of Apple's SpeechAnalyzer will be so good that nobody will need to use 3rd party software.
Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.
Been using it for a podcast app I have been developing for half a year lol (I hope I publish it by version 27) and I can confirm it’s real fast.
Splitting the audio in multiple segments and firing it up without hitting the maximum limit of concurrent decoding streams makes it blazing fast. Fair enough you loose the cut, but it’s good enough for just podcast. In one minute it chews through one hour of audio. This on an iPhone 17 Pro.
You could perhaps run over the segment splitting points (plus a few seconds back and forward) in a second batch then merge the results in the end so you don't miss anything.
What's different about your podcast app?
Nothing really, except that I get to play with SpeechAnalyzer APIs, foundation models, translations. It’s basically my playground where to try all things. Been listening a lot of Chinese podcasts lately, transcribed and translated by local models.
Edit: all that said, the app is irrelevant. What I want to say is that live transcripts on iOS using Apples frameworks works very well. Only thing I miss is diarization support.
If it was faster but worse, maybe compare it to a smaller whisper model?
I imagine because quality of transcription is what matters.
Obligatory mention of Handy which is open-source and cross-platform:
https://github.com/cjpais/Handy
It's great for me when configured to use Parakeet v3.
Vs Voxtral would be a better comparison. No other model, open or closed, has been able to hit such a low AER (Acronym Error Rate ;)) for my meeting transcripts. Seems to understand/infer all the technobabble I use at work. Never have to edit anything. Whisper was catastrophically bad.
I typically disable autocorrect on Apple products because of this, cautiously optimistic about their improved speech models, but definitely worried that it's going to 'correct' technical jargon to more common words.
Is this the new dictation engine that I'm not allowed to run on my 1-YEAR-OLD IPHONE 17 because it's not Pro?
Why did you buy the cheap one, that's your own fault
At the time, there was very little advantage to buying the Pro model. Ironically, it's the first time I've ever bought the base model.
Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?
There are many [0], you can search and filter by streaming and open weight only as well.
Looks like Voxtral and Nvidia's Nemotron are best.
[0] https://artificialanalysis.ai/speech-to-text/non-streaming
There's tons, Parakeet was the last I remember seeing which seemed to gain traction (independent lightweight implementations etc).
I have tried everything (that will run on a 12GB RTX 4070) and I have yet to find anything with better accuracy than Whisper V2 Large for my dataset (discord audio from TTRPG sessions, isolated per-speaker, mostly non-American accents)
Same, for my English-only podcast
Not v3?
Nvidia's Nemotron subsumes their older Parakeet model now even for real time streaming.
Parakeet is way faster (on Nvidia hardware) but not quite as accurate in my experience.
It's also super fast on CPU.
Parakeet isn't as good as whisper large.
Impressive. Apple said they improved the models in 27 didn’t they? It would be interesting to see the numbers the beta turns in.
Whisper large v3 turbo works fine locally on recent iphones, weird to keep it out of the comparison
Excited to get next years diarization models - ArgMax is pretty incredible but cost prohibitive (when compared to 0 price from iOS)
I use Spokenly, offline-only mode with the Nvidia model. All local, totally free. Highly recommend
Any chance you can benchmark against whisper large and large v3 turbo? These run comfortably on older Macbooks and are still far more accurate in real life dictation compared to even the parakeet models( despite ASR leaderboards) with an RTF < 1.
Try MOSS-Transcribe-Diarize from a few days ago. I’m getting better results than those whisper models. And it’s very fast and small. Better suited to noisy audio too.
Was intrigued by the Inscribe product so I downloaded it to test it out. I uploaded 1 voice file about 4 minutes long and was promptly given a message that I've reached the free plan limits.
Kind of a bait and switch. How can we test the product with such short time limits and what, exactly are you offering if all the processing is done on device by Apple?
Finally. I‘d be delighted though if they actually implemented language autodetection (like everywhere else) though. There’s little more frustrating in my day to day than having dictated half a page to find that it‘s complete gibberish because Apple forces you to select the right language first…
Same with the keyboard. Apple is completely incapable of taking context into account for the input mechanisms of the operating system.
If I start typing and the existing text is in Spanish, then a sensible default is to select the Spanish keyboard I have installed and let me adjust otherwise.
App developers should also be allowed to supply mini-dictionaries within a context to allow autocorrect to work correctly in that context, so for example in this thread [SpeechAnalyzer, API, Whisper, Parakeet, Nemotron] should be supplied so that these terms are autocorrected.
I'm always confused by these phrases:
> The new API cuts word error rate by 3.5 to 4x on the same audio: from 9.02% to 2.12% on clean speech
Shouldn't they have said "cuts error rate by 78%" or something?
I don’t like it written that way either, and it always seems like the type of number you put on a slide for a head of sales or something. It rankles because:
- it implies that error could be increased n-times, but a 15x _increase_ in 9% error would be an error rate of 135%, which is nonsensical.
- a reduction from 90% error to 20% error is clearly a bigger improvement in rightness to a reduction from 9% to 2%. One is “almost all wrong to almost all right”, the other is “more right”, but they are both a 4.5x reduction in error which means that the 4.5 quantity doesn’t have a constant meaning.
The answer is something like log odds ratios, but that introduces the additional need for a reader to know what that is, and that would be unusual.
For my current purposes, I need a speech-to-text model/API to also emit word-level timestamps - for now, that makes ElevenLabs's Scribe v2 the best multiplatform, multi-language choice though it does look like this SpeechAnalyzer API provides them (although only for English).
I second that! Can you run your benchmarks against the iOS 27 beta?
If you are running ios 27 beta is this the model when you hit voice to text on the native keyboard?
Cloud is so cheap and quick. I use local too but my api bill is like 3 quid a month. You would have to be very cheap or have compliance needs to tolerate the error gap
Every single asr model I tested so far did not support timestamps properly though. Some use external aligner to create timestamp, but the accuracy is still much inferior than whipser in case the audio is noisy.
Lots of comments are "you should compare against X and Y" - even better, just get the results on a standard benchmark, so you can compare against all,
https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
well, it's nice, but the multi-lingual diff is rather limited (only European languages).
I'm looking for good and cheap transcription + speaker diarization on my Mac for a small personal project. Recs?
i wish they had benchmarked it against parakeet-unified-en-0.6b and cohere-transcribe-03-2026. i am using parakeet with https://handy.computer daily and it's amazing.
Would this end up replacing the default iOS keyboard dictation functionality in iOS 27?
No, this is the current dictation functionality on iOS 26.
If you want to use this on the CLI: https://github.com/finnvoor/yap.
Supports SRT/TXT/VTT or JSON-with-optional-word-level-timestamps output and progress meter.
Also it can transcribe live system audio.
Yeah i do find apple's speech to text very good lately and no need to use openai or anything that seem to market their services better
This is great marketing, I had no idea what inscribe was, but a blog like this going viral did something no ad could do for me.
Anyone know the best choice these days specifically for speaker diarization?
I'd be interested too. Last time I looked the state of the art was pretty bad.
I stopped reading after seeing they compared only with Whisper Small, Base, Tiny
This is useless test and benchmark when you have these day Whisper-V3-Large and Whisper V3-Turbo that you can faster than realtime on 5 years old macbook on apple sillicon (ANE). They didn't even compared to parakeet v2 or parakeet v3. And only english language...
this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.
Probably not worth the effort (or legal trouble), unless you can show it's better than other recent open models like Cohere Transcribe.
Is that copyright infringement?
This hasn't been tested in court. But there's a high chance that model weights are not copyrightable, only the code to generate them is.
Cloud models are usually protected by trade secret laws, leaking them would get you in trouble. However if the model is made available publicly, as long as you don't break the law to get them, anything after that would be fair game unless Apple can prove that humans have significant authorship over the weights, which hasn't been tested and is a significant burden to prove/disprove.
Copyright protects original forms of expression, not arbitrary data. It is very arguable whether it applies to model weights. However, it would likely constitute a license violation.
Aside from the legality of it, I think you are underestimating how complex it can be to do that. It is possible in theory but not something that will be a fun side quest like you are making it seem.
With a IDA Pro decompiler license & MCP server, paired with Codex/Claude Code... it would be a fun side quest.
You likely don't need to disassemble the inference code, the weights are "just an array of numbers" in MLX format.
I believe they are are protected on disk
This.
The Jedi Hand Wave-y nature of the way people talk about AI these days is going to make reigning in the AI superpowers nearly impossible. Because there are people out here who believe models of this quality are easily replicated or reverse engineered. Neither is really doable on any reasonable timeline by people who are not AI experts. Real AI experts. Not TF/PyTorch monkeys or Agent Slop Slingers.
And those people are already highly incentivized to not make anything performing better than SOTA models open source.
OK, but how does it compare against Parakeet TDT2 (english), TDT3 (many languages), and Parakeet TDT3 Streaming? And what about whisper large?
Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15.
Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking all the extra layers of privacy and security might be slowing them down?
Overall, Apple who is suing Open AI should just buy them and let me have the best conversational AI out there baked into my old ass iPhone. Because as so far the new Siri on my old phone (tho again GPT works great talking to it and for years) doesnt come close. It's the same old "Could you try that again," Siri. BOO!!!
Yeah, ChatGPT voice is great experience vs. Siri on that phone. In case you haven't done something like this already:
This is a pretty slick integration, but yeah, if it were baked in that would be all the better.You can also map the Action Button to GPT voice mode.
Ridiculous that for past many years we can talk to GPT on our iPhones without any hiccups and this new Siri is still the same old horse crap (at least for me and this latest beta). Buy them already Apple or possibly be replaced by them as their path & trajectory (working on Ai devices now and they are stealing like Jobs did with Xerox) mirrors yours in the late 1970s.
Thanks for the tip and if Im not mistaken it's similar to asking Siri to ask chatGPT to ask XYZ?
> similar to asking Siri to ask chatGPT to ask XYZ
Effectively, it sort of does that, but really it just listens to the wakeword and opens/switches to the requested app & modality.
FWIW, I get a very different functional result using the Shortcut method vs. asking Siri to delegate natively. To compare, I asked Siri (non-beta here) now to "ask ChatGPT <x>" and I got a top-card with some fairly low quality SEO-ranked weblinks.
I’m on iPhone 17 Pro Max, 27 beta 3.
New Siri is impressive in that it answers satisfactorily now 80% of the time vs 10% with old Siri.
But it’s slow as shit. GPT, Claude, and Gemini can answer me in 5-10 seconds. Google AI Mode can answer in 2 seconds.
New Siri usually takes 25 seconds to respond to me. This morning it timed out (with strong network connection) when asked a simple multiplication question.
Damn slow on your newer phone too.
That's crappy, but it's also a beta, so it's not yet time to render a verdict.
> Im hoping Apple gets the new Siri working better on older phones.
Apple would never do that, if anything they did not offer their Siri with the most advanced AI on iPhone 16 Pro Max, which is one year-old only.
Author here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro.
Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has been guesswork. Short version: the new API cuts WER 3.5-4x vs the old one (2.12% vs 9.02% on test-clean), and it also beat Whisper Small on both splits at about 3x the speed. The old API came in last on clean speech, behind even Whisper Tiny.
On "why should I trust a vendor benchmark": the Whisper column reproduces OpenAI's published LibriSpeech WERs within +0.11 to +0.42 on all six measurements (same corpus, same normalizer, same scorer for every engine), and the raw per-utterance transcripts are downloadable from the article if anyone wants to rescore with their own normalizer.
Limitations worth stating up front: English only, read speech rather than meeting audio, one machine. Precise per-engine timing isn't in the article yet because the accuracy runs shared the machine with a dev workload; WER is load-independent, timing isn't.
Two things that might interest people migrating: SFSpeechRecognizer sends audio to Apple's servers unless you set requiresOnDeviceRecognition, and with SpeechAnalyzer, finishing your input stream is not enough to end a session. If you never call finalizeAndFinishThroughEndOfInput(), the results sequence never terminates and your await hangs forever. I found that one because it was shipping in my own app.
Happy to answer questions about the harness or the normalizer.
At this point, I would not recommend ignoring Parakeet TDT 0.6b v2/v3 (english-only versus multilingual). Those models have been out for a year, give or take, and they are both accurate and fast. I would choose Parakeet over Whisper in almost all situations these days. Parakeet works great even on my several year old iPhone 15 Pro Max, so if an app is going to ship a dedicated model, I strongly recommend investigating Parakeet.
On the more cutting edge front, Granite Speech 4.1 has proven to be a reliable workhorse for me, but it is larger than Parakeet. Cohere Transcribe is interesting, but how strong it is seems to vary more from task to task.
Parakeet Unified 0.6B came out a few months ago, combining both online streaming and offline transcription into one model, and that is one that I need to test more, but it seems promising.
As others have mentioned macOS 27/iOS 27 is supposed to have a new model, particularly on devices with 12GB of RAM or more. I have not actually seen the option to enable that new model yet, though, despite being on the beta on a device that meets the requirements. Maybe a benchmark would reveal that it is already active?
Why do you not use Whisper large models when on macOS? They're still fast even when streaming and yield a much lower WER.
Also, just out of curiosity, seems like everyone and their mother is making Whisper wrappers, how is your app different?
Why use relatively ancient models like whisper and not e. g. parakeet?
Please run your benchmark on this new and very impressive model https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize in my testing it outperforms all mentioned especially on noisy audio
MOSS-Transcribe-Diarize
Still nothing beats OpenAI's VTT. Anthropic's sucks and Apple's isn't even usable.
Edit: Getting downvoted by Apple fanboys for telling the truth is a badge of honor.
Which OpenAI model/API do you mean?
Whisper and GPT-4o (for diarization).
In which world 98% accuracy is not usable?
In a world where you say "tmux" and Apple's VTT writes "T Max".
You can't reasonably expect generic ASR to infer tmux from "tee-mucks". "tee-em-you-ex" works reliably if you're ok with capitalisation for your use case.
agreed. plus all the languages supported!
If this isn't open source/weights and can't run locally, I don't see how this is a replacement for Whisper or other open models, e.g. within Home Assistant.
It's a local model so it's essentially open weight such that you could feasibly export it somehow since it's already on the laptop somewhere. Apfel is a wrapper app like ChatGPT but using Apple Foundation Models, I assume something similar will happen with this transcription model.
https://apfel.franzai.com/
It's not open weight, but the point is to be an on device (and thus local, privacy preserving) option. The article mentions that as the caveat
> What this means if you just want good transcription
> If you are on a current iPhone or Mac, the best on-device transcription engine for English is already in the operating system, and the private option is no longer the compromise option
> It's not open weight, but the point is to be an on device (and thus local, privacy preserving) option.
How can you be sure this isn't leaking data or metadata to Apple? Can Apple really be trusted?
The article states:
> If you are on a current iPhone or Mac
Presumably if you don't trust apple you wouldn't purchase their products and even if you were for example forced to use it via work or something you wouldn't use this feature ... so it doesn't really change the calculus as presented by this article - IF you ALREADY HAVE a MODERN Mac (and trust apple) this is your best option
Test it! Does it make network requests? Unplug the internet and see if it still works!
You are being naive. An Apple device makes dozens of network requests every minute or so to Apple. It is neigh on impossible to verify what is being requested or sent. Also unplugging the Internet and verifying that something still works does not mean the app won't phone home behind your back when it can. These things are designed to fail silently.
It is local
The appeal is that users only have to download it once across all apps that use it. Instead of convincing a user to give a couple gigs for just your one app