Apple's on-device transcription model is about 3x slower than their reported results on my more powerful computer, with a similar error rate in my test (which I consider pretty good results)
The reference to Apache 2.0 on the page refers to Google's base model, not this app. Based on the page alone, the app is free, but the source code is not open-source.
"EmbeddingGemma 2 is made by Google DeepMind and released under Apache 2.0. This is an independent project, not affiliated with or endorsed by Google."
Interesting... is it overkill to use a multimodal model like this on just audio? Could the model size be decreased dramatically, or does the multimodality help the accuracy and speed in some way?
It's not being played through a desktop audio stack, they're operating on the waveform decoded from the audio file. There's no constraints relative to the "natural speed" of the source, just like for video transcoding, it's limited by compute.
during transcription, nothing is actually "playing" the file in the way a human would to listen to it. the audio is stored as a sequence of bytes representing the amplitude of the physical sound waves over time (the recording device measures, or "samples", the amount that its microphone is vibrating and records that amount, some tens of thousands of times per second) with some metadata and compression applied so that other software knows how to interpret the bytes correctly. this can be given to an audio program to convert the bytes back to physical sound waves using your speakers, but it's just bytes at the end of the day, so any software that can read bytes out of the file can do so at whatever speed it wants, regardless of how long the actual recording is.
so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.
in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.
Probably significant parallelism. Split the audio into a lot of slightly overlapping portions to ensure the transitions between words are caught. I’m sure there’s some relatively simple ways to ensure you don’t double up on words.
Yea, I was wondering how they dealt with word boundaries and different word audio lengths generally (short and long). Your point about parallelism is interesting. Doing what they say they are doing, this should be ridiculously parallel, possibly with a reduce step at the end to remove redundant words if their slicing algorithm allows for that to happen. But yea, extremely parallel.
Does this work for Hindi or other non English languages though?
I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files
This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.
I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it
out!
I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing
An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.
https://openai.com/index/whisper/
I'll test theirs too if I get around to it
Here's a sample CLI wrapping the apple model and providing a simple benchmark: https://github.com/llimllib/transcribe
update: doesn't seem like they provide their model and I don't feel like trying to dig it out of their app bundle, so I'm not going to test this one
"EmbeddingGemma 2 is made by Google DeepMind and released under Apache 2.0. This is an independent project, not affiliated with or endorsed by Google."
Or is there another approach?
so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.
in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.
Does this work for Hindi or other non English languages though?
I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files
This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.
I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it out!
I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing
An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.
[0]: https://huggingface.co/parismitaglobalsolutions/indicconform...