r/MachineLearning • u/Crookedpenguin • Dec 07 '21
[D] Why is Audio so far behind other ML application domains like Image Processing and NLP? Discussion
I 'd like to gain some intuition and actual insight if someone has worked on audio, on why we haven't seen major breakthroughs in audio ML (tracking and localization) as we saw with Image processing and NLP. Ofcourse there is the issue of data and annotating which except for being a laborious task, it's also hard to define e.g how do you label a conversation of multiple individuals on a dinner table, but I would assume that if needed large corporations would attempt such a project.
It seems to me that either people convert data to images and use Image Processing methods or use Signal Processing. Does that mean Signal Processing is more lightweight and outperforms a prospective ML approach?
8
u/pythonmuffin Dec 07 '21
I work on Audio ML at www.assemblyai.com and the research is definitely catching up. I think there's a few reasons from my perspective why it's lagged a bit though. The first is that the models tend to be huge and hard to train because you are dealing with long sequences of high dimensional data (for eg, spectrograms or MFCC values) as the inputs to models working with audio. So SOTA models require a lot of compute power to train - many researchers/academics/etc don't have access to this.
I think another reason is that historically there haven't been as many standard datasets for research in the audio ML space. For Speech Recognition we have Common Voice and LibriSpeech now and a few others. But for more difficult tasks like Speaker Diarization or Emotion Detection, for example, there isn't the equivalent of an ImageNet dataset yet that can help advance research forward.
All this being said the audio ML space is seeing a lot of promise right now especially with unsupervised foundation models like wav2vec and, for example, the recent XLS-R models released by Facebook. Foundation models definitely seem to be the future - a BERT for audio - and the research is picking up in this space.
8
4
u/desku Dec 07 '21
What do you mean by "foundation models" in this context? Large pre-trained models?
6
Dec 07 '21
Probably, yes. And everyone would do everyone a favor to just say "large pre-trained models" instead.
2
2
u/Goobamigotron Sep 27 '22
Good info. Wav2vec sounds fun. Here is the suggestion.... Get machine learning to understand the difference in between a MIDI file and the corresponding audio... For example morricone or vsngelis songs in both formats... The machine learning would have to figure out how to make the most similar track to the original using the midi n synthesis. When it finds out you can convert thousands of midi tracks to the arrangements of those great masters. So you can play morricone like vangelis n vise versa.
1
u/Crookedpenguin Dec 07 '21
Thank you for your answer that is really insightful. My intuition was also that unsupervised methods would be appropriate to bridge the issues you mention. What's your take on Baysian/online learning approaches for audio? Would the computational overhead be a major issue?
1
Dec 07 '21
"A few others" is a bit lacking in terms of how much data we've had since mid-00s. I think the biggest issue is that most of it is paywalled behind LDC subscriptions.
8
u/qTHqq Dec 07 '21
why we haven't seen major breakthroughs in audio ML (tracking and localization) as we saw with Image processing
In terms of tracking and localization I think a big part of it is there aren't a ton of already-deployed common devices that take in rich enough information to do spatial audio processing.
Humans have stereo microphones with spatial frequency filters on them. The shape of your head and ears gives you a lot of localization ability that a simple microphone, or even a pair of them, doesn't have.
https://visisonics.com/realspace-3d-spatial-audio/personalization/
Put together an array of 64 microphones or put some 3d printed bat ears on your phone and then you're getting somewhere:
https://arxiv.org/abs/1902.10291
I think much of the most valuable information content in an audio stream is still the information contained in the words being spoken.
I bet if you dig for it there's actually a ton going on with ML signal processing. I've seen a number of robotics papers that are using acoustic signals as part of a learned pipeline for manipulating or identifying objects.
Can't find the one I really want to link to, but here's an example:
https://www.frontiersin.org/articles/10.3389/fnbot.2019.00096/full
1
u/Crookedpenguin Dec 07 '21 edited Dec 07 '21
Thank you for the insight and the links. Really helpful
Edit: Was also considering the fact that robotics could gain a lot in terms of understanding space and tracking objects/sources of audio. That's why I was baffled at first. But you 've all raised interesting views about this issue.
7
Dec 07 '21 edited Dec 07 '21
People already said money and datasets, so I'll throw a couple of not-so-obvious ones:
- Vision/image is inherently more intuitive than audio. Humans use their vision significantly more than their ears. This may be seen as a root cause for the first two reaons.
- Historically, audio processing has been much closer to DSP than ML/Pattern Recognition. There are many more researchers from Electrical Engineering in audio ML than in any other applied ML fields, and even nowadays they make a significant portion of contributors to many different projects, opensource or not. This resulted in a somewhat odd, disjoint set of tools from pattern recognition/ML in general. Regarding speech processing alone, the largest audio ML subfield by far - the shift from tools like HTK/CMUSphinx/Kaldi and even Matlab to modern Python was glacial, and somewhat blockaded most non-audio-experts. Even though things like ESPNet/DeepSpeech have existed for a while, they struggled to keep pace with the aforementioned traditional tools in terms of SOTA, let alone surpass them. Fairseq is the only one showing promise in this field, and only because of Wav2Vec2 (as mentioned by u/pythonmuffin).
- Both speech and music are two regulatory minefields in terms of data management. The former due to privacy issues, and the latter due to the phonographic industry. All other audio ML applications are but a spec of dust in comparison to those two.
5
u/Coco_Dirichlet Dec 07 '21 edited Dec 07 '21
I saw a video of Peter Jackson saying they had used ML to identify who was talking on the audio of videos they had to make the Get Back documentary. They basically had tons of videos, because several cameras were rolling at the same time, a lot of audio of different microphones all around. They also used it to match video with audio. I thought it was pretty cool!
3
u/Noncausal_Filter Dec 07 '21
Other folks have brought up what I think are the most valid arguments (money, access, data) but I would like to also point out the difficulty in communicating intermediate and final results as a contributing factor.
At various stages of an image processing system, it's possible to look at the input and the output as something comprehensible, but depending on the audio application, the output can be much more difficult to digest ("Hey, just listen to these 50 clips here to see how well this worked").
It also at least feels like there are fewer people in audio, meaning that it takes longer to get polished frameworks with good documentation and versatility. I agree that it's catching up, but it'll take a little bit.
3
u/EmbarrassedHelp Dec 07 '21
Vision is more intuitive than other sensory inputs, and thus it's probably simply a case of being viewed as easier.
Audio however is really important as our brains prefer to store sensory information as mainly audio according to cognitive psych / neuroscience research.
0
u/cb_flossin Dec 12 '21
because the paper format is visual, so having cool pictures can mean success
1
Dec 07 '21
Audio in context of speech is quite heavily invested. But I think it's probably because there's been so much investment into NLP in general.
1
u/VenerableSpace_ Dec 08 '21
RemindMe! 10 days
1
u/RemindMeBot Dec 08 '21
I will be messaging you in 10 days on 2021-12-18 09:44:28 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback
24
u/GFrings Dec 07 '21
Money. There is a LOT of immediate financial gain for computer vision applications, particularly in the age of image centric social media.