A result of Speech Enhancement
This project focuses on speech enhancement with machine learning, and implementation to embedded devices, explicitly targeting STM32F746. And its repository will guide the sequence to make the tiny machine enhance streaming quality.
On an embedded device for generating speech from the microphone, the most important point is its clarity. The speech enhancement can be improved previously based on Speech Enhancement book written by Philipos C. Loizou. Still, the improvement had a limitation whose method removed most noise and parts of speech. This method can use for a hearing-aid since the loudness is already too high, but the limitation is still the same. At this time, two studies indicate the effect of implementing machine learning on a tiny device. [1], [2]
The first research, named tiny lstm, shows almost similar performance in the model, not in the constraint. And the second research, called Clear buds, shows the entire open source from PCB to iOS and tiny devices implementation. However, those research needs to be more fitting with the product having a real-time device using a microphone and its receiver, and bose’s research shows only the result. So, I started to build the open source and the guide to implementing the ML model to the tiny device, especially for speech enhancement as a personal project.
This project is currently an on-going project, which is on the modification of the training model, and will complete by 2023. The details of this project is on the below lists.
Applications
- Dataset for clean and noisy sound
- Trained Model
- Model Compression
- [On-going] Microphone streaming Pipeline in STM32F746
- The guide to implementing the model into specifically STM32F746
- Development in Quantization and Pruning Model
The code includes training pipeline for the ML model, and the guide to implement the model into the tiny device in Tensorflow and Tensorflow Lite.
Code in Research
- Evaluation method for speech quality
- Model with Tensorflow
- Custom Model with Tensorflow inspiring from demuc made by facebook
- Related documents
Reference
[1] Fedorov, Igor, et al. “TinyLSTMs: Efficient neural speech enhancement for hearing aids.” arXiv preprint arXiv:2005.11138 (2020).
[2] Chatterjee, Ishan, et al. “ClearBuds: wireless binaural earbuds for learning-based speech enhancement.” Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services. 2022.