OpenAI Whisper and it’s amazing power to do fine-tuning demonstrated on my mother-tongue

Kurian Benoy (~kurianbenoy)




The core of the talk covers how we can fine tune a whisper model. I will specifically talk about fine tuning whisper to achieve state of the art results in a low resource language like Malayalam. We have got great results on fine-tuning on Malayalam whisper model. The original model weights of Whisper was reported with a WER of 108% and we were able to fine tune to reach a WER of close to 10%(approx 90 percentage accuracy) in CommonVoice 11 Malayalam subset and even 1% WER in MSC dataset.

Talk Outline:

  1. What is OpenAI Whisper?
  2. Features of OpenAI Whisper
  3. What is Fine-tuning and how to fine-tune Whisper?
  4. About my mother tongue
  5. Methodology of benchmarking whisper models
  6. Results on benchmarking Whisper model
  7. Future Ideas & Conclusion


  • Any developer with 1 years experience, they needn't have any prior ML experience. I will try to demystify any jargons during my talk that's why I am saying no experience is needed in ML.
  • Lot of interest in your own mother-tongue is always appreciated.

Content URLs:


Github project:


Benchmarking results:


Speaker Info:

Kurian is a Team Lead and AI Engineer working in, a fast-paced startup based in Singapore. I have multiple years of experience in Python and Machine learning experience, where I am now looking more into the MLOPs side. I have contributed to various open source organizations like Keras, DVC, HuggingFace,, Swathanthra Malayalam Computing, CloudCV etc.

More details about my previous talks can be found in below link.

Section: Data Science, AI & ML
Type: Talks
Target Audience: Intermediate
Last Updated: