Building a Lie Detector: Multi-Modal Sentiment Analysis

Mimansa Jaiswal (~mimansa)




The workshop is to be divided in two parts: 1. Using LIWC and n-gram models for processing sentiments in sentences and short-texts. [By Mimansa] 2. Using image processing in OpenCV for processing sentiment of images posted. [By Sairam]

The workshop would then aim to go over the plausible applications that it could have.

The textual part of the talk aims to cover the following topics under NLP:

  1. I would then proceed to discuss about the classification methods like bag-of-words, random forests etc. and where and when they should be used (Whether in aspect extraction, if yes, where). In here, I would also explain the bias induced in dataset regarding the industry it is dealing with. I would also touch briefly on binary classification (positive, negative) or probability value vector in case of multi-label classification.

  2. I would go over the various areas that sentiment analysis can be used (product reviews, social media posts) and how that information about sentiment can be used. And then I would conclude by discussing about the projects that I have worked upon, that is, giving AI the benefit of recognising and empathising with emotions and how it would be helpful.

  3. I would then introduce the audience to the process of using word2vec and glove or building embeddings from scratch. The data and iJupyter notebooks along with dataset for the purpose would be available in form of a virtual image.

  4. I would then proceed on finding n-grams and building a classifier around various categories (as stated in paper) on real-life deception dataset.

The image processing part would cover the following topics under vision:

  1. Introduction to OpenCV and how it is used to process images. Explain with some examples how the morphological transformation, image transformation, gradient descent works. Learn about the use of basic classification techniques deployed in image classification, for either binary or multi-labelling.

  2. Extraction of features, Facial action points and movements of eyebrows and coding them into feature vectors. This would cover Action Units to be recognised using masking technique.

  3. Combining both the features (achieved using text and image) to build a classifier that predicts whether the person is lying or speaking the truth.

  4. Explain in short about how the sound can be used as well (pitch and MFCC features) can be used. [The audio processing part won't be hands-on due to shortage of time.]


Linear Algebra, Basic matrix manipulation and vectors. Basics of NLP (POS, NER). Experience with nltk module.

Content URLs:

PyCon Singapore talk description:

Slide Deck:

Workshop Data and content is based on: 1. 2.

Speaker Info:

I am a third year student of engineering majoring computer science. My past experience with Sentiment Analysis is of 3 years, with varied internships in Human Computer Interaction development for short texts, improved document sentiment tagger and presently interning in NTU (group Sentic Net) for personality detection and mental health disorder prediction purposes. I have previously given at talk at PyCon Singapore ( on the same topic.

Sairam is a third year student at IIT BHU, majoring in Electronic Engineering. His experience with Computer Vision has been of 3 years where he has been the lead of many workshops and tutorials conducted at his college, for acquainting freshmen with the subject. He has worked in facial and gesture recognition, and presently working as research assistant in NTU for object detection in maritime environment.

Speaker Links:

My LinkedIn profile can be viewed at:

Sairam Tabibu's LinkedIn profile can be viewed at:

Section: Scientific Computing
Type: Workshops
Target Audience: Intermediate
Last Updated: