{"id":51249,"date":"2020-12-17T16:32:36","date_gmt":"2020-12-17T16:32:36","guid":{"rendered":"https:\/\/www.sotrender.com\/resources\/?p=8147\/"},"modified":"2020-12-17T16:32:36","modified_gmt":"2020-12-17T16:32:36","slug":"emotion-recognition-images-videos","status":"publish","type":"knowledge_base","link":"https:\/\/www.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/","title":{"rendered":"What is emotion recognition and how does it work on images and videos?"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Marketers always look for innovative ways to improve their communication strategy on their social media platforms, but they still seem to be somewhat afraid of machine learning and AI. The thing is, ML and AI models can help you make data-driven decisions about how your brand is representing itself to your fanbase, so it\u2019s worth learning how to implement those models.<\/span><\/p>\n<p><!--more--><\/p>\n<p><span style=\"font-weight: 400;\">In this post, we\u2019ll show you <\/span><b>what emotion recognition is<\/b><span style=\"font-weight: 400;\">, and <\/span><b>how it can be used to extract emotions from social media visual posts<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<h2><strong>Definition of emotion recognition<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">There are many AI-powered solutions and programs that can be used to detect and recognize emotions. Here\u2019s how you can utilize emotion recognition for your communication strategy.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The program or model will try to assign an emotional expression to a face. Usually, it will use hints like mouth and eyebrow position, but as AI gets more advanced, there will be more cues to pick up on. Currently, emotion recognition models can recognize a person\u2019s facial expression as:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Happy<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Neutral<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Sad<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Surprised<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Fear (scared)<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Disgusted<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Angry<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Contemptuous<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">To help you understand how emotion extraction and recognition work, we\u2019ll show you some examples of the two machine learning models in action.<\/span><\/p>\n<h2><strong>How do we detect faces and emotions using Sotrender\u2019s machine learning models?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Sotrender uses data-driven machine learning approaches, specifically, <\/span><a href=\"https:\/\/mc.ai\/face-detection-neural-network-structure\/\" target=\"_blank\" rel=\"nofollow noopener\"><span style=\"font-weight: 400;\">MTCNN architecture<\/span><\/a><span style=\"font-weight: 400;\"> for face detection and <\/span><a href=\"https:\/\/medium.com\/@14prakash\/understanding-and-implementing-architectures-of-resnet-and-resnext-for-state-of-the-art-image-cf51669e1624\" target=\"_blank\" rel=\"nofollow noopener\"><span style=\"font-weight: 400;\">ResNet50 architecture<\/span><\/a><span style=\"font-weight: 400;\"> to recognize emotions. MTCNN combines three neural networks in order to predict face and landmark locations, which represent <\/span><b>key points in the face<\/b><span style=\"font-weight: 400;\"> such as the eyes. Eventually, you can draw a box around the detected face. ResNet50, on the other hand, is a neural network with multiple layers that recognizes emotions. Using an iterative process, it detects edges in the picture and then uses them to form representations of bigger features like a smile or eyes. Ultimately, they are used to predict an associated emotion. <\/span><b>To cut a long story short, we combine both of these architectures into a single model that will extract faces from the image and then predict emotions in the face.<\/b><\/p>\n<p><span style=\"font-weight: 400;\">As of right now, our model has 96% of the accuracy as the state-of-the-art models. Sotrender\u2019s models are able to analyze videos as well, which makes it possible for brands and organizations to learn more about <\/span><b>how often their ads and video content portray certain emotions<\/b><span style=\"font-weight: 400;\">. Since video content gets more engagement and requires more effort to create than static image posts, <\/span><b>it\u2019s worth it for brands and organizations to effectively plan and execute videos for their audiences<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<div style=\"width: 720px;\" class=\"wp-video\"><!--[if lt IE 9]><script>document.createElement('video');<\/script><![endif]-->\n<video class=\"wp-video-shortcode\" id=\"video-51249-1\" width=\"720\" height=\"480\" preload=\"metadata\" controls=\"controls\"><source type=\"video\/mp4\" src=\"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/SMW-LONDON-video_Trim_Trim.mp4?_=1\" \/><a href=\"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/SMW-LONDON-video_Trim_Trim.mp4\">https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/SMW-LONDON-video_Trim_Trim.mp4<\/a><\/video><\/div>\n<p>&nbsp;<\/p>\n<h2><strong>Step-by-step guide for face and emotion recognition<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Since you know a bit more about the methods we use, we\u2019ll show you how all of this comes together for a single image.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Step 1. We find ourselves an input image, and feed it into the face detector model (using MTCNN).<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Step 2. The faces get cropped out and become separate images from the original we used in the beginning.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Step 3. We can classify the emotions at this point, and use ResNet50. The images will no longer be cropped, but will instead be joined with the original input image. Here\u2019s what that looks like with visuals.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-8149 aligncenter\" src=\"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/image1-2.png\" alt=\"\" width=\"731\" height=\"287\" \/><\/p>\n<h2><strong>What Facebook and Instagram posts can we analyze with these models?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Facebook explained how users can download\/analyze images, videos, Stories, or albums from their platform. You can acquire the direct link to these forms of media through endpoints in Instagram\u2019s <\/span><a href=\"https:\/\/developers.facebook.com\/docs\/instagram-api\/\" target=\"_blank\" rel=\"nofollow noopener\"><span style=\"font-weight: 400;\">Graph API<\/span><\/a><span style=\"font-weight: 400;\">. Basically, users can get media URLs to get access to the images that you want to analyze from a given profile. Since Instagram is owned by Facebook, there would be similar endpoints for Facebook\u2019s API.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">You should know that there are some limitations. For one, <\/span><b>the API can only give you access to Business accounts<\/b><span style=\"font-weight: 400;\">. Second, there are also limitations when it comes to getting some media forms such as <\/span><a href=\"https:\/\/developers.facebook.com\/docs\/instagram-api\/reference\/user\/stories\/\" target=\"_blank\" rel=\"nofollow noopener\"><span style=\"font-weight: 400;\">Stories<\/span><\/a><span style=\"font-weight: 400;\">. In the case of Instagram Stories, we cannot download Instagram Live Stories, and since Stories are only available for 24 hours, if you\u2019re downloading a lot of data, you might not get to download all the Stories you were hoping to get.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Overall, it\u2019s good to be aware of the limitations and adjust your expectations for social media data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">You can read more about how we used the <\/span><a href=\"https:\/\/www.sotrender.com\/blog\/2020\/07\/machine-learning-emotion-recognition-pr\/\" target=\"_blank\" rel=\"dofollow noopener\"><span style=\"font-weight: 400;\">emotion recognition model to analyze Elon Musk\u2019s and Richard Branson\u2019s Instagram posts<\/span><\/a><span style=\"font-weight: 400;\"> and check what emotions they displayed the most on their profiles.<\/span><\/p>\n<h2><strong>How can I get more insights about my performance?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">You wouldn\u2019t be getting the full picture with only one set of data. Actually, knowing only how often emotions appear in your posts won\u2019t tell you how your audience reacts to it, or what you should change about your content.\u00a0<\/span><\/p>\n<p><b>Instead, we recommend taking all of it into consideration<\/b><span style=\"font-weight: 400;\">. You can combine emotion data and performance data to see how your posts have impacted your reach, engagement, and follower count. You can check these kinds of performance insights using native social media analytics built into platforms like Facebook or Instagram, or an external tool. If you use <\/span><a href=\"http:\/\/app.sotrender.com\/\" target=\"_blank\" rel=\"dofollow noopener\"><span style=\"font-weight: 400;\">Sotrender<\/span><\/a><span style=\"font-weight: 400;\">, you could gather data from longer and customizable periods of time to give you more specialized answers.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Just by clicking on the interactive graphs in the Sotrender app, you could check which of your posts had the best engagement and highest reach, and see if there is a particular emotion that comes up. If organic content that includes surprised and happy faces tends to get more comments, likes, and shares, then you\u2019ll have your answer. \ud83d\ude09<\/span><\/p>\n<div id=\"attachment_8151\" style=\"width: 755px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-8151\" class=\" wp-image-8151\" src=\"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/image2-1-1024x426.png\" alt=\"\" width=\"745\" height=\"310\" \/><p id=\"caption-attachment-8151\" class=\"wp-caption-text\">Sotrender\u2019s own engagement metric &#8211; Interactivity Index<\/p><\/div>\n<h2><strong>Ready to start using emotion recognition on your own profiles?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Machine learning can seem intimidating at first glance. However, if you start to understand the basics and logic behind the models, you\u2019ll get a better understanding of how your audience perceives your profile. At some point, simple social media analytics won\u2019t be enough to tell you more about the qualitative side of your social media presence, but machine learning will.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Interested in learning more about how you can use ML to discover new insights about your profile? Contact our <a href=\"mailto:sales@sotrender.com\">Sales Team<\/a> and we can discuss the ways that Sotrender\u2019s models can be useful for you.<\/span><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400;\">Read more about AI and machine learning here:<\/span><\/p>\n<ul>\n<li><a href=\"https:\/\/www.sotrender.com\/resources\/knowledge-base\/artificial-intelligence\/brand-visibility\" target=\"_blank\" rel=\"ndofollow noopener\">Can I compare the brand visibility of my own brand and my competition&#8217;s?<\/a><\/li>\n<li><a href=\"https:\/\/www.sotrender.com\/resources\/knowledge-base\/artificial-intelligence\/what-is-sentiment-analysis\/\" target=\"_blank\" rel=\"dofollow noopener\">What is sentiment analysis in social media?<\/a><\/li>\n<li><a href=\"https:\/\/www.sotrender.com\/resources\/knowledge-base\/artificial-intelligence\/audience-topic-modeling\" target=\"_blank\" rel=\"dofollow noopener\">What can I learn about my audience&#8217;s interests through topic modeling?<\/a><\/li>\n<\/ul>\n","protected":false},"author":131,"featured_media":51726,"template":"","yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What is emotion recognition and how does it work on images and videos?<\/title>\n<meta name=\"description\" content=\"In this post, we\u2019ll show you what emotion recognition is, and how it can be used to extract emotions from social media visual posts.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is emotion recognition and how does it work on images and videos?\" \/>\n<meta property=\"og:description\" content=\"In this post, we\u2019ll show you what emotion recognition is, and how it can be used to extract emotions from social media visual posts.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/\" \/>\n<meta property=\"og:site_name\" content=\"Sotrender Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Sotrender\" \/>\n<meta property=\"og:image\" content=\"https:\/\/wwwprod.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/domingo-alvarez-e-Cs3y8Mn6-Gk-unsplash-1-scaled.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1707\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@Sotrender\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/\",\"url\":\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/\",\"name\":\"What is emotion recognition and how does it work on images and videos?\",\"isPartOf\":{\"@id\":\"https:\/\/www.sotrender.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/domingo-alvarez-e-Cs3y8Mn6-Gk-unsplash-1-scaled.jpg\",\"datePublished\":\"2020-12-17T16:32:36+00:00\",\"dateModified\":\"2020-12-17T16:32:36+00:00\",\"description\":\"In this post, we\u2019ll show you what emotion recognition is, and how it can be used to extract emotions from social media visual posts.\",\"breadcrumb\":{\"@id\":\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#primaryimage\",\"url\":\"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/domingo-alvarez-e-Cs3y8Mn6-Gk-unsplash-1-scaled.jpg\",\"contentUrl\":\"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/domingo-alvarez-e-Cs3y8Mn6-Gk-unsplash-1-scaled.jpg\",\"width\":2560,\"height\":1707},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.sotrender.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"KBs\",\"item\":\"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"What is emotion recognition and how does it work on images and videos?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.sotrender.com\/blog\/#website\",\"url\":\"https:\/\/www.sotrender.com\/blog\/\",\"name\":\"Sotrender Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.sotrender.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is emotion recognition and how does it work on images and videos?","description":"In this post, we\u2019ll show you what emotion recognition is, and how it can be used to extract emotions from social media visual posts.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/","og_locale":"en_US","og_type":"article","og_title":"What is emotion recognition and how does it work on images and videos?","og_description":"In this post, we\u2019ll show you what emotion recognition is, and how it can be used to extract emotions from social media visual posts.","og_url":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/","og_site_name":"Sotrender Blog","article_publisher":"https:\/\/www.facebook.com\/Sotrender","og_image":[{"width":2560,"height":1707,"url":"https:\/\/wwwprod.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/domingo-alvarez-e-Cs3y8Mn6-Gk-unsplash-1-scaled.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_site":"@Sotrender","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/","url":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/","name":"What is emotion recognition and how does it work on images and videos?","isPartOf":{"@id":"https:\/\/www.sotrender.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#primaryimage"},"image":{"@id":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#primaryimage"},"thumbnailUrl":"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/domingo-alvarez-e-Cs3y8Mn6-Gk-unsplash-1-scaled.jpg","datePublished":"2020-12-17T16:32:36+00:00","dateModified":"2020-12-17T16:32:36+00:00","description":"In this post, we\u2019ll show you what emotion recognition is, and how it can be used to extract emotions from social media visual posts.","breadcrumb":{"@id":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#primaryimage","url":"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/domingo-alvarez-e-Cs3y8Mn6-Gk-unsplash-1-scaled.jpg","contentUrl":"https:\/\/www.sotrender.com\/blog\/wp-content\/uploads\/2020\/12\/domingo-alvarez-e-Cs3y8Mn6-Gk-unsplash-1-scaled.jpg","width":2560,"height":1707},{"@type":"BreadcrumbList","@id":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/emotion-recognition-images-videos\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.sotrender.com\/blog\/"},{"@type":"ListItem","position":2,"name":"KBs","item":"https:\/\/wwwprod.sotrender.com\/blog\/knowledge_base\/"},{"@type":"ListItem","position":3,"name":"What is emotion recognition and how does it work on images and videos?"}]},{"@type":"WebSite","@id":"https:\/\/www.sotrender.com\/blog\/#website","url":"https:\/\/www.sotrender.com\/blog\/","name":"Sotrender Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.sotrender.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/www.sotrender.com\/blog\/wp-json\/wp\/v2\/knowledge_base\/51249"}],"collection":[{"href":"https:\/\/www.sotrender.com\/blog\/wp-json\/wp\/v2\/knowledge_base"}],"about":[{"href":"https:\/\/www.sotrender.com\/blog\/wp-json\/wp\/v2\/types\/knowledge_base"}],"author":[{"embeddable":true,"href":"https:\/\/www.sotrender.com\/blog\/wp-json\/wp\/v2\/users\/131"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.sotrender.com\/blog\/wp-json\/wp\/v2\/media\/51726"}],"wp:attachment":[{"href":"https:\/\/www.sotrender.com\/blog\/wp-json\/wp\/v2\/media?parent=51249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}