Universitat Politècnica de Catalunya, profile picture
Uploaded byUniversitat Politècnica de Catalunya
PDF, PPTX296 views

Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)

The document outlines various models and techniques related to generative and discriminative tasks in language and vision, focusing on image and video captioning, visual question answering, and sign language translation. It references numerous studies and methods, including encoder-decoder representations, attention mechanisms, and multimodal machine translation. Key topics include combating data bias, dynamic memory networks, and advancements in visual reasoning and object grounding through language.

Download as PDF, PPTX
Xavier Giro-i-Nieto @DocXavi xavier.giro@upc.edu Associate Professor Universitat Politècnica de Catalunya Barcelona Supercomputing Center Slides @DocXavi Tutorial: One Perceptron to Rule Them All Part III: Language & Vision
2 Acknowledgments Mariona Carós Benet Oriol Amaia Salvador Santiago Pascual Marta R. Costa-jussà Francisco Roldan Issey Masuda Ionut Sorodoc Carina Silberer Gemma Boleda Carles Ventura Ioannis Kazakos Míriam Bellver Alba M. Herrera Amanda Duarte
3
4 Outline 1. Generative Models a. Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Representation Learning 4. Control Tasks
5 Outline 1. Generative Models a. Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Representation Learning 4. Control Tasks
6 Encoder Decoder Representation
7 #ShowAndTell Vinyals, Oriol, Alexander Toshev, Samy Bengio, and Dumitru Erhan. "Show and tell: A neural image caption generator." CVPR 2015. Image Captioning with RNN
8 #DeepImageSent Karpathy, Andrej, and Li Fei-Fei. "Deep visual-semantic alignments for generating image descriptions." CVPR 2015 (Slides by Marc Bolaños) Image Captioning with RNN
9 Xu, Kelvin, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention." ICML 2015 Image Captioning with RNN & Attention
10 Xu, Kelvin, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention." ICML 2015 Image Captioning with RNN & Attention
11 Cornia, Marcella, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. "Meshed-Memory Transformer for Image Captioning." CVPR 2020. [tweet] Image Captioning with Transformers
12 Johnson, Justin, Andrej Karpathy, and Li Fei-Fei. "Densecap: Fully convolutional localization networks for dense captioning." CVPR 2016 Dense Captioning
13 XAVI: “man has short hair”, “man with short hair” AMAIA:”a woman wearing a black shirt”, “ BOTH: “two men wearing black glasses” Johnson, Justin, Andrej Karpathy, and Li Fei-Fei. "Densecap: Fully convolutional localization networks for dense captioning." CVPR 2016 Dense Captioning
14 Recipe Generation Salvador, Amaia, Michal Drozdzal, Xavier Giro-i-Nieto, and Adriana Romero. "Inverse Cooking: Recipe Generation from Food Images." CVPR 2019.
15 Recipe Generation Title: Edamame corn salad Ingredients pepper, corn, onion, edamame, salt, vinegar, cilantro, avocado, oil Instructions - In a large bowl, combine edamame, corn, red onion, cilantro, avocado, and red bell pepper. - In a small bowl, whisk together olive oil, vinegar, salt, and pepper. - Pour dressing over edamame mixture and toss to coat. - Cover and refrigerate for at least 1 hour before serving. Salvador, Amaia, Michal Drozdzal, Xavier Giro-i-Nieto, and Adriana Romero. "Inverse Cooking: Recipe Generation from Food Images." CVPR 2019.
16 #Equalizer Burns, Kaylee, Lisa Anne Hendricks, Trevor Darrell, and Anna Rohrbach. "Women also Snowboard: Overcoming Bias in Captioning Models." ECCV 2018. Fighting Data Bias in Captioning
17 #Equalizer Burns, Kaylee, Lisa Anne Hendricks, Trevor Darrell, and Anna Rohrbach. "Women also Snowboard: Overcoming Bias in Captioning Models." ECCV 2018. Fighting Data Bias in Captioning
18 Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrel. Long-term Recurrent Convolutional Networks for Visual Recognition and Description, CVPR 2015. code Video Captioning
19 (Slides by Marc Bolaños) Pingbo Pan, Zhongwen Xu, Yi Yang,Fei Wu,Yueting Zhuang Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning, CVPR 2016. LSTM unit (2nd layer) Time Image t = 1 t = T hidden state at t = T first chunk of data Captioning: Video
20 Multimodal Machine Translation Challenge on Multimodal Image Translation: http://www.statmt.org/wmt17/multimodal-task.html#task1
21 Multimodal Machine Translation Sulubacak, Umut, Ozan Caglayan, Stig-Arne Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann. "Multimodal machine translation through visuals and speech." Machine Translation (2020): 1-51. [tweet]
22 Sign Language Translation with RNN+Att Camgoz, Necati Cihan, et al. Neural Sign Language Translation. CVPR 2018.
23 Sign Language Translation with Transformers Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, Richard Bowden, “Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation” CVPR 2020.
24 Assael, Yannis M., Brendan Shillingford, Shimon Whiteson, and Nando de Freitas. "LipNet: End-to-End Sentence-level Lipreading." (2016).
25 Lip Reading Assael, Yannis M., Brendan Shillingford, Shimon Whiteson, and Nando de Freitas. "LipNet: End-to-End Sentence-level Lipreading." (2016).
26 Chung, Joon Son, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. "Lip reading sentences in the wild." CVPR 2017
27 Lipreading: Watch, Listen, Attend & Spell Audio features Image features Chung, Joon Son, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. "Lip reading sentences in the wild." CVPR 2017
28 Lipreading: Watch, Listen, Attend & Spell Chung, Joon Son, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. "Lip reading sentences in the wild." CVPR 2017 Attention over output states from audio and video is computed at each timestep
29 Lipreading Afouras, Triantafyllos, Joon Son Chung, and Andrew Zisserman. "Deep Lip Reading: a comparison of models and an online application." Interspeech 2018.
30 Image Captioning Grounded on Detected Objects Lu, Jiasen and Yang, Jianwei and Batra, Dhruv and Parikh, Devi “Neural Baby Talk” CVPR 2018 [code]
31Lu, Jiasen and Yang, Jianwei and Batra, Dhruv and Parikh, Devi “Neural Baby Talk” CVPR 2018 [code] Image Captioning Grounded on Detected Objects
32Akbari, Hassan, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, and Shih-Fu Chang. "Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding." CVPR 2019. [code] Image Captioning Grounded on Heatmaps
33 Outline 1. Generative Models a. Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Representation Learning 4. Control Tasks
34 Encoder Decoder Representation
35 Reed, Scott, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. "Generative adversarial text to image synthesis." ICML 2016. Image Generation
36 Reed, Scott, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. "Generative adversarial text to image synthesis." ICML 2016. [code] Image Generation
37 Image Synthesis #StackGAN Zhang, Han, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, and Dimitris Metaxas. "Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks." ICCV 2017. [code]
38 Image Synthesis with Cycle Consistency #MirroGAN Qiao, Tingting, Jing Zhang, Duanqing Xu, and Dacheng Tao. "Mirrorgan: Learning text-to-image generation by redescription." CVPR 2019. [code]
39 Image Synthesis with Cycle Consistency #MirroGAN Qiao, Tingting, Jing Zhang, Duanqing Xu, and Dacheng Tao. "Mirrorgan: Learning text-to-image generation by redescription." CVPR 2019. [code]
40Justin Johnson, Agrim Gupta, Li Fei-Fei, “Image Generation from Scene Graphs” CVPR 2018 Image Generation via Scene Graphs
41 #Text2Scene Tan, Fuwen, Song Feng, and Vicente Ordonez. "Text2Scene: Generating Compositional Scenes From Textual Descriptions." CVPR 2019 [blog].
42 #CRAFT Gupta, Tanmay, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. "Imagine this! scripts to compositions to videos." ECCV 2018 Video Generation by Composition
43 Saunders, B., Camgoz, N. C., & Bowden, R. (2020). Progressive Transformers for End-to-End Sign Language Production. ECCV 2020. Sign Language Generation with Transformers
44 Lucas Ventura, Amanda Duarte, Xavier Giro-i-Nieto, “Can Everybody Sign Now ? Exploring Sign Language Video Generation from 2D Poses”. ECCV SLRTP Workshop 2020. Sign Language Generation (pose 2 pixels)
45 Outline 1. Generative Models a. Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Representation Learning 4. Control Tasks
46 Encoder Decoder Representation Encoder Representation
47 Visual Question Answering Antol, Stanislaw, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. "VQA: Visual question answering." CVPR 2015.
48 Visual Question Answering (VQA) Francisco Roldán, Issey Masuda, Santiago Pascual de la Puente, and Xavier Giro-i-Nieto. "Visual Question-Answering 2.0." ETSETB UPC TelecomBCN (2017).
49 Noh, H., Seo, P. H., & Han, B. Image question answering using convolutional neural network with dynamic parameter prediction. CVPR 2016 Dynamic Parameter Prediction Network (DPPnet) Visual Question Answering (VQA)
50 VQA: Dynamic Memory Networks (Slides and Slidecast by Santi Pascual): Xiong, Caiming, Stephen Merity, and Richard Socher. "Dynamic Memory Networks for Visual and Textual Question Answering." ICML 2016
51 Visual Reasoning #Clevr Johnson, Justin, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. "CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning." CVPR 2017
52 Visual Reasoning: Programming (Slides by Fran Roldan) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Fei-Fei Li, Larry Zitnick, Ross Girshick , “Inferring and Executing Programs for Visual Reasoning”. ICCV 2017 Program Generator Execution Engine
53 Visual Reasoning: Relation Networks #RN Santoro, Adam, David Raposo, David G. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. "A simple neural network module for relational reasoning." NIPS 2017. Relation Networks concatenate all possible pairs of objects with the an encoded question to later find the answer with a MLP.
54 Visual Dialog Das, Abhishek, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. "Visual Dialog." CVPR 2017 [Project]
55 Visual Dialog Caros, Mariona, Maite Garolera, Petia Radeva, and Xavier Giro-i-Nieto. "Automatic Reminiscence Therapy for Dementia." ICMR 2020. [talk] Demo @ ICMR 2020 (Wednesday 11:00am)
56 Visual Dialog Caros, Mariona, Maite Garolera, Petia Radeva, and Xavier Giro-i-Nieto. "Automatic Reminiscence Therapy for Dementia." ICMR 2020. [talk]
57 Hate Speech Detection in Memes Benet Oriol, Cristian Canton, Xavier Giro-i-Nieto, “Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderation”. NeurIPS 2019 AI for Good Workshop. [code] Hate Speech Detection
58 Outline 1. Generative Models a. Text b. Image 2. Discriminative Models a. Text b. Image 3. Representation Learning 4. Control Tasks
59 Encoder Decoder Representation Encoder Representation
60 Niu, Yulei, Hanwang Zhang, Zhiwu Lu, and Shih-Fu Chang. "Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions." arXiv preprint arXiv:1907.03609 (2019). Objects from Referring Expressions
61 Video Objects from Referring Expressions Li, Zhenyang, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Arnold WM Smeulders. "Tracking by natural language specification." CVPR 2017. [code]
62 Video Object Detection with Transformers Sadhu, A., Chen, K., & Nevatia, R. (2020). Video Object Grounding using Semantic Roles in Language Description. CVPR 2020.
63 #Mattnet Yu, Licheng, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg. "Mattnet: Modular attention network for referring expression comprehension." CVPR 2018. [code] Segments from Referring Expressions
64 Khoreva, Anna, Anna Rohrbach, and Bernt Schiele. "Video object segmentation with language referring expressions." ACCV 2018. Video Objects from Referring Expressions
65 Herrera-Palacio, Alba, Carles Ventura, and Xavier Giro-i-Nieto. "Video object linguistic grounding." ACM Multimedia Workshops 2019. Video Objects from Referring Expressions
66 #RefVOS Bellver, Miriam, Carles Ventura, Carina Silberer, Ioannis Kazakos, Jordi Torres, and Xavier Giro-i-Nieto. "RefVOS: A Closer Look at Referring Expressions for Video Object Segmentation." arXiv preprint arXiv:2010.00263 (2020). Video Objects from Referring Expressions
67 #RefVOS Bellver, Miriam, Carles Ventura, Carina Silberer, Ioannis Kazakos, Jordi Torres, and Xavier Giro-i-Nieto. "RefVOS: A Closer Look at Referring Expressions for Video Object Segmentation." arXiv preprint arXiv:2010.00263 (2020). Video Objects from Referring Expressions
68 #SynthRef Ioannis Kazakos, Bellver, Miriam, Carles Ventura, Carina Silberer, and Xavier Giro-i-Nieto, “Generation of Synthetic Referring Expressions for Object Segmentation” (submitted) Synthetic Expressions w/ Scene Graphs
69 #SynthRef Ioannis Kazakos, Bellver, Miriam, Carles Ventura, Carina Silberer, and Xavier Giro-i-Nieto, “Generation of Synthetic Referring Expressions for Object Segmentation” (submitted)
Segments from Questions Gan, Chuang, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. "VQS: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation." ICCV 2017.
71 Outline 1. Generative Models a. Text b. Image 2. Discriminative Models a. Text b. Image 3. Representation Learning 4. Control Tasks
72 Encoder Encoder Representation
73 Joint Representations (Embeddings) #Devise Frome, Andrea, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeff Dean, and Tomas Mikolov. "Devise: A deep visual-semantic embedding model." NIPS 2013
74 Zero-shot learning Socher, R., Ganjoo, M., Manning, C. D., & Ng, A., Zero-shot learning through cross-modal transfer. NIPS 2013 [slides] [code] No images from “cat” in the training set... ...but they can still be recognised as “cats” thanks to the representations learned from text .
75 Multimodal Retrieval Kiros, Ryan, Ruslan Salakhutdinov, and Richard S. Zemel. "Unifying visual-semantic embeddings with multimodal neural language models." NeurIPS 2014 Deep Learning Workshop.
76 Multimodal Retrieval Aytar, Yusuf, Lluis Castrejon, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. "Cross-Modal Scene Networks." CVPR 2016.
77 Multimodal Retrieval Aytar, Yusuf, Lluis Castrejon, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. "Cross-Modal Scene Networks." CVPR 2016.
78 Image and text retrieval with joint embeddings. Joint Neural Embeddings #pic2recipe Amaia Salvador, Nicholas Haynes, Yusuf Aytar, Javier Marín, Ferda Ofli, Ingmar Weber, Antonio Torralba, “Learning Cross-modal Embeddings for Cooking Recipes and Food Images”. CVPR 2017 [video]
79 #pic2recipe Amaia Salvador, Nicholas Haynes, Yusuf Aytar, Javier Marín, Ferda Ofli, Ingmar Weber, Antonio Torralba, “Learning Cross-modal Embeddings for Cooking Recipes and Food Images”. CVPR 2017 [video] Joint Neural Embeddings
80 Joint Neural Embeddings joint embedding LSTM Bidirectional LSTM #pic2recipe Amaia Salvador, Nicholas Haynes, Yusuf Aytar, Javier Marín, Ferda Ofli, Ingmar Weber, Antonio Torralba, “Learning Cross-modal Embeddings for Cooking Recipes and Food Images”. CVPR 2017
81 Representations Sariyildiz, Mert Bulent, Julien Perez, and Diane Larlus. "Learning Visual Representations with Caption Annotations." ECCV 2020. [tweet]
82 Representations #ViLBERT Lu, Jiasen, Dhruv Batra, Devi Parikh, and Stefan Lee. "Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks." NeurIPS 2019. [MIT talk by Devih Parikh] [demo] Visual Task: Predict the visual categories for the masked video frame Language Task: Predict the masked word (same as in language-only BERT).
83 Representations #ViLBERT Lu, Jiasen, Dhruv Batra, Devi Parikh, and Stefan Lee. "Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks." NeurIPS 2019. [MIT talk by Devih Parikh] [demo] Multimodal Task: Predict whether the video frames correspond to the caption.
84 Representations #VideoBERT Sun, Chen, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. "Videobert: A joint model for video and language representation learning." ICCV 2019.
85 Representations #VideoBERT Sun, Chen, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. "Videobert: A joint model for video and language representation learning." ICCV 2019. Rich representations can be used to retrieve matching video frames, which are encoded after vector quantization.
86 Representations #VirTEX Karan Desai, Justin Johnson, “VirTex: Learning Visual Representations from Textual Annotations” arXiv 2020 [tweet]
87 Learning Language from Video Doughty, Hazel, Ivan Laptev, Walterio Mayol-Cuevas, and Dima Damen. "Action Modifiers: Learning from Adverbs in Instructional Videos." CVPR 2020..
88 Learning Language from Video Surís, Dídac, Dave Epstein, Heng Ji, Shih-Fu Chang, and Carl Vondrick. "Learning to Learn Words from Visual Scenes." ECCV 2020.
89 Outline 1. Generative Models a. Text b. Image 2. Discriminative Models a. Text b. Image 3. Representation Learning 4. Control Tasks
90 Platforms for Embodied AI #Habitat Savva, Manolis, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub et al. "Habitat: A platform for embodied ai research." ICCV 2019. [site]
91 Navigation Fried, Daniel, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. "Speaker-Follower Models for Vision-and-Language Navigation." NeurIPS 2018.
92 Navigation #R2R Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., ... & van den Hengel, A. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. CVPR 2018. [tweet]
93 Navigation #RxR Alexander Ku and Peter Anderson and Roma Patel and Eugene Ie and Jason Baldridge, “Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding” EMNLP 2020.
94 Navigation Ünal, Emre, Ozan Arkan Can, and Yücel Yemez. "Visually Grounded Language Learning For Robot Navigation." ACMMM Workshops 2019.
95 Object manipulation Hill, F., Lampinen, A. K., Schneider, R., Clark, S., Botvinick, M., McClelland, J. L., & Santoro, A. Environmental drivers of systematicity and generalization in a situated agent. ICLR 2020. [talk]
96 Outline 1. Generative Models a. Text b. Image 2. Discriminative Models a. Text b. Image 3. Representation Learning 4. Control Tasks
97 My take home message 1. Generative Models a. Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Feature Learning 4. Control Tasks
Xavier Giro-i-Nieto @DocXavi xavier.giro@upc.edu Associate Professor Universitat Politècnica de Catalunya Barcelona Supercomputing Center Was this tutorial helpful ? Please consider citing: Go raibh maith agat / Thank you Giro-i-Nieto, X. One Perceptron to Rule Them All: Language, Vision, Audio and Speech. In Proceedings of the 2020 International Conference on Multimedia Retrieval (pp. 7-8).

More Related Content

PDF
One Perceptron to Rule Them All: Language and Vision
byUniversitat Politècnica de Catalunya
 
PDF
Self-supervised Audiovisual Learning - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
PDF
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
byUniversitat Politècnica de Catalunya
 
PDF
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
byUniversitat Politècnica de Catalunya
 
PDF
Deep Video Object Tracking 2020 - Xavier Giro - UPC TelecomBCN Barcelona
byUniversitat Politècnica de Catalunya
 
PDF
Self-supervised Audiovisual Learning 2020 - Xavier Giro-i-Nieto - UPC Telecom...
byUniversitat Politècnica de Catalunya
 
PDF
Deep Video Object Tracking - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
PDF
Deep Audio and Vision - Eva Mohedano - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
One Perceptron to Rule Them All: Language and Vision
byUniversitat Politècnica de Catalunya
 
Self-supervised Audiovisual Learning - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
byUniversitat Politècnica de Catalunya
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
byUniversitat Politècnica de Catalunya
 
Deep Video Object Tracking 2020 - Xavier Giro - UPC TelecomBCN Barcelona
byUniversitat Politècnica de Catalunya
 
Self-supervised Audiovisual Learning 2020 - Xavier Giro-i-Nieto - UPC Telecom...
byUniversitat Politècnica de Catalunya
 
Deep Video Object Tracking - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
Deep Audio and Vision - Eva Mohedano - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 

What's hot

PDF
Multimodal Deep Learning (D4L4 Deep Learning for Speech and Language UPC 2017)
byUniversitat Politècnica de Catalunya
 
PDF
Self-supervised Visual Learning 2020 - Xavier Giro-i-Nieto - UPC Barcelona
byUniversitat Politècnica de Catalunya
 
PDF
Wav2Pix: Speech-conditioned face generation using Generative Adversarial Netw...
byUniversitat Politècnica de Catalunya
 
PDF
Video Analysis (D4L2 2017 UPC Deep Learning for Computer Vision)
byUniversitat Politècnica de Catalunya
 
PDF
Neural Architectures for Still Images - Xavier Giro- UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
PDF
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
byUniversitat Politècnica de Catalunya
 
PDF
Advanced Deep Architectures (D2L6 Deep Learning for Speech and Language UPC 2...
byUniversitat Politècnica de Catalunya
 
PDF
Deep Learning for Video: Action Recognition (UPC 2018)
byUniversitat Politècnica de Catalunya
 
PDF
Self-supervised Learning from Video Sequences - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
PDF
Deep Language and Vision (DLSL D2L4 2018 UPC Deep Learning for Speech and Lan...
byUniversitat Politècnica de Catalunya
 
PDF
Language and Vision (D3L5 2017 UPC Deep Learning for Computer Vision)
byUniversitat Politècnica de Catalunya
 
PDF
Deep Video Object Segmentation - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
PDF
Video Analysis with Convolutional Neural Networks (Master Computer Vision Bar...
byUniversitat Politècnica de Catalunya
 
PDF
Deep Learning for Video: Language (UPC 2018)
byUniversitat Politècnica de Catalunya
 
PDF
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
byUniversitat Politècnica de Catalunya
 
PDF
Deep Learning Architectures for Video - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
PDF
Deep Learning from Videos (UPC 2018)
byUniversitat Politècnica de Catalunya
 
PDF
Learning with Videos (D4L4 2017 UPC Deep Learning for Computer Vision)
byUniversitat Politècnica de Catalunya
 
PDF
Speaker ID II (D4L1 Deep Learning for Speech and Language UPC 2017)
byUniversitat Politècnica de Catalunya
 
PDF
Welcome (D1L1 2017 UPC Deep Learning for Computer Vision)
byUniversitat Politècnica de Catalunya
 
Multimodal Deep Learning (D4L4 Deep Learning for Speech and Language UPC 2017)
byUniversitat Politècnica de Catalunya
 
Self-supervised Visual Learning 2020 - Xavier Giro-i-Nieto - UPC Barcelona
byUniversitat Politècnica de Catalunya
 
Wav2Pix: Speech-conditioned face generation using Generative Adversarial Netw...
byUniversitat Politècnica de Catalunya
 
Video Analysis (D4L2 2017 UPC Deep Learning for Computer Vision)
byUniversitat Politècnica de Catalunya
 
Neural Architectures for Still Images - Xavier Giro- UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
byUniversitat Politècnica de Catalunya
 
Advanced Deep Architectures (D2L6 Deep Learning for Speech and Language UPC 2...
byUniversitat Politècnica de Catalunya
 
Deep Learning for Video: Action Recognition (UPC 2018)
byUniversitat Politècnica de Catalunya
 
Self-supervised Learning from Video Sequences - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
Deep Language and Vision (DLSL D2L4 2018 UPC Deep Learning for Speech and Lan...
byUniversitat Politècnica de Catalunya
 
Language and Vision (D3L5 2017 UPC Deep Learning for Computer Vision)
byUniversitat Politècnica de Catalunya
 
Deep Video Object Segmentation - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
Video Analysis with Convolutional Neural Networks (Master Computer Vision Bar...
byUniversitat Politècnica de Catalunya
 
Deep Learning for Video: Language (UPC 2018)
byUniversitat Politècnica de Catalunya
 
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
byUniversitat Politècnica de Catalunya
 
Deep Learning Architectures for Video - Xavier Giro - UPC Barcelona 2019
byUniversitat Politècnica de Catalunya
 
Deep Learning from Videos (UPC 2018)
byUniversitat Politècnica de Catalunya
 
Learning with Videos (D4L4 2017 UPC Deep Learning for Computer Vision)
byUniversitat Politècnica de Catalunya
 
Speaker ID II (D4L1 Deep Learning for Speech and Language UPC 2017)
byUniversitat Politècnica de Catalunya
 
Welcome (D1L1 2017 UPC Deep Learning for Computer Vision)
byUniversitat Politècnica de Catalunya
 

Similar to Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)

PDF
Deep Language and Vision by Amaia Salvador (Insight DCU 2018)
byUniversitat Politècnica de Catalunya
 
PDF
Deep Language and Vision - Xavier Giro-i-Nieto - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
PDF
Deep Speech and Vision - Xavier Giro-i-Nieto - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
PDF
Multi modal retrieval and generation with deep distributed models
byRoelof Pieters
 
PDF
A Survey on Cross-Modal Embedding
byYasuhide Miura
 
PDF
“Understand the Multimodal World with Minimal Supervision,” a Keynote Present...
byEdge AI and Vision Alliance
 
PDF
Show and Tell_ A Neural Image Caption Generator.pdf
byjyyyukk
 
PDF
Vision and Language: Past, Present and Future
byGoergen Institute for Data Science
 
PPTX
Image captioning
byMuhammad Zbeedat
 
PDF
One Perceptron to Rule them All: Deep Learning for Multimedia #A2IC2018
byUniversitat Politècnica de Catalunya
 
PDF
One Perceptron to Rule Them All (Re-Work Deep Learning Summit, London 2017)
byUniversitat Politècnica de Catalunya
 
PDF
Once Perceptron to Rule Them all: Deep Learning for Multimedia
byUniversitat Politècnica de Catalunya
 
PDF
Language and Vision (D2L11 Insight@DCU Machine Learning Workshop 2017)
byUniversitat Politècnica de Catalunya
 
PPTX
Semantic Summarization of videos, Semantic Summarization of videos
bydarsh228313
 
PPTX
Image captions.pptx
byRohanBorgalli
 
PDF
IRJET- Visual Information Narrator using Neural Network
byIRJET Journal
 
PDF
IRJET- Extension to Visual Information Narrator using Neural Network
byIRJET Journal
 
PPTX
Recognize, Describe, and Generate: Introduction of Recent Work at MIL
byYoshitaka Ushiku
 
PDF
Video + Language 2019
byGoergen Institute for Data Science
 
PDF
Video + Language
byGoergen Institute for Data Science
 
Deep Language and Vision by Amaia Salvador (Insight DCU 2018)
byUniversitat Politècnica de Catalunya
 
Deep Language and Vision - Xavier Giro-i-Nieto - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
Deep Speech and Vision - Xavier Giro-i-Nieto - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
Multi modal retrieval and generation with deep distributed models
byRoelof Pieters
 
A Survey on Cross-Modal Embedding
byYasuhide Miura
 
“Understand the Multimodal World with Minimal Supervision,” a Keynote Present...
byEdge AI and Vision Alliance
 
Show and Tell_ A Neural Image Caption Generator.pdf
byjyyyukk
 
Vision and Language: Past, Present and Future
byGoergen Institute for Data Science
 
Image captioning
byMuhammad Zbeedat
 
One Perceptron to Rule them All: Deep Learning for Multimedia #A2IC2018
byUniversitat Politècnica de Catalunya
 
One Perceptron to Rule Them All (Re-Work Deep Learning Summit, London 2017)
byUniversitat Politècnica de Catalunya
 
Once Perceptron to Rule Them all: Deep Learning for Multimedia
byUniversitat Politècnica de Catalunya
 
Language and Vision (D2L11 Insight@DCU Machine Learning Workshop 2017)
byUniversitat Politècnica de Catalunya
 
Semantic Summarization of videos, Semantic Summarization of videos
bydarsh228313
 
Image captions.pptx
byRohanBorgalli
 
IRJET- Visual Information Narrator using Neural Network
byIRJET Journal
 
IRJET- Extension to Visual Information Narrator using Neural Network
byIRJET Journal
 
Recognize, Describe, and Generate: Introduction of Recent Work at MIL
byYoshitaka Ushiku
 
Video + Language 2019
byGoergen Institute for Data Science
 
Video + Language
byGoergen Institute for Data Science
 

More from Universitat Politècnica de Catalunya

PDF
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
byUniversitat Politècnica de Catalunya
 
PDF
Open challenges in sign language translation and production
byUniversitat Politècnica de Catalunya
 
PDF
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
byUniversitat Politècnica de Catalunya
 
PDF
Deep Generative Learning for All - The Gen AI Hype (Spring 2024)
byUniversitat Politècnica de Catalunya
 
PPTX
Discovery and Learning of Navigation Goals from Pixels in Minecraft
byUniversitat Politècnica de Catalunya
 
PDF
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
byUniversitat Politècnica de Catalunya
 
PPTX
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
byUniversitat Politècnica de Catalunya
 
PDF
Intepretability / Explainable AI for Deep Neural Networks
byUniversitat Politècnica de Catalunya
 
PDF
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
byUniversitat Politècnica de Catalunya
 
PDF
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
byUniversitat Politècnica de Catalunya
 
PDF
Deep Generative Learning for All
byUniversitat Politècnica de Catalunya
 
PDF
Learn2Sign : Sign language recognition and translation using human keypoint e...
byUniversitat Politècnica de Catalunya
 
PDF
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
byUniversitat Politècnica de Catalunya
 
PDF
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
byUniversitat Politècnica de Catalunya
 
PDF
The Transformer - Xavier Giró - UPC Barcelona 2021
byUniversitat Politècnica de Catalunya
 
PDF
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
byUniversitat Politècnica de Catalunya
 
PDF
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
byUniversitat Politècnica de Catalunya
 
PDF
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
byUniversitat Politècnica de Catalunya
 
PDF
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
byUniversitat Politècnica de Catalunya
 
PDF
Curriculum Learning for Recurrent Video Object Segmentation
byUniversitat Politècnica de Catalunya
 
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
byUniversitat Politècnica de Catalunya
 
Open challenges in sign language translation and production
byUniversitat Politècnica de Catalunya
 
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
byUniversitat Politècnica de Catalunya
 
Deep Generative Learning for All - The Gen AI Hype (Spring 2024)
byUniversitat Politècnica de Catalunya
 
Discovery and Learning of Navigation Goals from Pixels in Minecraft
byUniversitat Politècnica de Catalunya
 
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
byUniversitat Politècnica de Catalunya
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
byUniversitat Politècnica de Catalunya
 
Intepretability / Explainable AI for Deep Neural Networks
byUniversitat Politècnica de Catalunya
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
byUniversitat Politècnica de Catalunya
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
byUniversitat Politècnica de Catalunya
 
Deep Generative Learning for All
byUniversitat Politècnica de Catalunya
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
byUniversitat Politècnica de Catalunya
 
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
byUniversitat Politècnica de Catalunya
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
byUniversitat Politècnica de Catalunya
 
The Transformer - Xavier Giró - UPC Barcelona 2021
byUniversitat Politècnica de Catalunya
 
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
byUniversitat Politècnica de Catalunya
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
byUniversitat Politècnica de Catalunya
 
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
byUniversitat Politècnica de Catalunya
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
byUniversitat Politècnica de Catalunya
 
Curriculum Learning for Recurrent Video Object Segmentation
byUniversitat Politècnica de Catalunya
 

Recently uploaded

PPTX
Sneha-Verma |MBA Business Analyst Portfolio | 2025
byimsnehavrma
 
PPTX
Factor Analysis in machine learning –.pptx
byVignan's foundation for science technology and research(deeemed to be university)
 
PPTX
D1. Mystery writing.pptxjdjjeiriekirirrre
byrehanatariq75070
 
PPTX
SALONI-ASPIRING-DATA-ANALYST MBA STUDENT
bySaloniNarwal
 
PDF
analisis mediante tecnicas de inteligencia artificial
byJULYANDREAGOMEZCAMPE
 
PPTX
Fraud_Detection_Using_Technology_Presentation Mehmet sarikurat.pptx
byMasihullahAli1
 
PPT
PROJECT SCHEDIULING ABOUT SOFTWARE DEVELOPMENT ACTIVITIES
bypriyankaR797112
 
PDF
2-Bilgisayarlı Görüntü İşlemeye Başlamak İçin Temel OpenCV Fonksiyonları - Ge...
byTaknTez
 
PPTX
Daily Slides Presentation in Colorful Polka Dot Style.pptx
byiamjinfe
 
PPTX
UCSP_Module_1_Presentation culture and society.pptx
byakcsbutuan
 
PPTX
relationalfffffffffffffffffffffffff algebra.pptx
byscrollinside07
 
PDF
wow super , fantastic booooo andtha pekuuu acunetics - testhtml5 Report.pdf
byferov57646
 
PDF
How to Buy Snapchat Account( 9.7) A Step-by-Step Guide ....pdf
bycfh9687
 
PPTX
ORACLE ADMIN WORKSHOP PRESENTATION OCP.pptx
bykausaratg
 
PPT
Lecturrrrrrrrrrrrrrrrrrrrrrrrre-4 SQL.ppt
byscrollinside07
 
PDF
How SunTec Data Boosted AI Accuracy with Scalable Data Annotation Services?
bySunTec Data
 
PPT
3.Relational M333333333333333333333333333333odel.ppt
byscrollinside07
 
PDF
Garden-Life-Brand-Launch-Case-Study-By-Accelero-Agency.pdf
byacceleroagency
 
PDF
Comprehensive Market Analysis of ESPN with Strategic Recommendations.pdf
byKelishaCyrus
 
DOCX
Buy Verified Apple Apple Pay Accounts - USA.docx
byhqpnrkbepwmr
 
Sneha-Verma |MBA Business Analyst Portfolio | 2025
byimsnehavrma
 
Factor Analysis in machine learning –.pptx
byVignan's foundation for science technology and research(deeemed to be university)
 
D1. Mystery writing.pptxjdjjeiriekirirrre
byrehanatariq75070
 
SALONI-ASPIRING-DATA-ANALYST MBA STUDENT
bySaloniNarwal
 
analisis mediante tecnicas de inteligencia artificial
byJULYANDREAGOMEZCAMPE
 
Fraud_Detection_Using_Technology_Presentation Mehmet sarikurat.pptx
byMasihullahAli1
 
PROJECT SCHEDIULING ABOUT SOFTWARE DEVELOPMENT ACTIVITIES
bypriyankaR797112
 
2-Bilgisayarlı Görüntü İşlemeye Başlamak İçin Temel OpenCV Fonksiyonları - Ge...
byTaknTez
 
Daily Slides Presentation in Colorful Polka Dot Style.pptx
byiamjinfe
 
UCSP_Module_1_Presentation culture and society.pptx
byakcsbutuan
 
relationalfffffffffffffffffffffffff algebra.pptx
byscrollinside07
 
wow super , fantastic booooo andtha pekuuu acunetics - testhtml5 Report.pdf
byferov57646
 
How to Buy Snapchat Account( 9.7) A Step-by-Step Guide ....pdf
bycfh9687
 
ORACLE ADMIN WORKSHOP PRESENTATION OCP.pptx
bykausaratg
 
Lecturrrrrrrrrrrrrrrrrrrrrrrrre-4 SQL.ppt
byscrollinside07
 
How SunTec Data Boosted AI Accuracy with Scalable Data Annotation Services?
bySunTec Data
 
3.Relational M333333333333333333333333333333odel.ppt
byscrollinside07
 
Garden-Life-Brand-Launch-Case-Study-By-Accelero-Agency.pdf
byacceleroagency
 
Comprehensive Market Analysis of ESPN with Strategic Recommendations.pdf
byKelishaCyrus
 
Buy Verified Apple Apple Pay Accounts - USA.docx
byhqpnrkbepwmr
 

Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)

  • 1.
    Xavier Giro-i-Nieto @DocXavi xavier.giro@upc.edu Associate Professor UniversitatPolitècnica de Catalunya Barcelona Supercomputing Center Slides @DocXavi Tutorial: One Perceptron to Rule Them All Part III: Language & Vision
  • 2.
  • 3.
  • 4.
    4 Outline 1. Generative Models a.Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Representation Learning 4. Control Tasks
  • 5.
    5 Outline 1. Generative Models a.Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Representation Learning 4. Control Tasks
  • 6.
  • 7.
    7 #ShowAndTell Vinyals, Oriol,Alexander Toshev, Samy Bengio, and Dumitru Erhan. "Show and tell: A neural image caption generator." CVPR 2015. Image Captioning with RNN
  • 8.
    8 #DeepImageSent Karpathy, Andrej,and Li Fei-Fei. "Deep visual-semantic alignments for generating image descriptions." CVPR 2015 (Slides by Marc Bolaños) Image Captioning with RNN
  • 9.
    9 Xu, Kelvin, JimmyBa, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention." ICML 2015 Image Captioning with RNN & Attention
  • 10.
    10 Xu, Kelvin, JimmyBa, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention." ICML 2015 Image Captioning with RNN & Attention
  • 11.
    11 Cornia, Marcella, MatteoStefanini, Lorenzo Baraldi, and Rita Cucchiara. "Meshed-Memory Transformer for Image Captioning." CVPR 2020. [tweet] Image Captioning with Transformers
  • 12.
    12 Johnson, Justin, AndrejKarpathy, and Li Fei-Fei. "Densecap: Fully convolutional localization networks for dense captioning." CVPR 2016 Dense Captioning
  • 13.
    13 XAVI: “man has shorthair”, “man with short hair” AMAIA:”a woman wearing a black shirt”, “ BOTH: “two men wearing black glasses” Johnson, Justin, Andrej Karpathy, and Li Fei-Fei. "Densecap: Fully convolutional localization networks for dense captioning." CVPR 2016 Dense Captioning
  • 14.
    14 Recipe Generation Salvador, Amaia,Michal Drozdzal, Xavier Giro-i-Nieto, and Adriana Romero. "Inverse Cooking: Recipe Generation from Food Images." CVPR 2019.
  • 15.
    15 Recipe Generation Title: Edamamecorn salad Ingredients pepper, corn, onion, edamame, salt, vinegar, cilantro, avocado, oil Instructions - In a large bowl, combine edamame, corn, red onion, cilantro, avocado, and red bell pepper. - In a small bowl, whisk together olive oil, vinegar, salt, and pepper. - Pour dressing over edamame mixture and toss to coat. - Cover and refrigerate for at least 1 hour before serving. Salvador, Amaia, Michal Drozdzal, Xavier Giro-i-Nieto, and Adriana Romero. "Inverse Cooking: Recipe Generation from Food Images." CVPR 2019.
  • 16.
    16 #Equalizer Burns, Kaylee,Lisa Anne Hendricks, Trevor Darrell, and Anna Rohrbach. "Women also Snowboard: Overcoming Bias in Captioning Models." ECCV 2018. Fighting Data Bias in Captioning
  • 17.
    17 #Equalizer Burns, Kaylee,Lisa Anne Hendricks, Trevor Darrell, and Anna Rohrbach. "Women also Snowboard: Overcoming Bias in Captioning Models." ECCV 2018. Fighting Data Bias in Captioning
  • 18.
    18 Jeffrey Donahue, LisaAnne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrel. Long-term Recurrent Convolutional Networks for Visual Recognition and Description, CVPR 2015. code Video Captioning
  • 19.
    19 (Slides by MarcBolaños) Pingbo Pan, Zhongwen Xu, Yi Yang,Fei Wu,Yueting Zhuang Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning, CVPR 2016. LSTM unit (2nd layer) Time Image t = 1 t = T hidden state at t = T first chunk of data Captioning: Video
  • 20.
    20 Multimodal Machine Translation Challengeon Multimodal Image Translation: http://www.statmt.org/wmt17/multimodal-task.html#task1
  • 21.
    21 Multimodal Machine Translation Sulubacak,Umut, Ozan Caglayan, Stig-Arne Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann. "Multimodal machine translation through visuals and speech." Machine Translation (2020): 1-51. [tweet]
  • 22.
    22 Sign Language Translationwith RNN+Att Camgoz, Necati Cihan, et al. Neural Sign Language Translation. CVPR 2018.
  • 23.
    23 Sign Language Translationwith Transformers Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, Richard Bowden, “Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation” CVPR 2020.
  • 24.
    24 Assael, Yannis M.,Brendan Shillingford, Shimon Whiteson, and Nando de Freitas. "LipNet: End-to-End Sentence-level Lipreading." (2016).
  • 25.
    25 Lip Reading Assael, YannisM., Brendan Shillingford, Shimon Whiteson, and Nando de Freitas. "LipNet: End-to-End Sentence-level Lipreading." (2016).
  • 26.
    26 Chung, Joon Son,Andrew Senior, Oriol Vinyals, and Andrew Zisserman. "Lip reading sentences in the wild." CVPR 2017
  • 27.
    27 Lipreading: Watch, Listen,Attend & Spell Audio features Image features Chung, Joon Son, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. "Lip reading sentences in the wild." CVPR 2017
  • 28.
    28 Lipreading: Watch, Listen,Attend & Spell Chung, Joon Son, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. "Lip reading sentences in the wild." CVPR 2017 Attention over output states from audio and video is computed at each timestep
  • 29.
    29 Lipreading Afouras, Triantafyllos, JoonSon Chung, and Andrew Zisserman. "Deep Lip Reading: a comparison of models and an online application." Interspeech 2018.
  • 30.
    30 Image Captioning Groundedon Detected Objects Lu, Jiasen and Yang, Jianwei and Batra, Dhruv and Parikh, Devi “Neural Baby Talk” CVPR 2018 [code]
  • 31.
    31Lu, Jiasen andYang, Jianwei and Batra, Dhruv and Parikh, Devi “Neural Baby Talk” CVPR 2018 [code] Image Captioning Grounded on Detected Objects
  • 32.
    32Akbari, Hassan, SveborKaraman, Surabhi Bhargava, Brian Chen, Carl Vondrick, and Shih-Fu Chang. "Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding." CVPR 2019. [code] Image Captioning Grounded on Heatmaps
  • 33.
    33 Outline 1. Generative Models a.Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Representation Learning 4. Control Tasks
  • 34.
  • 35.
    35 Reed, Scott, ZeynepAkata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. "Generative adversarial text to image synthesis." ICML 2016. Image Generation
  • 36.
    36 Reed, Scott, ZeynepAkata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. "Generative adversarial text to image synthesis." ICML 2016. [code] Image Generation
  • 37.
    37 Image Synthesis #StackGAN Zhang,Han, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, and Dimitris Metaxas. "Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks." ICCV 2017. [code]
  • 38.
    38 Image Synthesis withCycle Consistency #MirroGAN Qiao, Tingting, Jing Zhang, Duanqing Xu, and Dacheng Tao. "Mirrorgan: Learning text-to-image generation by redescription." CVPR 2019. [code]
  • 39.
    39 Image Synthesis withCycle Consistency #MirroGAN Qiao, Tingting, Jing Zhang, Duanqing Xu, and Dacheng Tao. "Mirrorgan: Learning text-to-image generation by redescription." CVPR 2019. [code]
  • 40.
    40Justin Johnson, AgrimGupta, Li Fei-Fei, “Image Generation from Scene Graphs” CVPR 2018 Image Generation via Scene Graphs
  • 41.
    41 #Text2Scene Tan, Fuwen,Song Feng, and Vicente Ordonez. "Text2Scene: Generating Compositional Scenes From Textual Descriptions." CVPR 2019 [blog].
  • 42.
    42 #CRAFT Gupta, Tanmay,Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. "Imagine this! scripts to compositions to videos." ECCV 2018 Video Generation by Composition
  • 43.
    43 Saunders, B., Camgoz,N. C., & Bowden, R. (2020). Progressive Transformers for End-to-End Sign Language Production. ECCV 2020. Sign Language Generation with Transformers
  • 44.
    44 Lucas Ventura, AmandaDuarte, Xavier Giro-i-Nieto, “Can Everybody Sign Now ? Exploring Sign Language Video Generation from 2D Poses”. ECCV SLRTP Workshop 2020. Sign Language Generation (pose 2 pixels)
  • 45.
    45 Outline 1. Generative Models a.Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Representation Learning 4. Control Tasks
  • 46.
  • 47.
    47 Visual Question Answering Antol,Stanislaw, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. "VQA: Visual question answering." CVPR 2015.
  • 48.
    48 Visual Question Answering(VQA) Francisco Roldán, Issey Masuda, Santiago Pascual de la Puente, and Xavier Giro-i-Nieto. "Visual Question-Answering 2.0." ETSETB UPC TelecomBCN (2017).
  • 49.
    49 Noh, H., Seo,P. H., & Han, B. Image question answering using convolutional neural network with dynamic parameter prediction. CVPR 2016 Dynamic Parameter Prediction Network (DPPnet) Visual Question Answering (VQA)
  • 50.
    50 VQA: Dynamic MemoryNetworks (Slides and Slidecast by Santi Pascual): Xiong, Caiming, Stephen Merity, and Richard Socher. "Dynamic Memory Networks for Visual and Textual Question Answering." ICML 2016
  • 51.
    51 Visual Reasoning #Clevr Johnson,Justin, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. "CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning." CVPR 2017
  • 52.
    52 Visual Reasoning: Programming (Slidesby Fran Roldan) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Fei-Fei Li, Larry Zitnick, Ross Girshick , “Inferring and Executing Programs for Visual Reasoning”. ICCV 2017 Program Generator Execution Engine
  • 53.
    53 Visual Reasoning: RelationNetworks #RN Santoro, Adam, David Raposo, David G. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. "A simple neural network module for relational reasoning." NIPS 2017. Relation Networks concatenate all possible pairs of objects with the an encoded question to later find the answer with a MLP.
  • 54.
    54 Visual Dialog Das, Abhishek,Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. "Visual Dialog." CVPR 2017 [Project]
  • 55.
    55 Visual Dialog Caros, Mariona,Maite Garolera, Petia Radeva, and Xavier Giro-i-Nieto. "Automatic Reminiscence Therapy for Dementia." ICMR 2020. [talk] Demo @ ICMR 2020 (Wednesday 11:00am)
  • 56.
    56 Visual Dialog Caros, Mariona,Maite Garolera, Petia Radeva, and Xavier Giro-i-Nieto. "Automatic Reminiscence Therapy for Dementia." ICMR 2020. [talk]
  • 57.
    57 Hate Speech Detectionin Memes Benet Oriol, Cristian Canton, Xavier Giro-i-Nieto, “Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderation”. NeurIPS 2019 AI for Good Workshop. [code] Hate Speech Detection
  • 58.
    58 Outline 1. Generative Models a.Text b. Image 2. Discriminative Models a. Text b. Image 3. Representation Learning 4. Control Tasks
  • 59.
  • 60.
    60 Niu, Yulei, HanwangZhang, Zhiwu Lu, and Shih-Fu Chang. "Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions." arXiv preprint arXiv:1907.03609 (2019). Objects from Referring Expressions
  • 61.
    61 Video Objects fromReferring Expressions Li, Zhenyang, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Arnold WM Smeulders. "Tracking by natural language specification." CVPR 2017. [code]
  • 62.
    62 Video Object Detectionwith Transformers Sadhu, A., Chen, K., & Nevatia, R. (2020). Video Object Grounding using Semantic Roles in Language Description. CVPR 2020.
  • 63.
    63 #Mattnet Yu, Licheng,Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg. "Mattnet: Modular attention network for referring expression comprehension." CVPR 2018. [code] Segments from Referring Expressions
  • 64.
    64 Khoreva, Anna, AnnaRohrbach, and Bernt Schiele. "Video object segmentation with language referring expressions." ACCV 2018. Video Objects from Referring Expressions
  • 65.
    65 Herrera-Palacio, Alba, CarlesVentura, and Xavier Giro-i-Nieto. "Video object linguistic grounding." ACM Multimedia Workshops 2019. Video Objects from Referring Expressions
  • 66.
    66 #RefVOS Bellver, Miriam,Carles Ventura, Carina Silberer, Ioannis Kazakos, Jordi Torres, and Xavier Giro-i-Nieto. "RefVOS: A Closer Look at Referring Expressions for Video Object Segmentation." arXiv preprint arXiv:2010.00263 (2020). Video Objects from Referring Expressions
  • 67.
    67 #RefVOS Bellver, Miriam,Carles Ventura, Carina Silberer, Ioannis Kazakos, Jordi Torres, and Xavier Giro-i-Nieto. "RefVOS: A Closer Look at Referring Expressions for Video Object Segmentation." arXiv preprint arXiv:2010.00263 (2020). Video Objects from Referring Expressions
  • 68.
    68 #SynthRef Ioannis Kazakos,Bellver, Miriam, Carles Ventura, Carina Silberer, and Xavier Giro-i-Nieto, “Generation of Synthetic Referring Expressions for Object Segmentation” (submitted) Synthetic Expressions w/ Scene Graphs
  • 69.
    69 #SynthRef Ioannis Kazakos,Bellver, Miriam, Carles Ventura, Carina Silberer, and Xavier Giro-i-Nieto, “Generation of Synthetic Referring Expressions for Object Segmentation” (submitted)
  • 70.
    Segments from Questions Gan,Chuang, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. "VQS: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation." ICCV 2017.
  • 71.
    71 Outline 1. Generative Models a.Text b. Image 2. Discriminative Models a. Text b. Image 3. Representation Learning 4. Control Tasks
  • 72.
  • 73.
    73 Joint Representations (Embeddings) #DeviseFrome, Andrea, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeff Dean, and Tomas Mikolov. "Devise: A deep visual-semantic embedding model." NIPS 2013
  • 74.
    74 Zero-shot learning Socher, R.,Ganjoo, M., Manning, C. D., & Ng, A., Zero-shot learning through cross-modal transfer. NIPS 2013 [slides] [code] No images from “cat” in the training set... ...but they can still be recognised as “cats” thanks to the representations learned from text .
  • 75.
    75 Multimodal Retrieval Kiros, Ryan,Ruslan Salakhutdinov, and Richard S. Zemel. "Unifying visual-semantic embeddings with multimodal neural language models." NeurIPS 2014 Deep Learning Workshop.
  • 76.
    76 Multimodal Retrieval Aytar, Yusuf,Lluis Castrejon, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. "Cross-Modal Scene Networks." CVPR 2016.
  • 77.
    77 Multimodal Retrieval Aytar, Yusuf,Lluis Castrejon, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. "Cross-Modal Scene Networks." CVPR 2016.
  • 78.
    78 Image and textretrieval with joint embeddings. Joint Neural Embeddings #pic2recipe Amaia Salvador, Nicholas Haynes, Yusuf Aytar, Javier Marín, Ferda Ofli, Ingmar Weber, Antonio Torralba, “Learning Cross-modal Embeddings for Cooking Recipes and Food Images”. CVPR 2017 [video]
  • 79.
    79 #pic2recipe Amaia Salvador,Nicholas Haynes, Yusuf Aytar, Javier Marín, Ferda Ofli, Ingmar Weber, Antonio Torralba, “Learning Cross-modal Embeddings for Cooking Recipes and Food Images”. CVPR 2017 [video] Joint Neural Embeddings
  • 80.
    80 Joint Neural Embeddings joint embedding LSTMBidirectional LSTM #pic2recipe Amaia Salvador, Nicholas Haynes, Yusuf Aytar, Javier Marín, Ferda Ofli, Ingmar Weber, Antonio Torralba, “Learning Cross-modal Embeddings for Cooking Recipes and Food Images”. CVPR 2017
  • 81.
    81 Representations Sariyildiz, Mert Bulent,Julien Perez, and Diane Larlus. "Learning Visual Representations with Caption Annotations." ECCV 2020. [tweet]
  • 82.
    82 Representations #ViLBERT Lu, Jiasen,Dhruv Batra, Devi Parikh, and Stefan Lee. "Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks." NeurIPS 2019. [MIT talk by Devih Parikh] [demo] Visual Task: Predict the visual categories for the masked video frame Language Task: Predict the masked word (same as in language-only BERT).
  • 83.
    83 Representations #ViLBERT Lu, Jiasen,Dhruv Batra, Devi Parikh, and Stefan Lee. "Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks." NeurIPS 2019. [MIT talk by Devih Parikh] [demo] Multimodal Task: Predict whether the video frames correspond to the caption.
  • 84.
    84 Representations #VideoBERT Sun, Chen,Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. "Videobert: A joint model for video and language representation learning." ICCV 2019.
  • 85.
    85 Representations #VideoBERT Sun, Chen,Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. "Videobert: A joint model for video and language representation learning." ICCV 2019. Rich representations can be used to retrieve matching video frames, which are encoded after vector quantization.
  • 86.
    86 Representations #VirTEX Karan Desai,Justin Johnson, “VirTex: Learning Visual Representations from Textual Annotations” arXiv 2020 [tweet]
  • 87.
    87 Learning Language fromVideo Doughty, Hazel, Ivan Laptev, Walterio Mayol-Cuevas, and Dima Damen. "Action Modifiers: Learning from Adverbs in Instructional Videos." CVPR 2020..
  • 88.
    88 Learning Language fromVideo Surís, Dídac, Dave Epstein, Heng Ji, Shih-Fu Chang, and Carl Vondrick. "Learning to Learn Words from Visual Scenes." ECCV 2020.
  • 89.
    89 Outline 1. Generative Models a.Text b. Image 2. Discriminative Models a. Text b. Image 3. Representation Learning 4. Control Tasks
  • 90.
    90 Platforms for EmbodiedAI #Habitat Savva, Manolis, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub et al. "Habitat: A platform for embodied ai research." ICCV 2019. [site]
  • 91.
    91 Navigation Fried, Daniel, RonghangHu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. "Speaker-Follower Models for Vision-and-Language Navigation." NeurIPS 2018.
  • 92.
    92 Navigation #R2R Anderson, P.,Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., ... & van den Hengel, A. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. CVPR 2018. [tweet]
  • 93.
    93 Navigation #RxR Alexander Kuand Peter Anderson and Roma Patel and Eugene Ie and Jason Baldridge, “Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding” EMNLP 2020.
  • 94.
    94 Navigation Ünal, Emre, OzanArkan Can, and Yücel Yemez. "Visually Grounded Language Learning For Robot Navigation." ACMMM Workshops 2019.
  • 95.
    95 Object manipulation Hill, F.,Lampinen, A. K., Schneider, R., Clark, S., Botvinick, M., McClelland, J. L., & Santoro, A. Environmental drivers of systematicity and generalization in a situated agent. ICLR 2020. [talk]
  • 96.
    96 Outline 1. Generative Models a.Text b. Image 2. Discriminative Models a. Text b. Image 3. Representation Learning 4. Control Tasks
  • 97.
    97 My take homemessage 1. Generative Models a. Text b. Vision 2. Discriminative Models a. Text b. Vision 3. Feature Learning 4. Control Tasks
  • 98.
    Xavier Giro-i-Nieto @DocXavi xavier.giro@upc.edu Associate Professor UniversitatPolitècnica de Catalunya Barcelona Supercomputing Center Was this tutorial helpful ? Please consider citing: Go raibh maith agat / Thank you Giro-i-Nieto, X. One Perceptron to Rule Them All: Language, Vision, Audio and Speech. In Proceedings of the 2020 International Conference on Multimedia Retrieval (pp. 7-8).