Inferring the Why in Images

Summary

The results suggest that transferring knowledge from language into vision can help machines understand why a person might be performing an action in an image, and recently developed natural language models to mine knowledge stored in massive amounts of text.

Keywords

Computer science, Artificial intelligence, Computer vision, Computer graphics (images)

References

Anticipating Human Activities Using Object Affordances for Reactive Robotic ResponseHema Swetha Koppula, Ashutosh Saxena — 2013
Task assignment with unknown durationMor Harchol‐Balter — 2000
NEIL: Extracting Visual Knowledge from Web DataXinlei Chen, Abhinav Shrivastava, Abhinav Gupta — 2013
BabyTalk: Understanding and Generating Simple Image DescriptionsGirish Kulkarni, Visruth Premraj, Vicente Ordóñez — 2013
Language for learning complex human-object interactionsMitesh Patel, Carl Henrik Ek, Νikolaos Kyriazis — 2013
A PROCEDURE FOR COMPUTING THE K BEST SOLUTIONS TO DISCRETE OPTIMIZATION PROBLEMS AND ITS APPLICATION TO THE SHORTEST PATH PROBLEMEugene L. Lawler — 1972
Bringing Semantics into Focus Using Visual AbstractionC. Lawrence Zitnick, Devi Parikh — 2013
Max-Margin Early Event DetectorsMinh Hoai, Fernando De la Torre — 2012
Learning intentions for improved human motion predictionJos Elfring, René van de Molengraft, M. Steinbuch — 2013
Exploiting language models to recognize unseen actionsDieu Thu Le, Raffaella Bernardi, Jasper Uijlings — 2013
Cutting-plane training of structural SVMsThorsten Joachims, Thomas Finley, Chun-Nam Yu — 2009
Human Intent Prediction Using Markov Decision ProcessesCatharine McGhan, Ali Nasir, Ella Atkins — 2012
The Pascal Visual Object Classes (VOC) ChallengeMark Everingham, Luc Van Gool, Christopher K. I. Williams — 2009
Deep networks for predicting human intent with respect to objectsRichard Kelley, Liesl Wigand, Brian Hamilton — 2012
Predicting human intention in visual observations of hand/object interactionsDan Song, Νikolaos Kyriazis, Iason Oikonomidis — 2013
Patch to the Future: Unsupervised Visual PredictionJacob Walker, Abhinav Gupta, Martial Hebert — 2014
Visual Persuasion: Inferring Communicative Intents of ImagesJungseock Joo, Weixin Li, Francis F. Steen — 2014
Learning Models for Object Recognition from Natural Language DescriptionsJosiah Wang, Katja Markert, Mark Everingham — 2009
WordNet: A Lexical Database for EnglishGeorge A. Miller — 1995
Learning Everything about Anything: Webly-Supervised Visual Concept LearningSantosh Divvala, Ali Farhadi, Carlos Guestrin — 2014

Cited by

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading BooksYukun Zhu, Ryan Kiros, Rich Zemel — 2015
Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasksXiao Lin, Devi Parikh — 2015
Grasp type revisited: A modern perspective on a classical feature for visionYezhou Yang, Cornelia Fermüller, Yi Li — 2015
Visual Commonsense for Scene Understanding Using Perception, Semantic Parsing and ReasoningSomak Aditya, Yezhou Yang, Chitta Baral — 2015
MovieQA: Understanding Stories in Movies through Question-AnsweringMakarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen — 2015
Learning Common Sense through Visual AbstractionRamakrishna Vedantam, Xiao Lin, Tanmay Batra — 2015
Subjects and Their Objects: Localizing Interactees for a Person-Centric View of ImportanceChao-Yeh Chen, Kristen Grauman — 2016
Social LSTM: Human Trajectory Prediction in Crowded SpacesAlexandre Alahi, Kratarth Goel, Vignesh Ramanathan — 2016
Annotation Methodologies for Vision and Language Dataset CreationGitit Kehat, James Pustejovsky — 2016
Prediction of Manipulation ActionsCornelia Fermüller, Fang Wang, Yezhou Yang — 2016
A Corpus for Event LocalizationChris M. Ward — 2016
Learning human activities and poses with interconnected data sourcesChao-Yeh Chen — 2016
Automatic Generation of Grounded Visual QuestionsShijie Zhang, Lizhen Qu, Shaodi You — 2016
Asynchronous Temporal Fields for Action RecognitionGunnar A. Sigurdsson, Santosh Divvala, Ali Farhadi — 2016
Leveraging Multimodal Perspectives to Learn Common Sense for Vision and Language TasksXiaoyu Lin — 2017
From Line Drawings to Human Actions: Deep Neural Networks for Visual Data RepresentationF. Wang — 2017
From Recognition to Cognition: Visual Commonsense ReasoningRowan Zellers, Yonatan Bisk, Ali Farhadi — 2018
Leveraging Visual Question Answering for Image-Caption RankingXiao Lin, Devi Parikh — 2016
Visual7W: Grounded Question Answering in ImagesYuke Zhu, Oliver Groth, Michael S. Bernstein — 2015
Learning the Semantics of Manipulation ActionYezhou Yang, Yiannis Aloimonos, Cornelia Fermüller — 2015

Related papers

6-DOF object localization by combining monocular vision and robot arm kinematicsKun Liu, Weiwei Shang, Shuang Du — 2017
3-D tracking of a moving object by an active stereo vision systemMaki Tanaka, Noriaki Maru, F. Miyazaki — 1994
Self-monitoring to improve robustness of 3D object tracking for roboticsThomas Mörwald, Michael Zillich, Johann Prankl — 2011
Object-oriented stripe structured-light vision-guided robotLiang Zhang, Jianzhou Zhang — 2017
Robust object tracking based on RGB-D cameraWenjing Qi, Yinfei Yang, Meng Yi — 2014
An Object Detection and Pose Estimation Approach for Position Based Visual ServoingLei Shi — 2017
Tracking in 3D: Image Variability Decomposition for Recovering Object Pose and IlluminationPeter N. Belhumeur, Gregory D. Hager — 1999
Hand-eye calibration using a single image and robotic picking up using images lacking in contrastRiby Abraham Boby — 2020
Motion-based Object Detection and Tracking in Color Image SequenceBernd Heisele — 2000
Foreground object segmentation from binocular stereo videoK. M. Law, Stan Sclaroff — 2005