Aligning where to see and what to tell: image caption with region-based attention and scene factorization

Junqi Jin, Kun Fu, Runpeng Cui, Fei Sha, Changshui Zhang · arXiv.org · 2015

Summary

This paper proposes an image caption system that exploits the parallel structures between images and sentences and makes another novel modeling contribution by introducing scene-specific contexts that capture higher-level semantic information encoded in an image.

Type: preprint
Published: 2015-06-20
Cited by: 121
References: 30
Access: Open access
DOI: https://doi.org/10.48550/arxiv.1506.06272
arXiv: https://arxiv.org/abs/1506.06272
OpenAlex: https://openalex.org/W1785460851
Semantic Scholar: https://api.semanticscholar.org/CorpusID:17326626

Keywords

Computer science, Artificial intelligence, Salient, Perception, Natural language processing

References

Midge: Generating Image Descriptions From Computer Vision DetectionsMargaret Mitchell, Jesse Dodge, Amit Goyal — 2012
Generating Text with Recurrent Neural NetworksIlya Sutskever, James Martens, Geoffrey E. Hinton — 2011
Show, Attend and Tell: Neural Image Caption Generation with Visual AttentionKelvin Xu, Jimmy Ba, Ryan Kiros — 2015
Unifying Visual-Semantic Embeddings with Multimodal Neural Language ModelsRyan Kiros, Ruslan Salakhutdinov, Richard S. Zemel — 2014
Very Deep Convolutional Networks for Large-Scale Image RecognitionKaren Simonyan, Andrew Zisserman — 2014
Composing Simple Image Descriptions using Web-scale N-gramsSiming Li, Girish Kulkarni, Tamara L. Berg — 2011
Corpus-Guided Sentence Generation of Natural ImagesYezhou Yang, Ching L. Teo, Hal Daumé — 2011
Microsoft COCO Captions: Data Collection and Evaluation ServerXinlei Chen, Hao Fang, Tsung-Yi Lin — 2015
Show and tell: A neural image caption generatorOriol Vinyals, Alexander Toshev, Samy Bengio — 2014
Long-term recurrent convolutional networks for visual recognition and descriptionJeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama — 2014
BabyTalk: Understanding and Generating Simple Image DescriptionsGirish Kulkarni, Visruth Premraj, Vicente Ordóñez — 2013
Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber — 1997
Selective Search for Object RecognitionJasper Uijlings, Koen E. A. van de Sande, Theo Gevers — 2013
Factored conditional restricted Boltzmann Machines for modeling motion styleGraham W. Taylor, Geoffrey E. Hinton — 2009
Collecting Image Annotations Using Amazon’s Mechanical TurkCyrus Rashtchian, Peter Young, Micah Hodosh — 2010
The Stanford CoreNLP Natural Language Processing ToolkitChristopher D. Manning, Mihai Surdeanu, John Bauer — 2014
Learning Deep Features for Scene Recognition using Places DatabaseBolei Zhou, Àgata Lapedriza, Jianxiong Xiao — 2014
Image Description using Visual Dependency RepresentationsDesmond Elliott, Frank Keller — 2013
Collective Generation of Natural Image DescriptionsПолина Кузнецова, Vicente Ordóñez, Alexander Berg — 2012
Explain Images with Multimodal Recurrent Neural NetworksJunhua Mao, Wei Xu, Yi Yang — 2014

Cited by

ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question AnsweringKan Chen, Jiang Wang, Liang-Chieh Chen — 2015
What value high level concepts in vision to language problemsQi Wu, Chunhua Shen, Anton van den Hengel — 2015
Survey on the attention based RNN model and its applications in computer visionFeng Wang, David M. J. Tax — 2016
Image Captioning and Visual Question Answering Based on Attributes and Their Related External KnowledgeQi Wu, Chunhua Shen, Anton van den Hengel — 2016
What Value Do Explicit High Level Concepts Have in Vision to Language Problems?Qi Wu, Chunhua Shen, Lingqiao Liu — 2015
LSTM-in-LSTM for generating long descriptions of imagesJun Song, Siliang Tang, Jun Xiao — 2016
Image Captioning with both Object and Scene InformationXiangyang Li, Xinhang Song, Luis Herranz — 2016
Bootstrap, Review, Decode: Using Out-of-Domain Textual Data to Improve Image CaptioningWenhu Chen, Aurélien Lucchi, Thomas Hofmann — 2016
Dense Captioning with Joint Inference and Visual ContextLinjie Yang, Kevin Tang, Jianchao Yang — 2016
Attention-based Memory Selection Recurrent Network for Language ModelingDa-Rong Liu, Shun-Po Chuang, Hung-yi Lee — 2016
Semantic Compositional Networks for Visual CaptioningZhe Gan, Chuang Gan, Xiaodong He — 2016
Areas of Attention for Image CaptioningMarco Pedersoli, Thomas W. Lucas, Cordelia Schmid — 2016
Image Captioning and Visual Question Answering Based on Attributes and External KnowledgeQi Wu, Chunhua Shen, Peng Wang — 2016
Diverse Image Captioning via GroupTalkZhuhao Wang, Fei Wu, Weiming Lü — 2016
Can a Machine Generate Humanlike Language Descriptions for a Remote Sensing Image?Zhenwei Shi, Zhengxia Zou — 2017
Reference Based LSTM for Image CaptioningMinghai Chen, Guiguang Ding, Sicheng Zhao — 2017
Image Caption with Global-Local AttentionLing‐Hui Li, Sheng Tang, Lixi Deng — 2017
AMC: Attention Guided Multi-modal Correlation Learning for Image SearchKan Chen, Trung Bui, Fang Chen — 2017
Deep Reinforcement Learning-Based Image Captioning with Embedding RewardZhou Ren, Xiaoyu Wang, Ning Zhang — 2017
Bottom-Up and Top-Down Attention for Image Captioning and VQAPeter Anderson, Xiaodong He, Chris Buehler — 2017

Aligning where to see and what to tell: image caption with region-based attention and scene factorization

Summary

Keywords

References

Cited by

Related papers