Representation Learning with Contrastive Predictive Coding
Explore this paper's citation graph
Summary
This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
- Type
- preprint
- Published
- 2018-07-10
- Cited by
- 14,322
- References
- 58
- Access
- Open access
- OpenAlex
- https://openalex.org/W2842511635
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:49670925
Keywords
Computer science, Predictive coding, Artificial intelligence, Feature learning, Machine learning
References
- Unsupervised Learning of Visual Representations Using Videos
- Unsupervised Visual Representation Learning by Context Prediction
- Librispeech: An ASR corpus based on public domain audio books
- The Kaldi Speech Recognition Toolkit
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books
- Efficient Estimation of Word Representations in Vector Space
- Show and tell: A neural image caption generator
- Annotating Expressions of Opinions and Emotions in Language
- Adaptive predictive coding of speech signals
- Learning Question Classifiers
- FaceNet: A unified embedding for face recognition and clustering
- Distance Metric Learning for Large Margin Nearest Neighbor Classification
- A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts
- ImageNet Large Scale Visual Recognition Challenge
- A theory of cortical responses
- Distributed Representations of Sentences and Documents
- Slow Feature Analysis: Unsupervised Learning of Invariances
- Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
- Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
- Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
Cited by
- Reservoir Computing Approaches for Representation and Classification of Multivariate Time Series
- Representation Learning in Sensory Cortex: A Theory
- Smart Health
- TherML: Thermodynamics of Machine Learning
- Mutual Information Maximization for Simple and Accurate Part-Of-Speech Induction
- Learning deep representations by mutual information estimation and maximization
- Unsupervised Learning via Meta-Learning
- EMI: Exploration with Mutual Information
- Generating Diverse Numbers of Diverse Keyphrases
- Variational Noise-Contrastive Estimation
- Invertible Residual Networks
- Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model
- Neural Image Compression for Gigapixel Histopathology Image Analysis
- Neural Predictive Belief Representations
- Improving speech emotion recognition via Transformer-based Predictive Coding through transfer learning
- Learning Actionable Representations with Goal-Conditioned Policies
- Learning Speaker Representations with Mutual Information
- β-VAEs can retain label information even at high compression
- Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks
- Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Related papers
- Unsupervised Visual Representation Learning by Context Prediction
- Improved Baselines with Momentum Contrastive Learning
- A Simple Framework for Contrastive Learning of Visual Representations
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Learning deep representations by mutual information estimation and maximization