Neural Word Embedding as Implicit Matrix Factorization
Explore this paper's citation graph
Summary
It is shown that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks, and conjecture that this stems from the weighted nature of SGNS's factorization.
- Type
- article
- Published
- 2014-12-08
- Cited by
- 2,049
- References
- 30
- OpenAlex
- https://openalex.org/W2125031621
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:1190093
Keywords
Word (group theory), Word embedding, Computer science, Context (archaeology), Matrix decomposition
References
- Experiments with LSA scoring: optimal rank and basis
- Efficient Estimation of Word Representations in Vector Space
- From Frequency to Meaning: Vector Space Models of Semantics
- Extracting semantic representations from word co-occurrence statistics: A computational study
- Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD
- The approximation of one matrix by another of lower rank
- Placing search in context: the concept revisited
- Matrix Factorization Techniques for Recommender Systems
- Similarity-Based Estimation of Word Cooccurrence Probabilities
- Learning word embeddings efficiently with noise-contrastive estimation
- A unified architecture for natural language processing: deep neural networks with multitask learning
- Domain and Function: A Dual-Space Model of Semantic Relations and Compositions
- Distributional Memory: A General Framework for Corpus-Based Semantics
- A Scalable Hierarchical Distributed Language Model
- word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
- Distributional Semantics in Technicolor
- Linguistic Regularities in Continuous Space Word Representations
- Distributed Representations of Words and Phrases and their Compositionality
- Word Representations: A Simple and General Method for Semi-Supervised Learning
- Word Association Norms, Mutual Information, and Lexicography
Cited by
- Contextual Correlation Preserving Multiview Featured Graph Clustering
- Semi-automatic Construction of Cross-period Thesaurus
- Analyzing Discourse Communities with Distributional Semantic Models
- WordRank: Learning Word Embeddings via Robust Ranking
- Neural context embeddings for automatic discovery of word senses
- How to Generate a Good Word Embedding
- Reasoning about Linguistic Regularities in Word Embeddings using Matrix Manifolds
- Random Walks on Context Spaces: Towards an Explanation of the Mysteries of Semantic Word Embeddings
- Linking GloVe with word2vec
- Text Segmentation based on Semantic Word Embeddings
- Comprehend DeepWalk as Matrix Factorization
- Improving Distributional Similarity with Lessons Learned from Word Embeddings
- "The Sum of Its Parts": Joint Learning of Word and Phrase Representations with Autoencoders
- Information and Incrementality in Syntactic Bootstrapping
- Using network science and text analytics to produce surveys in a scientific topic
- A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution
- Learning Word Representations with Hierarchical Sparse Coding
- Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding
- Book Reviews: Semantic Relations Between Nominals by Vivi Nastase, Preslav Nakov, Diarmuid Ó Séaghdha, and Stan Szpakowicz
- LINE: Large-scale Information Network Embedding
Related papers
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Efficient Estimation of Word Representations in Vector Space
- Distributional Structure
- Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
- GloVe: Global Vectors for Word Representation
- Linguistic Regularities in Sparse and Explicit Word Representations
- Visualizing Data using t-SNE