SciBERT: Pretrained Contextualized Embeddings for Scientific Text
Explore this paper's citation graph
Summary
SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks and demonstrates statistically significant improvements over BERT.
- Type
- article
- Published
- 2019-03-26
- Cited by
- 227
- References
- 29
- OpenAlex
- https://openalex.org/W2922551710
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:85518318
Keywords
Computer science, Suite, Dependency grammar, Natural language processing, Variety (cybernetics)
References
- Chemical–gene relation extraction using recursive neural network
- Evaluation of PICO as a Knowledge Representation for Clinical Questions
- Neural Machine Translation of Rare Words with Subword Units
- An Overview of Microsoft Academic Service (MAS) and Applications
- Introduction to the Bio-entity Recognition Task at JNLPBA
- GENIA corpus - a semantically annotated corpus for bio-textmining
- NCBI Disease Corpus: A Resource for Disease Name Recognition and Concept Normalization
- ChemProt-3.0: a global chemical biology diseases mapping
- BioCreative V CDR task corpus: a resource for chemical disease relation extraction
- Deep Biaffine Attention for Neural Dependency Parsing
- Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging
- PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts
- Deep Contextualized Word Representations
- AllenNLP: A Deep Semantic Natural Language Processing Platform
- A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature
- Construction of the Literature Graph in Semantic Scholar
- Measuring the Evolution of a Scientific Field through Citation Frames
- Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts
- From POS tagging to dependency parsing for biomedical event extraction
- CollaboNet: collaboration of deep neural networks for biomedical named entity recognition
Cited by
- Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data
- Unsupervised Domain Adaptation of Contextualized Embeddings: A Case Study in Early Modern English
- Structural Scaffolds for Citation Intent Classification in Scientific Publications
- DoubleTransfer at MEDIQA 2019: Multi-Source Transfer Learning for Natural Language Understanding in the Medical Domain
- Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow
- Classifying German Animal Experiment Summaries with Multi-lingual BERT at CLEF eHealth 2019 Task 1
- NaCTeM-UoM @ CL-SciSumm 2019
- What Does the Evidence Say? Models to Help Make Sense of the Biomedical Literature
- Distant Supervision for Silver Label Generation of Software Mentions in Social Scientific Publications
- BioFLAIR: Pretrained Pooled Contextualized Embeddings for Biomedical Sequence Labeling Tasks
- Release Strategies and the Social Impacts of Language Models
- A distantly supervised dataset for automated data extraction from diagnostic studies
- A Context-based Framework for Modeling the Role and Function of On-line Resource Citations in Scientific Literature
- Sieg at MEDIQA 2019: Multi-task Neural Ensemble for Biomedical Inference and Entailment
- KU_ai at MEDIQA 2019: Domain-specific Pre-training and Transfer Learning for Medical NLI
- PANLP at MEDIQA 2019: Pre-trained Language Models, Transfer Learning and Knowledge Distillation
- Pre-Training BERT on Domain Resources for Short Answer Grading
- Extracting relations between outcomes and significance levels in Randomized Controlled Trials (RCTs) publications
- BERT, ELMo, USE and InferSent Sentence Encoders: The Panacea for Research-Paper Recommendation?
- UU_TAILS at MEDIQA 2019: Learning Textual Entailment in the Medical Domain
Related papers
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- GloVe: Global Vectors for Word Representation
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Long Short-Term Memory
- Distributed Representations of Words and Phrases and their Compositionality