PANLP at MEDIQA 2019: Pre-trained Language Models, Transfer Learning and Knowledge Distillation
Explore this paper's citation graph
Summary
It is found that pre-trained language models can significantly outperform traditional deep learning models and be comparable with that obtained by the ensemble of that set of models.
- Type
- article
- Published
- 2019-08-01
- Cited by
- 37
- References
- 21
- Access
- Open access
- OpenAlex
- https://openalex.org/W2970645034
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:199379698
Keywords
Computer science, Artificial intelligence, Transformer, Task (project management), Transfer of learning
References
- Distilling the Knowledge in a Neural Network
- A large annotated corpus for learning natural language inference
- Learning deep structured semantic models for web search using clickthrough data
- GloVe: Global Vectors for Word Representation
- Recurrent Convolutional Neural Networks for Text Classification
- A Decomposable Attention Model for Natural Language Inference
- Enhanced LSTM for Natural Language Inference
- Weighted Transformer Network for Machine Translation
- Snorkel: Rapid Training Data Creation with Weak Supervision
- Stochastic Answer Networks for Machine Reading Comprehension
- Lessons from Natural Language Inference in the Clinical Domain
- Training Complex Models with Multi-Task Weak Supervision
- Multilingual Neural Machine Translation with Knowledge Distillation
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Star-Transformer
- SciBERT: Pretrained Contextualized Embeddings for Scientific Text
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Multi-Task Deep Neural Networks for Natural Language Understanding
- Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering
Cited by
- Surf at MEDIQA 2019: Improving Performance of Natural Language Inference in the Clinical Domain by Adopting Pre-trained Language Model
- Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering
- Investigating the Effect of Lexical Segmentation in Transformer-based Models on Medical Datasets
- TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
- Multi-Stage Distillation Framework for Massive Multi-lingual NER
- XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
- Medical Knowledge-enriched Textual Entailment Framework
- Reservoir Transformers
- Joint Summarization-Entailment Optimization for Consumer Health Question Understanding
- UCSD-Adobe at MEDIQA 2021: Transfer Learning and Answer Sentence Selection for Medical Summarization
- A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding
- Biomedical Question Answering: A Survey of Approaches and Challenges
- A survey on textual entailment based question answering
- F-PABEE: Flexible-Patience-Based Early Exiting For Single-Label and Multi-Label Text Classification Tasks
- Identifying the Question Similarity of Regulatory Documents in the Pharmaceutical Industry by Using the Recognizing Question Entailment System: Evaluation Study
- Learned Adapters Are Better Than Manually Designed Adapters
- BioSimCSE: BioMedical Sentence Embeddings using Contrastive learning
- Text2MDT: Extracting Medical Decision Trees from Medical Texts
- ReQuEST: A Small-Scale Multi-Task Model for Community Question-Answering Systems
- ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models
Related papers
- Performance of True Transfer Learning using CNN DenseNet121 for COVID-19 Detection from Chest X-Ray Images
- Deep Transfer Learning for the Multilabel Classification of Chest X-ray Images
- Improved topic-dependent language modeling using information retrieval techniques
- TRANSFER LEARNING BASED CNN FOR COVID 19
- Fake Reviews Detection through Ensemble Learning
- CNN Ensemble Learning Method for Transfer learning: A Review
- Enhancing sugarcane disease classification with ensemble deep learning: A comparative study with transfer learning techniques
- Ensemble Transfer Learning for Hand-sign Digit Image Classification
- Cross-domain and multi-task transfer learning of deep convolutional neural network for breast cancer diagnosis in digital breast tomosynthesis