Improving knowledge distillation using unified ensembles of specialized teachers
Explore this paper's citation graph
Summary
The effectiveness of the proposed method is demonstrated using three different image datasets, leading to improved distillation performance, even when compared with powerful state-of-the-art ensemble-based distillation methods.
- Type
- article
- Published
- 2021-03-21
- Cited by
- 16
- References
- 43
- Access
- Open access
- OpenAlex
- https://openalex.org/W3137193197
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:233304442
Keywords
Distillation, Computer science, Process (computing), Machine learning, Set (abstract data type)
References
- Unsupervised Knowledge Transfer Using Similarity Embeddings
- Data-free Parameter Pruning for Deep Neural Networks
- cuDNN: Efficient Primitives for Deep Learning
- Distilling the Knowledge in a Neural Network
- Knowledge Transfer Pre-training
- Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Quantized Convolutional Neural Networks for Mobile Devices
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size
- Model compression
- Policy Distillation
- Learning from Noisy Labels with Distillation
- In-datacenter performance analysis of a tensor processing unit
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Learning Efficient Object Detection Models with Knowledge Distillation
- Deep feature representation based on privileged knowledge transfer
- Boosting Self-Supervised Learning via Knowledge Transfer
- Neural Compatibility Modeling with Attentive Knowledge Distillation
- Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
Cited by
- Block-wise Dynamic Sparseness
- Detection of Male and Female Litchi Flowers Using YOLO-HPFD Multi-Teacher Feature Distillation and FPGA-Embedded Platform
- NRD-Net: a noise-resistant distillation network for accurate diagnosis of prostate cancer with bi-parametric MRI images
- Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices
- Blood Pressure Estimation Based on PPG and ECG Signals Using Knowledge Distillation
- Research on disease diagnosis based on teacher-student network and Raman spectroscopy
- LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives
- Research on the Great Multi-model Pyramid Training Framework and Enhanced Loss Function for Fine-grained Classification
- Personalized Federated Learning on long-tailed data via knowledge distillation and generated features
- Network Fission Ensembles for Low-Cost Self-Ensembles
- Online probabilistic knowledge distillation on cryptocurrency trading using Deep Reinforcement Learning
- Applications of Knowledge Distillation in Remote Sensing: A Survey
- LLM See, LLM Do: Leveraging Active Inheritance to Target Non-Differentiable Objectives
- Network Fission Ensembles for low-cost self-ensembles
- Applications of knowledge distillation in remote sensing: A survey
- Beyond text-image pairs: Triple-modal joint alignment and multi-teacher distillation for robust multimodal rumor detection
- Task Specialized Bone Fracture Detection with Attention Models and Knowledge distillation
Related papers
- Ensemble machine learning methods in screening electronic health records: A scoping review
- Theoretical Evaluation of Ensemble Machine Learning Techniques
- A Comparative Study of Different Ensemble Learning Techniques Using Wisconsin Breast Cancer Dataset
- Artificial Intelligence-Based Ensemble Learning Model for Prediction of Hepatitis C Disease
- ENSEMBLE LEARNING ALGORITHMS
- Computer-Aided Analysis and Planning of Set-Up Process
- Ensemble Learning Classification for Medical Diagnosis
- An efficient ensemble based machine learning approach for predicting Chronic Kidney Disease.