Transformer in Transformer
Explore this paper's citation graph
Summary
It is pointed out that the attention inside these local patches are also essential for building visual transformers with high performance and a new architecture, namely, Transformer iN Transformer (TNT), is explored.
- Type
- preprint
- Published
- 2021-02-27
- Cited by
- 2,273
- References
- 56
- Access
- Open access
- OpenAlex
- https://openalex.org/W3133696297
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:232076027
Keywords
Transformer, Computer science, Architecture, Sentence, Artificial intelligence
References
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Care of aged doctors
- ImageNet Large Scale Visual Recognition Challenge
- Object recognition from local scale-invariant features
- ImageNet classification with deep convolutional neural networks
- Rethinking the Inception Architecture for Computer Vision
- Visualizing Data using t-SNE
- Deep Residual Learning for Image Recognition
- FractalNet: Ultra-Deep Neural Networks without Residuals
- Automated Flower Classification over a Large Number of Classes
- Feature Pyramid Networks for Object Detection
- Scene Parsing through ADE20K Dataset
- Squeeze-and-Excitation Networks
- The iNaturalist Species Classification and Detection Dataset
- Gaussian Error Linear Units (GELUs)
- Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- Non-local Neural Networks
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Cited by
- Transformers in Vision: A Survey
- Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search
- Going deeper with Image Transformers
- CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
- Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
- Dual-stream Network for Visual Recognition
- Prioritized Architecture Sampling with Monto-Carlo Tree Search
- CAT: Cross Attention in Vision Transformer
- KVT: k-NN Attention for Boosting Vision Transformers
- MLTR: Multi-Label Classification with Transformer
- A Survey of Transformers
- PVT v2: Improved baselines with Pyramid Vision Transformer
- What Makes for Hierarchical Vision Transformer?
- Contextual Transformer Networks for Visual Recognition
- DPT: Deformable Patch-based Transformer for Visual Recognition
- A Survey on Vision Transformer
- Congested Crowd Instance Localization with Dilated Convolutional Swin Transformer
- Towards Transferable Adversarial Attacks on Vision Transformers
Related papers
- A Study of a New Colour Granularity Formula
- Information granularity,information entropy and decision tree
- Study of Approximate Periodicity Mining with Multi-granularity Time
- Dynamic granularity selection based on local weighted accuracy and local likelihood ratio
- Accelerated multi-granularity reduction based on neighborhood rough sets
- Granularity-density relationship for electrophotography incorporating paper and toner granularities
- Tri-granularity attribute reduction of three-way concept lattices
- Granularity of Electrophotographic Images