Knowledge distillation provides an effective way to transfer knowledge via teacher-student learning, where most existing distillation approaches apply a fixed pre-trained model as teacher to supervise the learning of student network. This manner usually brings in a big capability gap between teacher and student networks during learning. Recent researches have observed thata small teacher-student capability gap can facilitate knowledge transfer. Inspired by that, we propose an evolutionary knowledge distillation approach to improve the transfer effectiveness of teacher knowledge. Instead of a fixed pre-trained teacher, an evolutionary teacher is learned online and consistently transfers intermediate knowledge to supervise student network learning on-the-fly. To enhance intermediate knowledge representation and mimicking, several simple guided modules are introduced between corresponding teacher-student blocks. In this way, the student can simultaneously obtain rich internal knowledge and capture its growth process, leading to effective student network learning. Extensive experiments clearly demonstrate the effectiveness of our approach as well as good adaptability in the low-resolution and few-sample scenarios.

            
@article{zhang2021ekd,
    author={Kangkai Zhang, Chunhui Zhang, Shikun Li, Dan Zeng, and Shiming Ge},
    title={Student Network Learning via Evolutionary Knowledge Distillation}, 
    journal={IEEE Transactions on Circuits and Systems for Video Technology},
    year={2021},
    volume={},
    number={},
    pages={1-13}
}

        

Face recognition in the wild is now advancing towards light-weight models, fast inference speed and resolution-adapted capability. In this paper, we propose a bridge distillation approach to turn a complex face model pretrained on private high-resolution faces into a light-weight one for low-resolution face recognition. In our approach, such a cross-dataset resolution-adapted knowledge transfer problem is solved via two-step distillation. In the first step, we conduct cross-dataset distillation to transfer the prior knowledge from private high-resolution faces to public high-resolution faces and generate compact and discriminative features. In the second step, the resolution-adapted distillation is conducted to further transfer the prior knowledge to synthetic low-resolution faces via multi-task learning. By learning low-resolution face representations and mimicking the adapted high-resolution knowledge, a light-weight student model can be constructed with high efficiency and promising accuracy in recognizing low-resolution faces. Experimental results show that the student model performs impressively in recognizing low-resolution faces with only 0.21M parameters and 0.057MB memory. Meanwhile, its speed reaches up to 14,705, 934 and 763 faces per second on GPU, CPU and mobile phone, respectively.

            
@article{ge2020bd,
    author={Shiming  Ge and Shengwei Zhao and Chenyu Li and Yu Zhang and Jia Li},
    title={Efficient Low-Resolution Face Recognition via Bridge Distillation}, 
    journal={IEEE Transactions on Image Processing},
    year={2020},
    volume={29},
    number={},
    pages={6898-6908}
}

        

In spite of great success in many image recognition tasks achieved by recent deep models, directly applying them to recognize low-resolution images may suffer from low accuracy due to the missing of informative details during resolution degradation. However, these images are still recognizable for subjects who are familiar with the corresponding high-resolution ones. Inspired by that, we propose a teacher-student learning approach to facilitate low-resolution image recognition via hybrid order relational knowledge distillation. The approach refers to three streams: the teacher stream is pretrained to recognize high-resolution images in high accuracy, the student stream is learned to identify low-resolution images by mimicking the teacher's behaviors, and the extra assistant stream is introduced as bridge to help knowledge transfer across the teacher to the student. To extract sufficient knowledge for reducing the loss in accuracy, the learning of student is supervised with multiple losses, which preserves the similarities in various order relational structures. In this way, the capability of recovering missing details of familiar low-resolution images can be effectively enhanced, leading to a better knowledge transfer. Extensive experiments on metric learning, low-resolution image classification and low-resolution face recognition tasks show the effectiveness of our approach, while taking reduced models.

            
@inproceedings{AAAI20/Ge/HORKD,
    author    = {Shiming Ge and Kangkai Zhang and Haolin Liu and Yingying Hua and Shengwei Zhao and Xin Jin and Hao Wen},
    title     = {Look One and More: Distilling Hybrid Order Relational Knowledge for Cross-Resolution Image Recognition},
    booktitle = {The Thirty-Fourth {AAAI} Conference on Artificial Intelligence, {AAAI}
                    2020, The Thirty-Second Innovative Applications of Artificial Intelligence
                    Conference, {IAAI} 2020, The Tenth {AAAI} Symposium on Educational
                    Advances in Artificial Intelligence, {EAAI} 2020, New York, NY, USA,
                    February 7-12, 2020},
    pages     = {10845--10852},
    publisher = {{AAAI} Press},
    year      = {2020},
}

        

Typically, the deployment of face recognition models in the wild needs to identify low-resolution faces with extremely low computational cost. To address this problem, a feasible solution is compressing a complex face model to achieve higher speed and lower memory at the cost of minimal performance drop. Inspired by that, this paper proposes a learning approach to recognize low-resolution faces via selective knowledge distillation. In this approach, a two-stream convolutional neural network (CNN) is first initialized to recognize high-resolution faces and resolution-degraded faces with a teacher stream and a student stream, respectively. The teacher stream is represented by a complex CNN for high-accuracy recognition, and the student stream is represented by a much simpler CNN for low-complexity recognition. To avoid significant performance drop at the student stream, we then selectively distil the most informative facial features from the teacher stream by solving a sparse graph optimization problem, which are then used to regularize the fine-tuning process of the student stream. In this way, the student stream is actually trained by simultaneously handling two tasks with limited computational resources: approximating the most informative facial cues via feature regression, and recovering the missing facial cues via low-resolution face classification. Experimental results show that the student stream performs impressively in recognizing low-resolution faces and costs only 0.15-MB memory and runs at 418 faces per second on CPU and 9433 faces per second on GPU.

            
@article{TIP19/Ge/SKD,
    author    = {Shiming Ge and Shengwei Zhao and Chenyu Li and Jia Li},
    title     = {Low-Resolution Face Recognition in the Wild via Selective Knowledge Distillation},
    journal   = {{IEEE} Trans. Image Process.},
    volume    = {28},
    number    = {4},
    pages     = {2051--2062},
    year      = {2019},
}

        

Low-resolution face recognition in the wild still is an open problem. In this paper, we propose to address this problem via a novel learning approach called Mixed-Domain Distillation (MDD). The approach applies a teacher-student framework to mix and distill knowledge from four different domain datasets, including private high-resolution, public high-resolution, public low-resolution web and target lowresolution wild face datasets. In this way, high-resolution knowledge from the well-trained complex teacher model is first adapted to public high-resolution faces and then transferred to a simply student model. The student model is designed to identify low-resolution faces, and could perform face recognition in the wild effectively and efficiently. Experimental results show that our proposed model outperforms several existing models for low-resolution face recognition in the wild.

            
@inproceedings{BigMM19/Zhao/MDD,
    author    = {Shengwei Zhao and Xindi Gao and Shikun Li and Shiming Ge},
    title     = {Low-Resolution Face Recognition in the Wild with Mixed-Domain Distillation},
    booktitle = {Fifth {IEEE} International Conference on Multimedia Big Data, BigMM
                    2019, Singapore, September 11-13, 2019},
    pages     = {148--154},
    publisher = {{IEEE}},
    year      = {2019},
}        

        

Is it possible to train an effective face recognition model with fewer shots that works efficiently on low-resolution faces in the wild? To answer this question, this paper proposes a few-shot knowledge distillation approach to learn an ultrafast face recognizer via two steps. In the first step, we initialize a simple yet effective face recognition model on synthetic low-resolution faces by distilling knowledge from an existing complex model. By removing the redundancies in both face images and the model structure, the initial model can provide an ultrafast speed with impressive recognition accuracy. To further adapt this model into the wild scenarios with fewer faces per person, the second step refines the model via few-shot learning by incorporating a relation module that compares low-resolution query faces with faces in the support set. In this manner, the performance of the model can be further enhanced with only fewer low-resolution faces in the wild. Experimental results show that the proposed approach performs favorably against state-of-the-arts in recognizing low-resolution faces with an extremely low memory of 30KB and runs at an ultrafast speed of 1,460 faces per second on CPU or 21,598 faces per second on GPU.

            
@inproceedings{MM19/Ge/FSLR,
    author    = {Shiming Ge and Shengwei Zhao and Xindi Gao and Jia Li},
    title     = {Fewer-Shots and Lower-Resolutions: Towards Ultrafast Face Recognition in the Wild},
    booktitle = {Proceedings of the 27th {ACM} International Conference on Multimedia,
                    {MM} 2019, Nice, France, October 21-25, 2019},
    pages     = {229--237},
    publisher = {{ACM}},
    year      = {2019},
}