TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise

• DOI: 10.48550/arxiv.2310.19019
• Publication Date: 2023-01-01
• Authors: Nan He; Hanyu Lai; Chenyang Zhao; Zirui Cheng; Junting Pan; Ruoyu Qin; Ruofan Lu; Lu Rui; Yuncheng Zhang; Gangming Zhao; Zhaohui Hou; Zhiyuan Huang; Shaoqing Lu; Liang Ding; Mingjie Zhan
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computation and Language (cs.CL)

Large Language Models (LLMs) exhibit impressive reasoning and data augmentation capabilities in various NLP tasks. However, what about small models? In this work, we propose TeacherLM-7.1B, capable of annotating relevant fundamentals, chain thought, common mistakes for most samples, which makes annotation more than just an answer, thus allowing other models to learn "why" instead "what". The TeacherLM-7.1B model achieved a zero-shot score 52.3 on MMLU, surpassing with over 100B parameters. Even remarkable is its ability. Based augmented 58 datasets taught student different parameters from OPT BLOOM series multi-task setting. experimental results indicate that the provided by TeacherLM has brought significant benefits. We will release as open-source.