A Unified Causal View of Instruction Tuning

• DOI: 10.48550/arxiv.2402.06220
• Publication Date: 2024-02-09
• Authors: Lu Chen; Wei Huang; Ruqing Zhang; Wei Chen; Jiafeng Guo; Xueqi Cheng

Instruction tuning on a mixture of tasks has improved zero-shot capabilities in natural language processing (NLP). Nevertheless, existing methods often learn features that exhibit correlations between instruction-formatted samples and target labels, rather than causal relationships. Termed as ``spurious correlation'' statistics, such correlation may change drastically new task, making the effect from learned to be misleading. To this end, we develop meta Structural Causal Model (meta-SCM) integrate different NLP under single structure data. Specifically, meta-SCM introduces multiple latent factors represent properties source context, only some which causally influence labels for specific task. The key idea is task-required use those make predictions given Theoretically, prove factor can identified without mixing information others. Guided by identifiability, propose Tuning (SIT) method representations mimic each utility our approach verified improvements ability range unseen datasets tasks.