Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems

• DOI: 10.48550/arxiv.2502.04510
• Publication Date: 2025-02-06
• Authors: Shangbin Feng; Zifeng Wang; Palash Goyal; Yike Wang; Weijia Shi; Huang Xia; Hamid Palangi; Luke Zettlemoyer; Yulia Tsvetkov; Chen‐Yu Lee; Tomas Pfister
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Computation and Language (cs.CL)

We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights. represent as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool LLM experts utility function, Swarms employs two iterative steps: role-step weight-step. For role-step, we interpret learning DAG that specifies the flow inputs outputs between LLMs. Starting from swarm random continuous adjacency matrices, decode them into discrete DAGs, call in order, evaluate on function (e.g. accuracy task), optimize matrices particle optimization based score. weight-step, assess contribution individual weights intelligence. JFK-score quantify each best-found then JFK-score. Experiments demonstrate outperforms 15 role- and/or weight-based baselines 18.5% average across 12 tasks. Further analysis reveals discovers heterogeneous substantial gains, benefits diversity language models.