Can Machine Learning Support the Selection of Studies for Systematic Literature Review Updates?

• DOI: 10.48550/arxiv.2502.08050
• Publication Date: 2025-02-11
• Authors: Marcelo Costalonga; Bianca Minetto Na...; María Teresa Bald...; Katia Romero Feli...; Igor Steinmacher; Marcos Kalinowski
• Keywords: Software Engineering (cs.SE); FOS: Computer and information sciences; Computer Science - Software Engineering

[Background] Systematic literature reviews (SLRs) are essential for synthesizing evidence in Software Engineering (SE), but keeping them up-to-date requires substantial effort. Study selection, one of the most labor-intensive steps, involves reviewing numerous studies and multiple reviewers to minimize bias avoid loss evidence. [Objective] This study aims evaluate if Machine Learning (ML) text classification models can support selection SLR updates. [Method] We reproduce an update performed by three SE researchers. trained two supervised ML (Random Forest Support Vector Machines) with different configurations using data from original SLR. calculated effectiveness terms precision, recall, F-measure. also compared performance human-ML pairs human-only when selecting studies. [Results] The achieved a modest F-score 0.33, which is insufficient reliable automation. However, we found that such reduce effort 33.9% without (keeping 100% recall). Our analysis showed initial screening human produces results much better aligned final result. [Conclusion] Based on our results, conclude although help involved updates, achieving rigorous outcomes still expertise experienced phase.