Investigating and improving log parsing in practice

• DOI: 10.1145/3540250.3558947
• Publication Date: 2022-11-09T20:46:22Z
• Authors: Ying Fu; Meng Yan; Jian Xu; Jianguo Li; Zhongxin Liu; Xiaohong Zhang; Dan Yang
• Keywords: 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology

Logs are widely used for system behavior diagnosis by automatic log mining. Log parsing is an important data preprocessing step that converts semi-structured messages into structured as the feature input Currently, many studies devoted to proposing new parsers. However, best of our knowledge, no previous study comprehensively investigates effectiveness parsers in industrial practice. To investigate practice, this paper, we conduct empirical on six state-of-the-art 10 microservice applications Ant Group. Our results highlight two challenges practice: 1) various separators. There separators a message, and different event templates or also various. Current cannot perform well because they do not consider 2) Various lengths due nested objects. The belonging same template may have 6 out at Group with 4 deal In propose improved parser named Drain+ based Drain. includes innovative components address above challenges: statistical-based generation component, which generates automatically message splitting, candidate merging merges similarity method. We evaluate 16 public datasets. show outperforms Finally, conclude observations road ahead inspire other researchers practitioners.