Towards effective assessment of steady state performance in Java software: are we there yet?

• DOI: 10.1007/s10664-022-10247-x
• Publication Date: 2022-11-28T09:08:54Z
• Authors: Luca Traini; Vittorio Cortellessa; Daniele Di Pompeo; Michele Tucci
• Keywords: Software Engineering (cs.SE); FOS: Computer and information sciences; Computer Science - Software Engineering; Java, JMH, Microbenchmarking, Performance evaluation, Performance testing; 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology

Abstract Microbenchmarking is a widely used form of performance testing in Java software. A microbenchmark repeatedly executes small chunk code while collecting measurements related to its performance. Due Virtual Machine optimizations, microbenchmarks are usually subject severe fluctuations the first phase their execution (also known as warmup). For this reason, software developers typically discard and focus analysis when benchmarks reach steady state Developers estimate end warmup based on expertise, configure accordingly. Unfortunately, approach two strong assumptions: (i) always (ii) accurately warmup. In paper, we show that do not state, often fail phase. We found considerable portion studied hit estimates provided by inaccurate (with large error). This has significant implications both terms results quality time-effort. Furthermore, dynamic reconfiguration significantly improves estimation accuracy, but still it induces suboptimal relevant side-effects. envision paper starting point for supporting introduction more sophisticated automated techniques can ensure timely fashion.