Can Videos as a By-Product of GUI Testing Help Developers Understand GUI Tests?

Abstract

[Context] Software with graphical user interfaces (GUIs) is used in everyday life. Users expect working software when they interact with a software product. For this reason, testing and defect corrections are required. However, understanding GUI tests is challenging, as the generated textual test reports lack the dynamic details about interaction steps to reproduce a GUI defect. [Objective] We want to revise the practice of GUI testing and use videos as a by-product to capture and convey these dynamic details. [Method] Based on the video as a by-product approach, we propose to generate videos as a by-product of GUI testing, integrating annotations and test outputs directly into the videos. These videos serve as supplementary material for documenting dynamic test executions in addition to static test reports (e.g., texts, screenshots). In an experiment, we asked 11 participants to distinguish between true and false defects in watching videos and screenshots of four test cases. We also conducted structured interviews to ask the participants about their opinions for these two documentation options. [Results & Conclusion] Our results show visible differences between the video and the screenshots in effectiveness and efficiency in defect analysis, but cannot prove the differences statistically. The listed advantages and disadvantages of both documentations are complemental. Both documentations are helpful and thus videos as a by-product of GUI testing can foster the understanding of GUI tests.© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.