Text

Test Rsult Database

Photo: https://unsplash.com/

There is a growing body of knowledge in the computer science, software engineering, software testing and software test automation disciplines. However, there is a challenge for researchers to evaluate their research findings, innovations and tools due to lack of realistic data. This paper presents the Westermo test results data set, more than one million verdicts from testing of embedded systems, from more than five hundred consecutive days of nightly testing. The data also contains information on code changes in both the software under test and the test framework used for testing. This data set can support the research community in particular with respect to the regression test selection problem, flaky tests, test results visualization, etc.

Valuce of the data

This data can be useful for researchers working on test automation, e.g. we have used it to support work on regression test selection (Felding, 2022), (Strandberg et al. 2016, & 2017), flaky tests (Strandberg et al., 2020), test results visualization (Strandberg et al., 2022); and for exploring metrics in performance of testing (Abed Jaser and Al-Braichi, 2019). Furthermore, the data has been used in hackathons in the AIDOaRt research project, (Eramo et al., 2021), where also test case dependencies and anomaly detection of test automation performance was explored. This data was central in Strandberg's doctoral thesis (Strandberg, 2021). In short, students, researchers and practitioners in the field of software test automation, graphic design or artificial intelligence can benefit from the data. It can be used to evaluate algorithms, tools or visualizations for improved validity and generalizability.

Specifications

  • Subject: Computer Science - Embedded Systems, and Software Engineering
  • Specific subject area: Nightly testing of software in networked embedded systems.
  • Type of data: Table
  • How the data were acquired: Test results data were acquired using Westermo's test automation framework in the Westermo test software test lab. Data from source code changes in the software under test and in the test framework was stored in the git source code management system. See related research article for more details.
  • Data format: Raw
  • Description of data collection: Test results data were exported from the Westermo test results database, and anonymized. Data from source code changes were exported from Westermo's source code repository, and anonymized.
  • Data source location: Data was collected at Westermo Network Technologies AB, on Metallverksgatan, in Västerås, Sweden.
  • Data accessibility: Data is available at GitHub: https://github.com/westermo External link, opens in new window.
  • Related research article: P. E. Strandberg, Automated System-Level Software Testing of Industrial Networked Embedded Systems, PhD Thesis, Mälardalen University, 2021. ISBN: 978-91-7485-529-6. Archived in arXiv: https://arxiv.org/abs/2111.08312 External link, opens in new window. as well as in the Swedish National Library http://urn.kb.se/resolve?urn=urn:nbn:se:mdh:diva-56036. External link, opens in new window.

Repository Link: https://github.com/westermo/test-results-dataset External link, opens in new window.

Main Contact: per.strandberg@westermo.se

License: This data set is licensed with the Creative Commons Attribution 4.0 International External link..

In short, you are free to: share, copy and redistribute the material; and to adapt, remix, transform, and build upon it for any purpose; under the condition that you you give appropriate credit, and do not restrict others from doing anything the license permits. Read the license for details.

Suggested attribution: Strandberg, P E. (2022). The Westermo test results data set. Retrieved from https://github.com/westermo External link.