In crowdsourced mobile application testing, workers are often inexperienced in and unfamiliar with software testing. Meanwhile, workers edit test reports in descriptive natural language on mobile devices. Thus, these test reports generally lack important details and challenge developers in understanding the bugs. To improve the quality of inspected test reports, we issue a new problem of test report augmentation by leveraging the additional useful information contained in duplicate test reports. In this paper, we propose a new framework named test report augmentation framework (TRAF) towards resolving the problem. First, natural language processing (NLP) techniques are adopted to preprocess the crowdsourced test reports. Then, three strategies are proposed to augment the environments, inputs, and descriptions of the inspected test reports, respectively. Finally, we visualize the augmented test reports to help developers distinguish the added information. To evaluate TRAF, we conduct experiments over five industrial datasets with 757 crowdsourced test reports. Experimental results show that TRAF can recommend relevant inputs to augment the inspected test reports with 98.49% in terms of NDCG and 88.65% in terms of precision on average, and identify valuable sentences from the descriptions of duplicates to augment the inspected test reports with 83.58% in terms of precision, 77.76% in terms of recall, and 78.72% in terms of F-measure on average. Meanwhile, empirical evaluation also demonstrates that augmented test reports can help developers understand and fix bugs better.
Frontiers of Computer Science
This work was partially supported by the National Natural Science Foundation of China (Grant Nos. 61370144, 61722202, 61403057, and 61772107)
Jiangsu Prospective Project of Industry- University-Research (BY2015069-03)
Besides, the authors would thank the three graduate students who devote their efforts for the data annotation.
Xin CHEN received the PhD degree in software engineering in 2018 from the School o f Software, Dalian University o f Technology, China. He is currently a lecturer of Hangzhou Dianzi University, China. His research interests include mining software repositories and evolutionary computation. He is a member of the CCF and the ACM;He Jiang received the PhD degree from University of Science and Technology of China. He is currently a professor and PhD supervisor, at the School of Software, Dalian University of Technology, China. He has published prolifically in refereed journals and conference proceedings, e.g., TKDE, TSE, TSC, TOIT, TSMCB, TCYB, ICSE, and SANER. Prof. Jiang is a member of the CCF and the ACM. His current research interests include search based software engineering, mining software repositories, and evolutionary computation.E-mail:email@example.com;Zhenyu Chen is a professor at Software Institute, Nanjing University, China. He got his BS and PhD degrees in mathematics from Nanjing University, China in 2001 and 2006, respectively. His research interests include intelligent software engineering and mining software repositories. He is a member of the CCF and the ACM;Tieke He is currently a research assistant at Software Institute, Nanjing University, China. He got his BS, MS, and PhD degrees in software engineering from Nanjing University, China in 2010, 2012 and 2017, respectively. His research interests include recommender systems and knowledge graph;Liming Nie received the PhD degree in computer application technology from Dalian University of Technology, China in 2017. He is currently a lecturer with Zhejiang Sci-Tech University, China. His current research interests include intelligent software development and its application. Dr. Nie is a member of the ACM and the CCF.