Study Reviews 244 Papers to Assess the State of Modern Code Reviews

:::info Authors:

1. DEEPIKA BADAMPUDI
2. MICHAEL UNTERKALMSTEINER
3. RICARDO BRITTO

:::

Table Of Links

1 INTRODUCTION

2 BACKGROUND AND RELATED WORK

3 RESEARCH DESIGN

4 MAPPING STUDY RESULTS

5 SURVEY RESULTS

6 COMPARING THE STATE-OF-THE-ART AND THE PRACTITIONERS’ PERCEPTIONS

7 DISCUSSION

8 CONCLUSIONS AND ACKNOWLEDGMENTS

REFERENCES

\

1 INTRODUCTION

Software code review is the practice that involves the inspection of code before its integration into the code base and deployment. Software code reviews have evolved from being rigorous, co-located, and synchronous to lightweight, distributed, tool-based and asynchronous [34]. Modern Code Review (MCR) is a lightweight alternative to traditional code inspections [20], which focuses on code changes and allows software developers to improve code quality and reduce post-delivery defects [3, 7]. MCR is an essential practice in modern software development not only due to its contribution to quality assurance; it also helps with design improvement, knowledge sharing, and code ownership.

\ The research interest on code inspections declined in the middle of the 2000’s [25]. Due to the value of code reviews in general, it is reasonable to assume that the research focus has shifted to MCR. After over a decade of research on MCR, several initiatives were born to aggregate a body of knowledge on the increasing research of this essential quality assurance practice. To the best of our knowledge, we presented in our previous work [4] the first overview on the stateof-art of MCR research. In our previous mapping study, we reported the preliminary results of systematically searching and analyzing the existing literature (based on titles and abstracts) and identified major research themes.

\ Likely in parallel, other studies have also explored and made an attempt to aggregate the existing literature on MCR, either on particular aspects of the practice (refactoring-aware code reviews [16], benefits of MCR [30], MCR in education [22], reviewer recommendations [14]) or in general [18, 37]. Since there exists a considerable and diverse amount of research on the MCR practice, we were curious whether the research community has targeted themes that are also perceived as important by MCR practitioners. Similar investigations have been conducted in the past on software engineering research in general [13, 27] and requirements engineering research in particular [21].

\ The main goal of this study is therefore to provide an overview of the different research themes on MCR, analyze practitioners’ opinions on the importance of the research themes, and outline a roadmap for future research on MCR. To achieve this goal, we extended our earlier work [4] by including publications up until the year 2021 and synthesizing the contributions of the 244 identified primary studies in MCR research. Then we constructed 47 statements that describe the research covered in the primary studies and surveyed 28 practitioners using the Q-Methodology [41] to gauge their perception on the statements representing the research conducted in this field.

\ Finally, we compare the practitioners perception on the investigated themes in MCR research with the amount of publications and research impact of those themes. The main contributions of this paper are as follows:

• A comprehensive aggregation of research conducted on MCR research themes until and including 2021 – We identify potential gaps that researchers could address in the future and provide a summary on the state-of-the-art in MCR research that can be useful for practitioners (e.g., to benefit from existing findings and solutions).

\ • Level of alignment between MCR state-of-the-art and practitioners’ perception on the relevance of the MCR state-of-the-art – We assess the practitioners’ perception on the relevance of the MCR state-of-the-art represented by statements that summarize each topic in the MCR state-of-the-art. We assess the alignment between what the research community has focused on the most and how MCR practitioners perceive its relevance. This analysis can help researchers to focus on themes that are deemed relevant by practitioners but do not have enough research coverage. We propose a research roadmap based mainly on the analysis of the reviewed primary studies, and qualified by the responses from the survey.

\ The remainder of this paper is structured as follows: Section 2 presents background on the MCR practice and relevant related work to this study. Section 3 describes the design of our research, which is followed by Sections 4 and 5, where we describe the mapping study and survey results, respectively. In Section 6, we compare the state-of-the-art and practitioners’ perspectives. Section 7 discusses our results and illustrates our MCR research roadmap. Finally, Section 8 presents our conclusions and view on future work.

:::info This paper is available on arxiv under CC BY-NC-SA 4.0 license.

:::

\