Data not found

Are the code snippets what we are searching for? A benchmark and an empirical study on code search with natural-language queries

Country : Singapore
Department : Singapore Management University
Project Title : Are the code snippets what we are searching for? A benchmark and an empirical study on code search with natural-language queries
Researcher : YAN, Shuhan , YU, Hang , CHEN, Yuting , SHEN, Beijun
Keyword : natural-language code search , benchmarking , empirical study , information retrieval , machine learning , deep learning , word embedding , Software Engineering
Publisher : Institutional Knowledge at Singapore Management University
Year End : 2020
Identifier : https://ink.library.smu.edu.sg/sis_research/5975 , https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=6978&context=sis_research
Source : Research Collection School Of Computing and Information Systems
Abstract / Description :

Code search methods, especially those that allow programmers to raise queries in a natural language, plays an important role in software development. It helps to improve programmers' productivity by returning sample code snippets from the Internet and/or source-code repositories for their natural-language queries. Meanwhile, there are many code search methods in the literature that support natural-language queries. Difficulties exist in recognizing the strengths and weaknesses of each method and choosing the right one for different usage scenarios, because (1) the implementations of those methods and the datasets for evaluating them are usually not publicly available, and (2) some methods leverage different training datasets or auxiliary data sources and thus their effectiveness cannot be fairly measured and may be negatively affected in practical uses. To build a common ground for measuring code search methods, this paper builds CosBench, a dataset that consists of 1000 projects, 52 code-independent natural-language queries with ground truths, and a set of scripts for calculating four metrics on code research results. We have evaluated four IR (Information Retrieval)-based and two DL (Deep Learning)-based code search methods on CosBench. The empirical evaluation results clearly show the usefulness of the CosBench dataset and various strengths of each code search method. We found that DL-based methods are more suitable for queries on reusing code, and IR-based ones for queries on resolving bugs and learning API uses.

References

YAN, Shuhan and others / et al. (2020). Are the code snippets what we are searching for? A benchmark and an empirical study on code search with natural-language queries.  Singapore: Singapore Management University.
YAN, Shuhan and others / et al. 2020. "Are the code snippets what we are searching for? A benchmark and an empirical study on code search with natural-language queries".  Singapore: Singapore Management University.
YAN, Shuhan and others / et al. "Are the code snippets what we are searching for? A benchmark and an empirical study on code search with natural-language queries."  Singapore: Singapore Management University, 2020. Print.
YAN, Shuhan and others / et al. Are the code snippets what we are searching for? A benchmark and an empirical study on code search with natural-language queries. Singapore: Singapore Management University; 2020.

Export

Share