Clone Digger - discovers duplicate code in Python. Sangharsh Agarwal
What is Clone Digger? • Clone Digger aimed to detect similar code in Python and Java programs. The synonyms for the term "similar code" are "clone" and "duplicate code". • What is software clone? Two continuous fragments of code form clone if they are similar enough. • Why is it important to detect clones? The presence of clones can increase maintenance cost of the code. Detected clones can be refactored or just kept in mind.
Why is it important to detect code clones? • 5% - 20% of code in software systems are clones. • Why do programmers produce clones? – Development strategy – Maintenance benefits – Overcoming underlying limitations – Cloning by accident • Why is the presence of code clones bad? – Errors in the original must be fixed in every clone.
Why to use Clone Digger? • Why should I use Clone Digger to detect clones? There are several clone detection tools, they are listed here. The benefits of Clone Digger are: 1. Variety of handled clone types. Strictly speaking, a pair of sequences of statements is considered a clone if one sequence can be obtained from the other by replacing some small sub expressions. Particularly, changes of variable and function names and constants are allowed. 2. It's free (provided under the GPL license).
Definition of clone • Different clone definitions can be classified according to the level of granularity: – List of strings – Sequence of tokens – Abstract syntax trees (AST) – Semantic information • Clone Digger work on the AST level • Clone Digger consider two sequences of statements as a clone if one of them can be obtained from the other by replacing some sub trees.
Quick Start • $ easy_install clonedigger • $ clonedigger –o output.html <source_path> • $ firefox output.html • Additional parameters such as thresholds can be also set (use --help to know more)
Clone Digger – Run from command prompt -1 (pytests/xdcr) [root@cen-1413 testrunner]# sudo /root/.local/bin/clonedigger -o output.html pytests/xdcr/ Parsing pytests/xdcr/upgradeXDCR.py ... done Parsing pytests/xdcr/xdcrbasetests.py ... done Parsing pytests/xdcr/biXDCR.py ... done Parsing pytests/xdcr/esXDCR.py ... done Parsing pytests/xdcr/esbasetests.py ... done Parsing pytests/xdcr/uniXDCR.py ... done Parsing pytests/xdcr/checkpointXDCR.py ... done Parsing pytests/xdcr/observexdcrtest.py ... done Parsing pytests/xdcr/__init__.py ... done Parsing pytests/xdcr/xdcrMiscTests.py ... done Parsing pytests/xdcr/rebalanceXDCR.py ... done Parsing pytests/xdcr/pauseResumeXDCR.py ... done 376 sequences average sequence length: 5.752660 maximum sequence length: 69 Number of statements: 2163 Calculating size for each statement... done Building statement hash... done Number of different hash values: 284 Building patterns... 1000, 2000, 601 patterns were discovered Choosing pattern for each statement... 1000, 2000, done Finding similar sequences of statements... ----------------------------------------- Warning: sequence of statements starting at pytests/xdcr/xdcrbasetests.py:33 consists of many similar statements. It will be ignored. Use --force to override this restriction. Please refer to http://clonedigger.sourceforge.net/documentation.html ----------------------------------------- 369 sequences were found Refining candidates... 167 clones were found Removing dominated clones... -36 clones were removed
Plugins for Clone Digger • Integration with Hudson See a short manual by Pavol Zibrita to learn how to get Clone Digger and Hudson (continuous integration tool) to work together. • Eclipse Plugin Eclipse Plugin for Clone Digger has been implemented during Google Summer of Code 2008. See the manual to learn how install it and how to work with it.
What to do with found clones? • Remove clones by refactoring. Extract method and Pull Up method can be used. • Detect library candidates. • Search for bugs.
References • http://clonedigger.sourceforge.net/index.html • http://clonedigger.sourceforge.net/download.html • http://wiki.hudson-ci. org/display/HUDSON/Meet+Hudson

Clonedigger-Python

  • 1.
    Clone Digger -discovers duplicate code in Python. Sangharsh Agarwal
  • 2.
    What is CloneDigger? • Clone Digger aimed to detect similar code in Python and Java programs. The synonyms for the term "similar code" are "clone" and "duplicate code". • What is software clone? Two continuous fragments of code form clone if they are similar enough. • Why is it important to detect clones? The presence of clones can increase maintenance cost of the code. Detected clones can be refactored or just kept in mind.
  • 3.
    Why is itimportant to detect code clones? • 5% - 20% of code in software systems are clones. • Why do programmers produce clones? – Development strategy – Maintenance benefits – Overcoming underlying limitations – Cloning by accident • Why is the presence of code clones bad? – Errors in the original must be fixed in every clone.
  • 4.
    Why to useClone Digger? • Why should I use Clone Digger to detect clones? There are several clone detection tools, they are listed here. The benefits of Clone Digger are: 1. Variety of handled clone types. Strictly speaking, a pair of sequences of statements is considered a clone if one sequence can be obtained from the other by replacing some small sub expressions. Particularly, changes of variable and function names and constants are allowed. 2. It's free (provided under the GPL license).
  • 5.
    Definition of clone • Different clone definitions can be classified according to the level of granularity: – List of strings – Sequence of tokens – Abstract syntax trees (AST) – Semantic information • Clone Digger work on the AST level • Clone Digger consider two sequences of statements as a clone if one of them can be obtained from the other by replacing some sub trees.
  • 6.
    Quick Start •$ easy_install clonedigger • $ clonedigger –o output.html <source_path> • $ firefox output.html • Additional parameters such as thresholds can be also set (use --help to know more)
  • 7.
    Clone Digger –Run from command prompt -1 (pytests/xdcr) [root@cen-1413 testrunner]# sudo /root/.local/bin/clonedigger -o output.html pytests/xdcr/ Parsing pytests/xdcr/upgradeXDCR.py ... done Parsing pytests/xdcr/xdcrbasetests.py ... done Parsing pytests/xdcr/biXDCR.py ... done Parsing pytests/xdcr/esXDCR.py ... done Parsing pytests/xdcr/esbasetests.py ... done Parsing pytests/xdcr/uniXDCR.py ... done Parsing pytests/xdcr/checkpointXDCR.py ... done Parsing pytests/xdcr/observexdcrtest.py ... done Parsing pytests/xdcr/__init__.py ... done Parsing pytests/xdcr/xdcrMiscTests.py ... done Parsing pytests/xdcr/rebalanceXDCR.py ... done Parsing pytests/xdcr/pauseResumeXDCR.py ... done 376 sequences average sequence length: 5.752660 maximum sequence length: 69 Number of statements: 2163 Calculating size for each statement... done Building statement hash... done Number of different hash values: 284 Building patterns... 1000, 2000, 601 patterns were discovered Choosing pattern for each statement... 1000, 2000, done Finding similar sequences of statements... ----------------------------------------- Warning: sequence of statements starting at pytests/xdcr/xdcrbasetests.py:33 consists of many similar statements. It will be ignored. Use --force to override this restriction. Please refer to http://clonedigger.sourceforge.net/documentation.html ----------------------------------------- 369 sequences were found Refining candidates... 167 clones were found Removing dominated clones... -36 clones were removed
  • 8.
    Plugins for CloneDigger • Integration with Hudson See a short manual by Pavol Zibrita to learn how to get Clone Digger and Hudson (continuous integration tool) to work together. • Eclipse Plugin Eclipse Plugin for Clone Digger has been implemented during Google Summer of Code 2008. See the manual to learn how install it and how to work with it.
  • 9.
    What to dowith found clones? • Remove clones by refactoring. Extract method and Pull Up method can be used. • Detect library candidates. • Search for bugs.
  • 10.
    References • http://clonedigger.sourceforge.net/index.html • http://clonedigger.sourceforge.net/download.html • http://wiki.hudson-ci. org/display/HUDSON/Meet+Hudson