1 Coverage Analyser

The module cover provides a set of functions which can be used for coverage analysis of Erlang programs. Coverage analysis consists of monitoring executing programs and observing how often every line of code is executed. Statistics gained from the monitoring process can be used for various purposes. The most important are:

• Finding the most frequently called functions. This is useful for "tuning" code.
• Finding if parts of the code are never executed. This is useful during testing to find out how much of the code that is covered with test cases.

This section describes the Coverage Analyser and shows examples of typical coverage analysis. Refer to the Reference Manual, tools, cover for the complete details about the functions available with this tool.

The module cover will in due time be replaced by the module coast, performing coverage and call statistics analysis. Please see the Reference Manual, tools, coast for the complete details about the functions available with this new tool.

1.1 Functions

The module cover exports the following functions.

Cover functions

Function	Explanation
file(File)	Prepare File for coverage analysis.
file(File, Options)	Prepare File for coverage analysis using Options.
all_files(Dir)	Evaluate cover:file/1 for all .erl files in Dir.
all_files(Dir, Options)	Evaluate cover:file/2 for all .erl files in Dir.
quick_analyse(Module)	Prints a quick coverage analysis for Module.
quick_analyse_as_term(Module)	Performs a quick coverage analysis for Module and returns the result as an Erlang term.
analyse(Module)	Perform detailed coverage analysis of the functions in Module.
check(Module)	Returns true if Module is prepared for coverage analysis.
all_covered()	Returns a list of all modules prepared for coverage analysis.
most_called(Module)	Prints the most called functions in Module.
most_called()	Prints the most called functions for all modules returned by all_covered/0.
zero(Module)	Discards all collected coverage data for Module.
zero()	Discards all collected coverage data.

1.2 Using the Coverage Analyser

Follow these steps to perform coverage analysis of Erlang programs:

1. Make sure that the modules you want to analyse contain no syntax errors, and that they can be run with reasonable test data.
2. Create or change to a directory where the coverage analysis will be run (see example below).
3. Prepare the modules for coverage analysis by evaluating cover:file(File [,Options]) or cover:all_files(Dir[,Options]) (see Note Prepare Modules below this list).
4. Restart the system from the directory where the coverage analysis will be run.
5. Run a set of test queries or test suites.
6. Use the functions most_called/1, quick_analyse/1, analyse/1 to analyse the results of the test run.

Files with extensions .COVER.erl must not be changed or moved after they have been created, or the coverage analysis will not work. They can be deleted when the coverage analysis has been completed.

cover:file(File [,Options]) and cover:all_files(Dir[,Options]) perform a special compilation of the modules to be tested, and produce files of the form module.COVER.erl and module.jam. The first file is a transformed version of the original module, the second is the compiled object code for the transformed module.

The following example illustrates how to perform coverage analysis. The module analysed io_lib, which is part of the standard Erlang distribution.

> cd ~/erl 
> mkdir cover_data 
> cd cover_data 
> erl -nostick
.....
> cover:file(io_lib).
Input file:otp_root/lib/stdlib-1.0/src/io_lib
Output file:io_lib.COVER.erl
Transforming otp_root/lib/stdlib-1.0/src/io_lib.erl for profiling
scan/3 scan/1 scan/2 reserved_word/1 fwrite/2 fread/2 fread/3 format/2 ...
...
Now compiling module io_lib
Module io_lib will now be purged and re-loaded
{module,io_lib}

This run creates the files io_lib.COVER.erl and io_lib.jam in the current directory, which is the directory cover_data in the example shown.

• io_lib.COVER.erl is the transformed version of the original io_lib.erl
• io_lib.jam is the compiled object code for the transformed code. The code in io_lib.jam, when evaluated, will log coverage statistics.

The -nostick option is given to the erl command because io_lib is a library module which resides in a "sticky" directory which means that the system is not normally allowed to load the module from any other place. The option -nostick removes this restriction. The option is used here only because io_lib provides a good example.

A "sticky" directory is a directory which contains system files which cannot be moved or modified.

Do not use the -nostick option unless you are really sure what you are doing.

The next step is to force usage of code in io_lib. In the example shown, the shell internal function i() is evaluated. This forces use of the i/o system. The function cover:most_called is then used to analyse the results.

2> i().
Pid          Initial Call            Current Function         
<0.0.0>      {init,boot,1}           {init,loop,8}           ...
<0.2.0>      {erl_prim_loader,start_ {erl_prim_loader,loop,5}...
<0.4.0>      {gen_event,init_it,6}   {gen_event,loop,4}      ...
<0.5.0>      {application_controller {gen_server,loop,6}     ...
...
3> cover:most_called(io_lib).
       1       3844  69.07  69.07 io_lib:deep_char_list/2 
       2        613  11.02  80.09 io_lib:quote_atom/1 
       3        256   4.60  84.69 io_lib:write/2 
       4        243   4.37  89.06 io_lib:write_atom/1 
       5        243   4.37  93.42 io_lib:quote_atom/2 
       6        128   2.30  95.72 io_lib:deep_char_list/1 
       7        114   2.05  97.77 io_lib:write_tail/2 
       8         99   1.78  99.55 io_lib:write/1 
       9         23   0.41  99.96 io_lib:format/2 
      10          2   0.04 100.00 io_lib:fwrite/2 
...

The function cover:most_called(io_lib) analyses the results and prints out a sorted list of the most called functions in io_lib. Printout from cover:most_called/1 shows the meaning of the numbers printed to the left of the function names.

Printout from cover:most_called/1

For example, io_lib:deep_char_list/2 accounted for 69.07% of all calls, and the first four calls of the list accounted for almost 90% of all calls.

A brief summary of the coverage of the code in io_lib can be obtained by evaluating cover:quick_analyse(io_lib).

4> cover:quick_analyse(io_lib).
COVERAGE ANALYSIS (QUICK) FOR MODULE io_lib
*** 22 PERCENT *** of the statements are covered
NOT COVERED:85 COVERED: 24 TOTAL NUMBER OF STATEMENTS: 109
 
COVERAGE FOR FUNCTIONS IN MODULE io_lib
*** NOT CALLED *** 
collect_chars/3 scan/2 scan/1 write_char/1 print/1 write_string/1
 indentation/2 char_list/1 nl/0 collect_line/2 collect_chars1/3 write_char/3
...
*** 28 PERCENT COVERED *** 
write/2 
*** 43 PERCENT COVERED *** 
quote_atom/1 
*** 50 PERCENT COVERED *** 
write_tail/2 
*** 60 PERCENT COVERED *** 
quote_atom/2 
*** 75 PERCENT COVERED *** 
write_atom/1 
*** 80 PERCENT COVERED *** 
deep_char_list/2 
*** 100 PERCENT COVERED *** 
write/1 deep_char_list/1 format/2 fwrite/2 
 
ok

The functions are sorted in groups according to the amount of code which has been covered. Functions grouped as NOT CALLED did not execute at all. Functions grouped as 100 PERCENT COVERED have executed every line of code at least once. The remainder of the functions are listed in groups which indicate the percentage of code within the function which has been executed and evaluated.

A more detailed analysis can be obtained by running cover:analyse as shown in the following example:

5> cover:analyse(io_lib).
Found Object code in:./io_lib.jam
Object code last modified:{1996,11,19,13,9,37}
Found source file in:io_lib.COVER.erl
Source code last modified:{1996,11,19,13,9,32}
Analysing Source:io_lib.COVER.erl module:io_lib
Output File:io_lib.COVER.erl.out
scan/3 scan/1 scan/2 reserved_word/1 fwrite/2 fread/2 fread/3 format/2 
print/1 print/4 indentation/2 write/1 write/3 write/2 write_tail/2 
...

The evaluation of cover:analyse(io_lib) produces the file io_lib.COVER.erl.out in the current directory. This file contains a detailed listing of the number of times each line of the code in io_lib.erl was called.

The following example shows first few lines in io_lib.COVER.erl.out:

$ more io_lib.COVER.erl.out
scan(Cont,Chars,Pos) ->
    "**** Not Covered *****",
    erl_scan:tokens(Cont,Chars,Pos).
......
fwrite(Format,Args) ->
    "**** 4 ****",
    io_lib_format:fwrite(Format,Args).

fread(Chars,Format) ->
    "**** Not Covered *****",
    io_lib_fread:fread(Chars,Format).

format(Format,Args) ->
    "**** 138 ****",
    io_lib_format:fwrite(Format,Args).
....
    

The listing shows how often each line of code in io_lib.erl was called.

**** Not Covered **** indicates that this line of code was never evaluated. In an extensive test suite, it is desirable to force every line in a module to be evaluated at least once, as this is a valuable way of discovering programming errors.

The following functions are used to clear the coverage data in order to provide a fresh start for the collection of new coverage data.

• cover:zero(Mod) clears the coverage data of the module Mod
• cover:zero() clears the coverage data for all modules

Coverage data is the statistical data accumulated by the coverage analyser as a result of executing code.

The following example illustrates the following interaction:

1. cover:zero() clears all current coverage data
2. additional test cases are run
3. cover:most_called(io_lib) produces a list which shows the most called functions in io-lib for the new test cases.

5> cover:zero().
... do some work ... 
6>  cover:most_called(io_lib).
       1      22272  63.45  63.45 io_lib:deep_char_list/2 
       2       3318   9.45  72.90 io_lib:quote_atom/1 
       3       2278   6.49  79.39 io_lib:write_string1/2 
       4       2154   6.14  85.52 io_lib:write_char/3 
       5       1182   3.37  88.89 io_lib:write_atom/1 
       6       1181   3.36  92.25 io_lib:quote_atom/2 
       7        827   2.36  94.61 io_lib:write/2 
       8        585   1.67  96.28 io_lib:format/2 
       9        422   1.20  97.48 io_lib:write/1 
      ...

Refer to Printout from cover:most_called/1 again for an explanation of this printout.

1.3 Precautions

When files have been compiled for coverage analysis, you must ensure that the correct version of the object code is used for coverage analysis. The correct files are those which have been prepared for analysis with the function cover:file(File).

This situation can be illustrated with the example described in the previous section. Suppose that the Erlang system is stopped and restarted and the current directory is the same as when io_lib was prepared for analysis.

1. Evaluate the function cover:all_covered(), using the current search path of the code server (refer Reference Manual, module code for information about the search path of the code server). This function returns a list of all modules prepared for coverage analysis.
2. The function returns [], which is an empty list.

This result indicates that io_lib is not prepared for coverage analysis. The reason is that the system has been restarted and io_lib is loaded from its standard location, where the original object code file resides. This code has not been prepared for coverage analysis.

It is possible to load the version of io_lib which was prepared for coverage analysis by entering the following command from the shell:

> code:load_file(io_lib).
{module,io_lib}
> cover:all_covered().
[io_lib].