Support #1512

Measure test coverage

Added by Eric Bohm over 2 years ago. Updated 11 months ago.

Build & Test Automation
Target version:
Start date:
Due date:
% Done:


Estimated time:
8.00 h
Spent time:


It would be helpful to know the coverage that our tests provide for the core functionality.

One commonly available toolset for this is the GCC coverage feature for recording of coverage, the gcov tool for profiling, and the lcov tool to generate more human readable summary data.

The process is then:

./build LIBS netlrts-linux-x86_64 smp -j8 --suffix coverage --coverage
cd netlrts-linux-x86_64-smp-coverage/test; make -j8 OPTS+="--coverage";
cd ../tmp; make test;

That will create many files of the form sourcename.gcda in the tmp dir.

Repeat the make test set for each machine target one wishes to see, combined into an aggregated result run the test and accumulate the results.
Such as for non-smp, smp, and whichever choices of --with-production or other build time selections make sense to composite.

Coverage will be accumulated into the cwd (machineblah/tmp) for each of those runs.

They can then be processed by visualization and reporting tools of various kinds. The simplest to use is probably lcov.

run lcov to process that data and accumulate it

cd yourAccumulationDirectory;
lcov --capture --directory machineblah/tmp --rc lcov_branch_coverage=1 --output-file
lcov --add-tracefile --output-file

For each of the targets you want aggregated.

When you have accumulated it all, process the aggregate

genhtml --rc lcov_branch_coverage=1 --output-directory youroutputdir

This will generate an html summary that will allow you to browse through the coverage along the directory heirarchy of your source on a file by file basis.


#1 Updated by Anirudh Manoj almost 2 years ago

I've added a commit - b134360992dc073e510f47cf7b1e47bd749b3161 - which adds a couple of coverage scripts.

Run the coverage script inside the coverage folder:

cd coverage/

The ck-convcore-summary.C file takes a tracefile created by lcov and parses out the coverage information about ck.C and convcore.c specifically, since those are important files to have coverage for.

The coverage script automates the entire process, using ck-convcore-summary.C as well. There are two outputs from the script:
  1. summary.txt holds very brief summary about all files and ck.C and convcore.c separately. For now, there's a bunch of output from genhtml at the beginning of the file that we will want to remove later.
  2. The cov-html folder which holds the html version of the tracefiles. This is available in the coverage folder as soon as the script is run and holds specific data about which lines in the files are covered and which lines aren't covered by tests and examples.

The coverage script does not follow the process stated above in the original post. gcov is very specific in not being thread-safe. There has been support recently, presumably in gcc 6.0+, but on gcc 5.4 and earlier, the last command with lcov which accumulates the tracefiles will fail. To circumvent this, I have accumulated the tracefiles with genhtml instead, which will not fail due to the thread-related errors. This will not make the errors go away, but will allow us to get some form of coverage, if not thorough.

For now, coverage includes netlrts-linux-x86_64 and multicore-linux-x86_64, has been tested and works on these two. To add builds to the coverage, add the names of the systems to BUILDS variable at the top of the coverage script, along with any flags that need to be passed to the ./build command with that build.

The ultimate aim of coverage would be to have the coverage script run along with the nightly build and run all the builds. At the end, display the summary details from the summary.txt and make the html version of the coverage available for viewing.

#2 Updated by Eric Bohm about 1 year ago

  • Assignee set to Eric Bohm

#3 Updated by Eric Bohm about 1 year ago

  • Status changed from New to In Progress

#4 Updated by Eric Bohm about 1 year ago

  • % Done changed from 0 to 90
  • Category set to Build & Test Automation
  • Estimated time set to 8.00 h
  • Tags set to build has the updated patch

It will build charm with --coverage, run make test on tests and examples, then assemble the results using lcov and provide browsable analysis using gen-html.

By default it checks netlrts-linux-x86_64-smp, but any number of targets can be run by putting them on the command line. The total coverage for ck.C and convcore.c is summarized in summary.txt

The verbose file by file result can be browsed in coverage/netlrts-linux-x86_64-smp-cov-html/tmp/index.html or by platform name from the command line arguments coverage/[platform-name]-cov-html/tmp/index.html

Once this is merged we can automate this to run with daily or weekly frequency via jenkins or cron and provide a link to the top level coverage url in our wiki.

#5 Updated by Eric Bohm about 1 year ago

  • Status changed from In Progress to Resolved

#6 Updated by Eric Bohm 11 months ago

Also available in: Atom PDF