Project

General

Profile

Support #1512

Measure test coverage

Added by Eric Bohm 4 months ago. Updated 23 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
04/18/2017
Due date:
% Done:

0%


Description

It would be helpful to know the coverage that our tests provide for the core functionality.

One commonly available toolset for this is the GCC coverage feature for recording of coverage, the gcov tool for profiling, and the lcov tool to generate more human readable summary data.

The process is then:

./build LIBS netlrts-linux-x86_64 smp -j8 --suffix coverage --coverage
cd netlrts-linux-x86_64-smp-coverage/test; make -j8 OPTS+="--coverage";
cd ../tmp; make test;


That will create many files of the form sourcename.gcda in the tmp dir.

Repeat the make test set for each machine target one wishes to see, combined into an aggregated result run the test and accumulate the results.
Such as for non-smp, smp, and whichever choices of --with-production or other build time selections make sense to composite.

Coverage will be accumulated into the cwd (machineblah/tmp) for each of those runs.

They can then be processed by visualization and reporting tools of various kinds. The simplest to use is probably lcov.

run lcov to process that data and accumulate it

cd yourAccumulationDirectory;
lcov --capture --directory machineblah/tmp --rc lcov_branch_coverage=1 --output-file smp-cov.info
lcov --add-tracefile smp-cov.info --output-file total.info

For each of the targets you want aggregated.

When you have accumulated it all, process the aggregate

genhtml total.info --rc lcov_branch_coverage=1 --output-directory youroutputdir

This will generate an html summary that will allow you to browse through the coverage along the directory heirarchy of your source on a file by file basis.

History

#1 Updated by Anirudh Manoj 23 days ago

I've added a commit - b134360992dc073e510f47cf7b1e47bd749b3161 - which adds a couple of coverage scripts.

Run the coverage script inside the coverage folder:

cd coverage/
./coverage

The ck-convcore-summary.C file takes a tracefile created by lcov and parses out the coverage information about ck.C and convcore.c specifically, since those are important files to have coverage for.

The coverage script automates the entire process, using ck-convcore-summary.C as well. There are two outputs from the script:
  1. summary.txt holds very brief summary about all files and ck.C and convcore.c separately. For now, there's a bunch of output from genhtml at the beginning of the file that we will want to remove later.
  2. The cov-html folder which holds the html version of the tracefiles. This is available in the coverage folder as soon as the script is run and holds specific data about which lines in the files are covered and which lines aren't covered by tests and examples.

The coverage script does not follow the process stated above in the original post. gcov is very specific in not being thread-safe. There has been support recently, presumably in gcc 6.0+, but on gcc 5.4 and earlier, the last command with lcov which accumulates the tracefiles will fail. To circumvent this, I have accumulated the tracefiles with genhtml instead, which will not fail due to the thread-related errors. This will not make the errors go away, but will allow us to get some form of coverage, if not thorough.

For now, coverage includes netlrts-linux-x86_64 and multicore-linux-x86_64, has been tested and works on these two. To add builds to the coverage, add the names of the systems to BUILDS variable at the top of the coverage script, along with any flags that need to be passed to the ./build command with that build.

The ultimate aim of coverage would be to have the coverage script run along with the nightly build and run all the builds. At the end, display the summary details from the summary.txt and make the html version of the coverage available for viewing.

Also available in: Atom PDF