Measure test coverage
It would be helpful to know the coverage that our tests provide for the core functionality.
One commonly available toolset for this is the GCC coverage feature for recording of coverage, the gcov tool for profiling, and the lcov tool to generate more human readable summary data.
The process is then:
./build LIBS netlrts-linux-x86_64 smp -j8 --suffix coverage --coverage cd netlrts-linux-x86_64-smp-coverage/test; make -j8 OPTS+="--coverage"; cd ../tmp; make test;
That will create many files of the form sourcename.gcda in the tmp dir.
Repeat the make test set for each machine target one wishes to see, combined into an aggregated result run the test and accumulate the results.
Such as for non-smp, smp, and whichever choices of --with-production or other build time selections make sense to composite.
Coverage will be accumulated into the cwd (machineblah/tmp) for each of those runs.
They can then be processed by visualization and reporting tools of various kinds. The simplest to use is probably lcov.
run lcov to process that data and accumulate it
cd yourAccumulationDirectory; lcov --capture --directory machineblah/tmp --rc lcov_branch_coverage=1 --output-file smp-cov.info lcov --add-tracefile smp-cov.info --output-file total.info
For each of the targets you want aggregated.
When you have accumulated it all, process the aggregate
genhtml total.info --rc lcov_branch_coverage=1 --output-directory youroutputdir
This will generate an html summary that will allow you to browse through the coverage along the directory heirarchy of your source on a file by file basis.
#1 Updated by Anirudh Manoj 23 days ago
I've added a commit -
b134360992dc073e510f47cf7b1e47bd749b3161 - which adds a couple of coverage scripts.
coverage script inside the
cd coverage/ ./coverage
ck-convcore-summary.C file takes a tracefile created by lcov and parses out the coverage information about
convcore.c specifically, since those are important files to have coverage for.
coveragescript automates the entire process, using
ck-convcore-summary.Cas well. There are two outputs from the script:
summary.txtholds very brief summary about all files and
convcore.cseparately. For now, there's a bunch of output from
genhtmlat the beginning of the file that we will want to remove later.
cov-htmlfolder which holds the html version of the tracefiles. This is available in the
coveragefolder as soon as the script is run and holds specific data about which lines in the files are covered and which lines aren't covered by
coverage script does not follow the process stated above in the original post.
gcov is very specific in not being thread-safe. There has been support recently, presumably in gcc 6.0+, but on gcc 5.4 and earlier, the last command with
lcov which accumulates the tracefiles will fail. To circumvent this, I have accumulated the tracefiles with
genhtml instead, which will not fail due to the thread-related errors. This will not make the errors go away, but will allow us to get some form of coverage, if not thorough.
multicore-linux-x86_64, has been tested and works on these two. To add builds to the coverage, add the names of the systems to
BUILDS variable at the top of the
coverage script, along with any flags that need to be passed to the
./build command with that build.
The ultimate aim of coverage would be to have the coverage script run along with the nightly build and run all the builds. At the end, display the summary details from the summary.txt and make the html version of the coverage available for viewing.