Measure test coverage
It would be helpful to know the coverage that our tests provide for the core functionality.
One commonly available toolset for this is the GCC coverage feature for recording of coverage, the gcov tool for profiling, and the lcov tool to generate more human readable summary data.
The process is then:
./build LIBS netlrts-linux-x86_64 smp -j8 --suffix coverage --coverage cd netlrts-linux-x86_64-smp-coverage/test; make -j8 OPTS+="--coverage"; cd ../tmp; make test;
That will create many files of the form sourcename.gcda in the tmp dir.
Repeat the make test set for each machine target one wishes to see, combined into an aggregated result run the test and accumulate the results.
Such as for non-smp, smp, and whichever choices of --with-production or other build time selections make sense to composite.
Coverage will be accumulated into the cwd (machineblah/tmp) for each of those runs.
They can then be processed by visualization and reporting tools of various kinds. The simplest to use is probably lcov.
run lcov to process that data and accumulate it
cd yourAccumulationDirectory; lcov --capture --directory machineblah/tmp --rc lcov_branch_coverage=1 --output-file smp-cov.info lcov --add-tracefile smp-cov.info --output-file total.info
For each of the targets you want aggregated.
When you have accumulated it all, process the aggregate
genhtml total.info --rc lcov_branch_coverage=1 --output-directory youroutputdir
This will generate an html summary that will allow you to browse through the coverage along the directory heirarchy of your source on a file by file basis.
#1 Updated by Anirudh Manoj almost 2 years ago
I've added a commit -
b134360992dc073e510f47cf7b1e47bd749b3161 - which adds a couple of coverage scripts.
coverage script inside the
cd coverage/ ./coverage
ck-convcore-summary.C file takes a tracefile created by lcov and parses out the coverage information about
convcore.c specifically, since those are important files to have coverage for.
coveragescript automates the entire process, using
ck-convcore-summary.Cas well. There are two outputs from the script:
summary.txtholds very brief summary about all files and
convcore.cseparately. For now, there's a bunch of output from
genhtmlat the beginning of the file that we will want to remove later.
cov-htmlfolder which holds the html version of the tracefiles. This is available in the
coveragefolder as soon as the script is run and holds specific data about which lines in the files are covered and which lines aren't covered by
coverage script does not follow the process stated above in the original post.
gcov is very specific in not being thread-safe. There has been support recently, presumably in gcc 6.0+, but on gcc 5.4 and earlier, the last command with
lcov which accumulates the tracefiles will fail. To circumvent this, I have accumulated the tracefiles with
genhtml instead, which will not fail due to the thread-related errors. This will not make the errors go away, but will allow us to get some form of coverage, if not thorough.
multicore-linux-x86_64, has been tested and works on these two. To add builds to the coverage, add the names of the systems to
BUILDS variable at the top of the
coverage script, along with any flags that need to be passed to the
./build command with that build.
The ultimate aim of coverage would be to have the coverage script run along with the nightly build and run all the builds. At the end, display the summary details from the summary.txt and make the html version of the coverage available for viewing.
#4 Updated by Eric Bohm about 1 year ago
- % Done changed from 0 to 90
- Category set to Build & Test Automation
- Estimated time set to 8.00 h
- Tags set to build
https://charm.cs.illinois.edu/gerrit/#/c/charm/+/2844/ has the updated patch
It will build charm with --coverage, run make test on tests and examples, then assemble the results using lcov and provide browsable analysis using gen-html.
By default it checks netlrts-linux-x86_64-smp, but any number of targets can be run by putting them on the command line. The total coverage for ck.C and convcore.c is summarized in summary.txt
The verbose file by file result can be browsed in coverage/netlrts-linux-x86_64-smp-cov-html/tmp/index.html or by platform name from the command line arguments coverage/[platform-name]-cov-html/tmp/index.html
Once this is merged we can automate this to run with daily or weekly frequency via jenkins or cron and provide a link to the top level coverage url in our wiki.
Currently tracked over time here: http://charm.cs.illinois.edu/autobuild/coverage/rrdimages/