Project

General

Profile

Cleanup #595

Cut long-running tests & examples out of 'make test'

Added by Phil Miller over 4 years ago. Updated almost 2 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
PPL
Category:
Build & Test Automation
Target version:
-
Start date:
10/28/2014
Due date:
% Done:

70%


Description

Identify tests/examples that take substantial time to run in the current make test. Either find parameters that make them run faster, or remove them from make test if they don't exercise features not covered by other tests. If necessary, add a separate make perftest target that runs them with parameters that would generate meaningful benchmark numbers.

Screen Shot 2014-11-03 at 11.30.59 PM.png View (71.8 KB) Xiang Ni, 11/03/2014 11:31 PM

History

#1 Updated by Xiang Ni over 4 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 50

Running all the test only takes 80s. Here is the time break down for each directory:
AMPI 8
Charm++ 37
Converse 34
FEM 0.2
Util 0.02

However, there are tests in the test directory but never be run by auto build. Some of them even crash. Attached is the file of the list along with the name of the person that either wrote it or made last significant modifications.

AMPI

chkpt                fails                        Gengbin
fallreduce compile fails Gengbin
jacobi3d no test target Esteban
mpich-test no test target
speed works! Gengbin

Charm++

kNeighbor        no test target        Chao
array4D crash Abhinav
pmetest compile fails Sameer
arrayPerf works! Sameer
python compile fails Filippo
topology works! Abhinav
broadcast works! Phil
io no test target Phil
reductiontesting no test target Rahul
jacobi3d no test target used for fttest
commSpeed works! Terry
jacobi3d-gauss no test target Yanhua
penciltest compile fails Sameer
commtest crash Abhishek
jacobi-sdag no test target used for fttest
ping crash Yanhua
startuptest compile fails Eric
hello-crosscorruption hangs

Note: for formated version please look at the attached file.

#2 Updated by Xiang Ni over 4 years ago

Running all the test only takes 80s. Here is the time break down for each directory:
AMPI 8
Charm++ 37
Converse 34
FEM 0.2
Util 0.02

However, there are tests in the test directory but never be run by auto build. Some of them even crash. Here is the list along with the name of the person that either wrote it or made last significant modifications.
AMPI
chkpt fails Gengbin
fallreduce compile fails Gengbin
jacobi3d no test target Esteban
mpich-test no test target
speed works! Gengbin
Charm++
kNeighbor no test target Chao
array4D crash Abhinav
pmetest compile fails Sameer
arrayPerf works! Sameer
python compile fails Filippo
topology works! Abhinav
broadcast works! Phil
io no test target Phil
reductiontesting no test target Rahul
jacobi3d no test target used for fttest
commSpeed works! Terry
jacobi3d-gauss no test target Yanhua
penciltest compile fails Sameer
commtest crash Abhishek
jacobi-sdag no test target used for fttest
ping crash Yanhua
startuptest compile fails Eric
hello-crosscorruption hangs

#3 Updated by Phil Miller over 4 years ago

Within tests/charm++ and tests/converse, there seem to be just a few things that make up the bulk of the 30 seconds each, and that add little in the way of correctness testing. The most noticeable is tests/charm++/queue/msgtest, with pgm in the same directory also being long-ish. tests/charm++/communication_overhead is similar. It's worth testing that the runtime doesn't fall over with long queues or large messages, but we don't need to hammer on them for nearly so long. charm++/taskSpawn{,Recursive} take 8 and 4 seconds, respectively, when their parameters could be reduced to run shorter.

#4 Updated by Phil Miller over 4 years ago

  • Target version set to 6.7.0

#5 Updated by Xiang Ni over 4 years ago

  • % Done changed from 50 to 70

Total time to run examples is 46s
charm++: 13s
converse: 0.317s
ampi: 32s
armci: 1.1s

#6 Updated by Nikhil Jain over 3 years ago

  • Target version changed from 6.7.0 to 6.7.1

#7 Updated by Sam White over 3 years ago

On my lab machine with a 'netlrts-linux-x86_64 --with-production' build, the total time to run 'tests' is 71s:
charm++: 39s
converse: 22s
ampi: 9s
fem: 0.2s
util: 0.01s

The total time to run 'examples' is 31s:
charm++: 28s
converse: 0.1s
ampi: 2s
armci: 0.7s

Are these runtimes acceptable or not?
Also, the perftest/ directory doesn't look like it does anything at all ... should we remove it?

#8 Updated by Sam White over 3 years ago

  • Category set to Build & Test Automation
  • Target version changed from 6.7.1 to 6.8.0

#9 Updated by Sam White over 2 years ago

  • Assignee changed from Xiang Ni to PPL

#10 Updated by Eric Bohm over 2 years ago

Here is an individual test time breakdown for make test netlrts-linux-x86_64 smp --with-production on intellect
https://docs.google.com/a/illinois.edu/spreadsheets/d/1xpCPCNWrrEIZmHGtAC_K1bkor2SOIDuxhp7kkbZq0x8/edit?usp=sharing

359s total time. However the wallclock time is somewhat larger as only the time of each testrun execution is recorded.

Only 5 tests require more than 10s.

Test time(s)
commbench/pgm 86.007
machinetest/multiping 19.88
megampi/pgm 16.885
Cjacobi3D/jacobi 14.72
kNeighbor/kNeighbor 10.774

So we have one major issue in that commbench can take long time. Then we have a minor issue in that each test takes over a second, so having over ninety of them means there is another minute and half minimum.

#11 Updated by Sam White over 2 years ago

We want megampi/pgm and Cjacobi3D/jacobi to run for more than a few seconds to test AMPI messaging/collectives/migration all in one go.

I think commbench and multiping can be cut down in runtime a bit...

#12 Updated by Eric Bohm over 2 years ago

The two commbench tests which take the longest are pingpong and flood. Each of these has entirely hard coded parameters. The iteration counters hard coded in to these tests appear to massively overkill the number required for a reasonable accuracy.

Reducing them by an order magnitude (or a factor of 2 for relatively small <1e2) cases cuts runtime down by a factor of 3.

#13 Updated by Sam White about 2 years ago

  • Target version changed from 6.8.0 to 6.8.1

#14 Updated by Eric Bohm almost 2 years ago

  • Target version changed from 6.8.1 to 6.9.0

#15 Updated by Phil Miller almost 2 years ago

  • Target version deleted (6.9.0)

Also available in: Atom PDF