Projections traces with fewer files than 1-per-PE
On trans-petascale systems with 100k+ cores, a file per PE is enough to seriously overwhelm parallel filesystem metadata servers, even in separate directories. We need to make an adjustment to reduce that burden.
#5 Updated by Ronak Buch over 2 years ago
This is a good point, and a change should definitely be made to plan ahead. I'll sketch out some possible designs as to what the future file formats can look like. Presumably things should be aggregated at the node level and files should be written at that granularity, too.