Project

General

Profile

Bug #1957

Out of bounds std::vector accesses in NDMeshStreamer

Added by Nitin Bhat 10 months ago. Updated about 2 months ago.

Status:
Merged
Priority:
High
Category:
-
Target version:
Start date:
08/07/2018
Due date:
% Done:

0%


Description

Reproducible with the following one-liner:

./build AMPI multicore-linux-x86_64 --suffix=asan -j8 -g3 -fsanitize=address && pushd multicore-linux-x86_64-asan/tests/charm++/streamingAllToAll/ && make OPTS="-g3 -fsanitize=address" -j8 && ./ataTest

Originally found on Windows, due to a bounds check only present in Microsoft's C++ STL in debug mode:

Charm build command: ./build LIBS mpi-win-x86_64 smp --enable-error-checking --without-romio --suffix=debug -j8 -g -O0 |& tee build_result_debug

$ make test
../../../bin/testrun  +p4 ./ataTest

Running on 4 processors:  ./ataTest
charmrun> /cygdrive/c/Program Files/Microsoft MPI/Bin/mpiexec -n 4  ./ataTest

Charm++> Running on MPI version: 2.0
Charm++> level of thread support used: MPI_THREAD_FUNNELED (desired: MPI_THREAD_FUNNELED)
Charm++> Running in SMP mode: 4 processes, 1 worker threads (PEs) + 1 comm threads per process, 0 PEs total
Charm++> The comm. thread both sends and receives messages
Charm++ warning> fences and atomic operations not available in native assembly
Converse/Charm++ Commit ID: v6.8.2-853-g4146bf788
Charm++> Disabling isomalloc because mmap() does not work.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.000 seconds.
size of envelope: 64

TEST 1: Using 2D TRAM Topology: 4 1
Elapsed time for all-to-all of       32 bytes sent in      1  iteration of 32 bytes each (    using TRAM): 0.000000 seconds
Elapsed time for all-to-all of       64 bytes sent in      2 iterations of 32 bytes each (    using TRAM): 0.000000 seconds
Elapsed time for all-to-all of      128 bytes sent in      4 iterations of 32 bytes each (    using TRAM): 0.000000 seconds
Elapsed time for all-to-all of      256 bytes sent in      8 iterations of 32 bytes each (    using TRAM): 0.000000 seconds
Elapsed time for all-to-all of      512 bytes sent in     16 iterations of 32 bytes each (    using TRAM): 0.000000 seconds
Elapsed time for all-to-all of     1024 bytes sent in     32 iterations of 32 bytes each (    using TRAM): 0.000000 seconds
Elapsed time for all-to-all of     2048 bytes sent in     64 iterations of 32 bytes each (    using TRAM): 0.000000 seconds
Elapsed time for all-to-all of     4096 bytes sent in    128 iterations of 32 bytes each (    using TRAM): 0.016000 seconds
Elapsed time for all-to-all of     8192 bytes sent in    256 iterations of 32 bytes each (    using TRAM): 0.000000 seconds

job aborted:
[ranks] message

[0-2] terminated

[3] process exited without calling finalize

---- error analysis -----

[3] on CS-DEXTERITY
./ataTest ended prematurely and may have crashed. exit code 0xc0000417

---- error analysis -----
make: *** [Makefile:22: test-ataTest] Error 127

0001-currentStage-debugging.patch View (3.09 KB) Evan Ramos, 03/19/2019 11:40 AM

History

#1 Updated by Evan Ramos 9 months ago

  • Target version deleted (6.9.0)

#2 Updated by Eric Bohm 8 months ago

  • Assignee set to Evan Ramos

#3 Updated by Evan Ramos 6 months ago

Was this fixed by #1960?

#4 Updated by Evan Ramos 3 months ago

  • Target version set to 6.10.0

#5 Updated by Evan Ramos 3 months ago

This problem is caused by indexing -1 into a vector.

ataTest.exe!std::vector<int,std::allocator<int> >::operator[](const unsigned __int64 _Pos) Line 1733
    at c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.16.27023\include\vector(1733)
ataTest.exe!MeshStreamer<DataItem,SimpleMeshRouter>::checkForCompletedStages() Line 219
    at include\ndmeshstreamer.h(219)
ataTest.exe!MeshStreamer<DataItem,SimpleMeshRouter>::receiveAlongRoute(MeshStreamerMessage<DataItem> * msg) Line 672
    at include\ndmeshstreamer.h(672)
ataTest.exe!CkIndex_MeshStreamer<DataItem,SimpleMeshRouter>::_call_receiveAlongRoute_MeshStreamerMessage(void * impl_msg, void * impl_obj_void) Line 720
    at include\ndmeshstreamer.def.h(720)
ataTest.exe!CkDeliverMessageFree(int epIdx, void * msg, void * obj) Line 578
    at tmp\ck.c(578)
ataTest.exe!_invokeEntryNoTrace(int epIdx, envelope * env, void * obj) Line 621
    at tmp\ck.c(621)
ataTest.exe!_invokeEntry(int epIdx, envelope * env, void * obj) Line 632
    at tmp\ck.c(632)
ataTest.exe!_deliverForBocMsg(CkCoreState * ck, int epIdx, envelope * env, IrrGroup * obj) Line 1096
    at tmp\ck.c(1096)
ataTest.exe!_processForBocMsg(CkCoreState * ck, envelope * env) Line 1111
    at tmp\ck.c(1111)
ataTest.exe!_processHandler(void * converseMsg, CkCoreState * ck) Line 1275
    at tmp\ck.c(1275)
ataTest.exe!CmiHandleMessage(void * msg) Line 1678
    at tmp\convcore.c(1678)
ataTest.exe!CsdScheduleForever() Line 1932
    at tmp\convcore.c(1932)
ataTest.exe!CsdScheduler(int maxmsgs) Line 1859
    at tmp\convcore.c(1859)
ataTest.exe!ConverseRunPE(int everReturn) Line 1607
    at tmp\machine-common-core.c(1607)
ataTest.exe!ConverseInit(int argc, char * * argv, void(*)(int, char * *) fn, int usched, int initret) Line 1500
    at tmp\machine-common-core.c(1500)
ataTest.exe!charm_main(int argc, char * * argv) Line 1713
    at tmp\init.c(1713)
ataTest.exe!main(int argc, char * * argv) Line 6
    at tmp\main.c(6)
[External Code]

NDMeshStreamer.h:219, currentStage is -1.

  inline void checkForCompletedStages() {
    int &currentStage = myCompletionStatus_.stageIndex;
    while (cntFinished_[currentStage] == myCompletionStatus_.numContributors &&
           cntMsgExpected_[currentStage] == cntMsgReceived_[currentStage]) {

#6 Updated by Evan Ramos 3 months ago

Running the streamingAllToAll test case on Linux with ASan also exposes these issues, and printf traces of the value of stageIndex show the same values on Windows and Linux. It looks like the reason this only crashes, as the issue title says, in Windows debug builds, is because Microsoft's C++ STL adds a bounds check to std::vector<T>::operator[] in debug mode.

public:
    _NODISCARD _Ty& operator[](const size_type _Pos)
        {    // subscript mutable sequence
 #if _ITERATOR_DEBUG_LEVEL != 0
        _STL_VERIFY(_Pos < size(), "vector subscript out of range");
 #endif /* _ITERATOR_DEBUG_LEVEL != 0 */

        return (this->_Myfirst()[_Pos]);
        }

#7 Updated by Evan Ramos 2 months ago

  • Status changed from New to In Progress

#8 Updated by Evan Ramos 2 months ago

  • Description updated (diff)
  • Subject changed from tests/charm++/sdag/streamingAllToAll fails on mpi-win-x86_64-smp with debug options (-g -O0) to Out of bounds std::vector accesses in NDMeshStreamer

Updating this issue to reflect its general nature as opposed to being Windows-specific.

The bug can be reproduced with this one-liner:

./build AMPI multicore-linux-x86_64 --suffix=asan -j8 -g3 -fsanitize=address && pushd multicore-linux-x86_64-asan/tests/charm++/streamingAllToAll/ && make OPTS="-g3 -fsanitize=address" -j8 && ./ataTest

#9 Updated by Evan Ramos 2 months ago

  • Assignee changed from Evan Ramos to Venkatasubrahmanian Narayanan

Reassigning this since it appears to be a logic error in NDMeshStreamer.

#10 Updated by Evan Ramos 2 months ago

Here is a patch for debugging that adds some printf tracing of the index value that goes out of bounds.

#11 Updated by Evan Ramos about 2 months ago

  • Status changed from In Progress to Merged

Also available in: Atom PDF