New-Multiforcer Segfault when using multiple GPUs

[dyngnosis]

I am getting seg faults when using the latest New-Multiforcer

[1] I used the build script on a fresh Ubuntu 12.04.1 install.
[2] Segfault only occurs when using multiple GPUs. If run alone each GPU is fine.
[3] Segfault only happens *most* of the time. If I keep running it it will eventually work.
[4] The first GPU always seems to start up fine. The crash happens when the second GPU starts up.

Here is the output with --debug --debug-dev
-------------------------------------------------------------

=========Charset Dumping========
isSingleCharset: 1
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 0.00 /s
MFND: Total rate: 0.00
Returning ClientID: 1804289383
MFND: Status: Td 1: CID 17767.
Returning ClientID: 846930886
MFND: Status: Td 0: CID 9158.
Popped WU ID 0
In flight WUs: 1
this->UseResumeFile not set: Returning.
WorkUnitID: 0
StartPoint: 0
EndPoint: 17179869183
WorkunitRequestedTimestamp: 4.991725
WorkunitCompletedTimestamp: 0.000000
ClientId: 9158
IsAssigned: 1
PasswordLength: 7
IsValid: 1
Flags: 0x00
Data size: 0
Thread 0 has workunit ID 0
ref_time: 0.013000
0.00% done with WU
ref_time: 12.110000
0.15% done with WU
ref_time: 11.745000
0.31% done with WU

Thread 0 Adjusting passwords per step to 851
ref_time: 183.134995
2.90% done with WU
ref_time: 183.033005
5.50% done with WU
ref_time: 183.210999
18.48% done with WU
MFND: Thread 0 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 2.44B/s
MFND: Total rate: 2.44B
Segmentation fault (core dumped)

And here is the output when it eventually works:
-------------------------------------------------------------

=========Charset Dumping========
isSingleCharset: 1
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 0.00 /s
MFND: Total rate: 0.00
Returning ClientID: 1804289383
MFND: Status: Td 1: CID 17767.
Returning ClientID: 846930886
MFND: Status: Td 0: CID 9158.
Popped WU ID 0
In flight WUs: 1
this->UseResumeFile not set: Returning.
WorkUnitID: 0
StartPoint: 0
EndPoint: 17179869183
WorkunitRequestedTimestamp: 4.986263
WorkunitCompletedTimestamp: 0.000000
ClientId: 9158
IsAssigned: 1
PasswordLength: 7
IsValid: 1
Flags: 0x00
Data size: 0
Thread 0 has workunit ID 0
ref_time: 0.014000
0.00% done with WU
ref_time: 12.113000
0.15% done with WU
ref_time: 11.739000
0.31% done with WU

Thread 0 Adjusting passwords per step to 851
ref_time: 183.201004
2.90% done with WU
ref_time: 183.097000
5.50% done with WU
ref_time: 183.276001
18.48% done with WU
MFND: Thread 0 setting rate to 2.43B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 2.43B/s
MFND: Total rate: 2.43B
Popped WU ID 1
In flight WUs: 2
this->UseResumeFile not set: Returning.
WorkUnitID: 1
StartPoint: 17179869184
EndPoint: 34359738367
WorkunitRequestedTimestamp: 7.383879
WorkunitCompletedTimestamp: 0.000000
ClientId: 17767
IsAssigned: 1
PasswordLength: 7
IsValid: 1
Flags: 0x00
Data size: 0
Thread 1 has workunit ID 1
ref_time: 0.002000
0.00% done with WU
ref_time: 12.069000
0.15% done with WU
ref_time: 11.646000
0.31% done with WU

Thread 1 Adjusting passwords per step to 858
ref_time: 184.619003
2.92% done with WU
ref_time: 183.054001
31.46% done with WU
MFND: Thread 0 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 2.44B/s
MFND: Total rate: 2.44B
ref_time: 184.572006
5.54% done with WU
ref_time: 183.007996
44.45% done with WU
MFND: Thread 0 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 2.44B/s
MFND: Total rate: 2.44B
ref_time: 184.643005
18.63% done with WU
MFND: Thread 1 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
ref_time: 183.132996
57.43% done with WU
MFND: Thread 0 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
ref_time: 184.472000
31.72% done with WU
MFND: Thread 1 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
ref_time: 183.138000
70.41% done with WU
MFND: Thread 0 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
ref_time: 184.557999
44.81% done with WU
MFND: Thread 1 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
ref_time: 183.108002
83.39% done with WU
MFND: Thread 0 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
ref_time: 184.550995
57.90% done with WU
MFND: Thread 1 setting rate to 2.44B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
^CMFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 4.87B/s
MFND: Total rate: 4.87B
MFND: Thread 0 setting rate to 0.00
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 2.44B/s
MFND: Total rate: 2.44B
MFND: Status: Waiting for workunits...
MFND: Thread 1 setting rate to 0.00
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 0.00 /s
MFND: Total rate: 0.00
MFND: Status: Waiting for workunits...
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 0.00 /s
MFND: Total rate: 0.00
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 0.00 /s
MFND: Total rate: 0.00
MFND: WU status: 0/4065
MFND: Hash status: 0/13
MFND: Total crack rate 0.00 /s
MFND: Total rate: 0.00
gpu@gpu:~/CHDev/Cryptohaze-Dev/build/bin$

Here is a backtrace from gdb:
-------------------------------------------------------------

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffeaffa700 (LWP 3371)]
0x0000000002514340 in ?? ()
(gdb) bt
#0 0x0000000002514340 in ?? ()
#1 0x00007ffff4d3aca7 in clEnqueueWriteBuffer () from /usr/lib/fglrx/libamdocl64.so
#2 0x00000000005ae938 in MFNHashTypePlainOpenCL::copyDataToDevice() ()
#3 0x000000000050cd0f in MFNHashTypePlain::crackPasswordLength(int) ()
#4 0x00000000005096ec in boost::_mfi::mf1<void, MFNHashType, int>::operator()(MFNHashType*, int) const ()
#5 0x0000000000509635 in void boost::_bi::list2<boost::_bi::value<MFNHashType*>, boost::_bi::value<unsigned short> >::operator()<boost::_mfi::mf1<void, MFNHashType, int>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, MFNHashType, int>&, boost::_bi::list0&, int) ()
#6 0x00000000005095c1 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, MFNHashType, int>, boost::_bi::list2<boost::_bi::value<MFNHashType*>, boost::_bi::value<unsigned short> > >::operator()() ()
#7 0x0000000000509586 in boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf1<void, MFNHashType, int>, boost::_bi::list2<boost::_bi::value<MFNHashType*>, boost::_bi::value<unsigned short> > > >::run() ()
#8 0x00000000005da938 in thread_proxy ()
#9 0x00007ffff79bde9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ffff6630cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

[dyngnosis]

I recompiled after enabling tracing. Here is the latest:

gpu@gpu:~/CHDev/Cryptohaze-Dev/build/bin$ gdb --args ./New-Multiforcer -h MD5 -c charsets/charsetall -f test_hashes/Hashes-MD5-Full.txt --min 1 --max 8 --vectorwidth=8 --nocuda --nocpu --opencldevice={0,1} --debug --devdebug
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /home/gpu/CHDev/Cryptohaze-Dev/build/bin/New-Multiforcer...done.
(gdb) run
Starting program: /home/gpu/CHDev/Cryptohaze-Dev/build/bin/New-Multiforcer -h MD5 -c charsets/charsetall -f test_hashes/Hashes-MD5-Full.txt --min 1 --max 8 --vectorwidth=8 --nocuda --nocpu --opencldevice=0 --opencldevice=1 --debug --devdebug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Device 0: OpenCL, p:0, d:0
Device 1: OpenCL, p:0, d:1
MFNWorkunitRobust::MFNWorkunitRobust()
MFNWorkunitRobust::ClearAllInternalState()
MFND: Setting hash name MD5
MFNHashType::MFNHashType()
MFNHashTypePlain::MFNHashTypePlain(16)
MFNHashTypePlainOpenCL::MFNHashTypePlainOpenCL(16)
MFNHashType GPU/OpenCL Thread ID 0
MFNHashTypePlainOpenCL_MD5::MFNHashTypePlainOpenCL_MD5()
MFNHashTypePlainOpenCL::setOpenCLDeviceID(0, 0)
[New Thread 0x7ffff0e08700 (LWP 6108)]
Thread 0 added OpenCL Device (0, 0)
MFNHashType::MFNHashType()
MFNHashTypePlain::MFNHashTypePlain(16)
MFNHashTypePlainOpenCL::MFNHashTypePlainOpenCL(16)
MFNHashType GPU/OpenCL Thread ID 1
MFNHashTypePlainOpenCL_MD5::MFNHashTypePlainOpenCL_MD5()
MFNHashTypePlainOpenCL::setOpenCLDeviceID(0, 1)
[New Thread 0x7ffff0342700 (LWP 6109)]
Thread 1 added OpenCL Device (0, 1)
MFND: Setting password len 1
MFND: Status: Starting pw len 1
MFNWorkunitRobust::CreateWorkunits(95, 34, 1)
MFNWorkunitRobust::ClearAllInternalState()
Elements per unit: 17179869184
Total number of workunits: 1
MFNWorkunitRobust::CreateMorePendingWorkunits(500)
WU 0: SP: 0 EP: 94
MFNWorkunitRobust::WriteSaveState(1)
this->UseResumeFile not set: Returning.
MFNHashClassLauncher launching thread 0
[New Thread 0x7fffeb7fb700 (LWP 6110)]
MFNHashClassLauncher launching thread 1
MFNHashTypePlain::crackPasswordLength(1)
Thread 0 doing MFNHashTypePlain setup.
MFND: Setting password len 1
Thread 0 Charset length: 1
=========Charset Dumping========
isSingleCharset: 1
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
MFNHashTypePlain::setupCharsetArrays()
MFND: WU status: 0/1
MFND: Hash status: 0/13
MFND: Total crack rate 0.00 /s
MFND: Total rate: 0.00
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlainOpenCL_MD5::preProcessHash()
MFNHashTypePlain::sortHashes()
hash 0: 8a5ecc06272cfa3bd02d637daa970d5b
hash 1: ccb1b10f9124fea8e0524f8c1a2ef478
hash 2: 4500aa10b0208cee47d2c7b587ecbc83
hash 3: 22d3762f6eb1b2bdc9cae6f72ea73ef3
hash 4: 1e89ce3148c12e9070ff05cfbbb0b07a
hash 5: e4446c46bfdb0a06fc20595afe3fad22
hash 6: c040c6558bf7ce0ce871a13237e82d64
hash 7: 51b4ca6ec932e28bed42ad68247830fa
hash 8: 2c092f81f612cef08eba3060fdb9cad8
hash 9: a3984cbbaa5f0c7cf7ab99ea16a44eb1
hash 10: 9c5484bdfa3211e34070d0c2b95b743e
hash 11: 6cb569c5da043c85e91f9fd9a13ae982
hash 12: c8d32ec752d028d33b5ebc1b67a70071
MFNHashTypePlain::copyHashesIntoDeviceFormat()
Created common hash array of 208 bytes.
MFNHashTypePlain::createLookupBitmaps()
Creating bitmaps for word 0/a
[New Thread 0x7fffeaffa700 (LWP 6111)]
MFNHashClassLauncher trying to join thread 0
Thread 0 is joinable
MFNHashTypePlain::crackPasswordLength(1)
Creating bitmaps for word 1/b
Creating bitmaps for word 2/c
Creating bitmaps for word 3/d
MFNWorkunitRobust::GetClientId()
Returning ClientID: 1804289383
MFND: Status: Td 1: CID 17767.
MFNWorkunitRobust::GetClientId()
Returning ClientID: 846930886
MFND: Status: Td 0: CID 9158.
CHHashTypeVPlainOpenCL::setupDevice()
Thread 0 setting OpenCL platform/device to 0, 0
MFNHashTypePlainOpenCL::allocateThreadAndDeviceMemory()
Attempting to openclMalloc 208 bytes for device hashlist for thread 0.
Attempting to cudaHostAlloc 13 bytes for HostSuccess
Successfully allocated 8kb Bitmap A
Successfully allocated 16kb Bitmap A
Successfully allocated 8kb Bitmap A
Successfully allocated 256kb Bitmap A
Successfully allocated Bitmap A
Bitmap A Address: 7fffe013ab40
Successfully allocated Bitmap B
Bitmap B Address: 7fffe013acf0
Successfully allocated Bitmap C
Not enough free GPU mem to attempt 128MB bitmap D alloc.
Thread 0 memory allocated successfully
CHHashTypeVPlainOpenCL::doKernelSetup()
Making single charset incrementors.
passLength: 1
vectorWidth: 8
numberBitmaps: 3
MFNHashTypePlainOpenCL_MD5::getHashFileNames()
CHHashTypeVPlainOpenCL::setupDevice()
Thread 1 setting OpenCL platform/device to 0, 1
MFNHashTypePlainOpenCL::allocateThreadAndDeviceMemory()
Attempting to openclMalloc 208 bytes for device hashlist for thread 1.
Attempting to cudaHostAlloc 13 bytes for HostSuccess
Successfully allocated 8kb Bitmap A
Successfully allocated 16kb Bitmap A
Successfully allocated 8kb Bitmap A
Successfully allocated 256kb Bitmap A
Successfully allocated Bitmap A
Bitmap A Address: 7fffd00822a0
Successfully allocated Bitmap B
Bitmap B Address: 7fffd0082410
Successfully allocated Bitmap C
Not enough free GPU mem to attempt 128MB bitmap D alloc.
Thread 1 memory allocated successfully
CHHashTypeVPlainOpenCL::doKernelSetup()
Making single charset incrementors.
passLength: 1
vectorWidth: 8
numberBitmaps: 4
MFNHashTypePlainOpenCL_MD5::getHashFileNames()
MFNHashTypePlainOpenCL_MD5::getHashKernelName()
MFNHashTypePlainOpenCL_MD5::getHashKernelName()
MFNHashTypePlainOpenCL::copyDataToDevice()
Thread 0: Copying bitmap A
Thread 0: Copying bitmap B
Thread 0: Copying bitmap C
MFNHashTypePlainOpenCL_MD5::copyConstantDataToDevice()
MFNWorkunitRobust::GetNextWorkunit(9158)
Popped WU ID 0
In flight WUs: 1
MFNWorkunitRobust::WriteSaveState(0)
this->UseResumeFile not set: Returning.
WorkUnitID: 0
StartPoint: 0
EndPoint: 94
WorkunitRequestedTimestamp: 4.572105
WorkunitCompletedTimestamp: 0.000000
ClientId: 9158
IsAssigned: 1
PasswordLength: 1
IsValid: 1
Flags: 0x00
Data size: 0
Thread 0 has workunit ID 0
MFNHashTypePlain::RunGPUWorkunitBF()
Thread 0 total kernel width: 524288
Thread 0 blocks/threads/vec: 256/256/8
Total kernel width: 524288
perThread: 1
divergence adjusted perThread to: 95
MFNHashTypePlain::setStartPasswords32()
Calculating start points for a single charset.
MFNHashTypePlainOpenCL::copyStartPointsToDevice()
MFNHashTypePlainOpenCL::synchronizeThreads()
ref_time: 0.000050 s
MFNHashTypePlainOpenCL::copyDeviceFoundPasswordsToHost()
MFNHashTypePlainOpenCL::outputFoundHashes()
ref_time: 0.050000
0.00% done with WU
Launching kernel:
start_point: 0
perStep: 50
num salts: 1
saltStartOffset: 0
MFNHashTypePlainOpenCL_MD5::launchKernel()
T 0: Platform/Device: 0/0
T 0: Workgroups/Workitems: 65536/256
start_point: 50
per_thread: 95
Will overrun by 5
Final per_step: 46
MFNHashTypePlainOpenCL::synchronizeThreads()
ref_time: 0.012051 s
MFNHashTypePlainOpenCL::copyDeviceFoundPasswordsToHost()
MFNHashTypePlainOpenCL::outputFoundHashes()
ref_time: 12.051000
52.63% done with WU
Launching kernel:
start_point: 50
perStep: 46
num salts: 1
saltStartOffset: 0
MFNHashTypePlainOpenCL_MD5::launchKernel()
T 0: Platform/Device: 0/0
T 0: Workgroups/Workitems: 65536/256
MFNHashTypePlainOpenCL::synchronizeThreads()
MFND: Thread 0 setting rate to 2.22B
MFND: WU status: 0/1
MFND: Hash status: 0/13
MFND: Total crack rate 2.22B/s
MFND: Total rate: 2.22B
MFNHashTypePlainOpenCL::copyDeviceFoundPasswordsToHost()
MFNHashTypePlainOpenCL::outputFoundHashes()
Workunit rate: 1084.448546
Workunit timer: 0.086680
ref_time_total: 0.012101
Correcting perStep from current 46 to perm 50
MFNWorkunitRobust::SubmitWorkunit(...)
MFNWorkunitRobust::SubmitWorkunitById(0)
Found inflight WU ID: 0
Inflight left: 0
MFNWorkunitRobust::PrintInternalState()
Number WU total: 1
Number WU completed: 1
Number WU left: 0
Number WU inflight: 0
Active client IDs: 1804289383, 846930886,
MFNWorkunitRobust::WriteSaveState(0)
this->UseResumeFile not set: Returning.
MFND: WU status: 1/1
MFND: Hash status: 0/13
MFND: Total crack rate 2.22B/s
MFND: Total rate: 2.22B
MFNWorkunitRobust::GetNextWorkunit(9158)
pendingWorkunits.size() == 0; returning.
WorkUnitID: 0
StartPoint: 0
EndPoint: 0
WorkunitRequestedTimestamp: 0.000000
WorkunitCompletedTimestamp: 0.000000
ClientId: 0
IsAssigned: 0
PasswordLength: 0
IsValid: 0
Flags: 0x04
Data size: 0
MFND: Status: Td 0: out of WU.
MFNHashTypePlainOpenCL::freeThreadAndDeviceMemory()
MFNHashTypePlainOpenCL::teardownDevice()
MFND: Thread 0 setting rate to 0.00
MFND: WU status: 1/1
MFND: Hash status: 0/13
MFND: Total crack rate 0.00 /s
MFND: Total rate: 0.00
Thread 0 is joined
MFNHashClassLauncher trying to join thread 1
Thread 1 is joinable
[Thread 0x7fffeb7fb700 (LWP 6110) exited]
MFNHashTypePlainOpenCL_MD5::getHashKernelName()
MFNHashTypePlainOpenCL_MD5::getHashKernelName()
MFNHashTypePlainOpenCL::copyDataToDevice()
Thread 1: Copying bitmap A
Thread 1: Copying bitmap B
Thread 1: Copying bitmap C
Thread 1: Copying bitmap D

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffeaffa700 (LWP 6111)]
0x00007ffff4d3ac9c in clEnqueueWriteBuffer () from /usr/lib/fglrx/libamdocl64.so

[tazy]

recompiled after enabling tracing.??????????????????????