Cryptohaze.com

With fixes for the GRTIndex issue on Windows as well as fixes to WebTables support that should dramatically improve performance & reliability.

http://sourceforge.net/projects/cryptoh ... -Combined/

Enjoy!

Great to see a new version, found a bug for ya though;
When manually setting --seed to a value > 2147483647, the generate seed = 2147483647 instead of the entered value. This was on Windows (Win7 x64 CUDA and OpenCL).

Found this when I was trying to validate some tables between my CUDA and OpenCL systems. Otherwise speeds are nice for OpenCL. Pulling ~540M/s doing SHA1 len7 chr95 on a 5850, over 3x the speed of my Quadro FX 4800 card.

I did waste a day of compute on my OpenCL system since I forgot that I had overclocked the GPU on that system . With the overclock that I was running I was getting silent corruption of the data and repeated runs with the same seed value resulted in generated tables with different file hashes. I still have the GPU overclocked a bit, but it is down where I could verify that the data produced was consistent.

MaddGamer wrote:Great to see a new version, found a bug for ya though;
When manually setting --seed to a value > 2147483647, the generate seed = 2147483647 instead of the entered value. This was on Windows (Win7 x64 CUDA and OpenCL).

Interesting. That looks like 2^31. It may clip above that - my seed is definitely a 32-bit number. It should be using an unsigned int, not a signed int - so I'll check into that. But, in general, I'm not sure that's really a "bug" so much as "Designed limit." Or is it that it's on Windows only, and Linux works up to 2^32?

Found this when I was trying to validate some tables between my CUDA and OpenCL systems. Otherwise speeds are nice for OpenCL. Pulling ~540M/s doing SHA1 len7 chr95 on a 5850, over 3x the speed of my Quadro FX 4800 card.

Cool. I don't have a fully optimized SHA1 algorithm for ATI yet - I'm not using BFI_INT. Need to fix that. Should be worth some speed on SHA1/etc.

I did waste a day of compute on my OpenCL system since I forgot that I had overclocked the GPU on that system . With the overclock that I was running I was getting silent corruption of the data and repeated runs with the same seed value resulted in generated tables with different file hashes. I still have the GPU overclocked a bit, but it is down where I could verify that the data produced was consistent.

Yep. That'll happen. There's also the GRTVerify tool that will check chains - you can run it against a table and it will, by default, check every chain. I'd be interested to see some of the corruption you're seeing. But, yes, overclocking is bad for compute, m'kay?

Thanks for the feedback, and glad it's working!

Bitweasil wrote:Interesting. That looks like 2^31. It may clip above that - my seed is definitely a 32-bit number. It should be using an unsigned int, not a signed int - so I'll check into that. But, in general, I'm not sure that's really a "bug" so much as "Designed limit." Or is it that it's on Windows only, and Linux works up to 2^32?

It looks like the argument parser is maybe setting a signed int, that was then assigned to the unsigned int variable that you use. The normal seed number generation is using a uint. I didn't dig too far into the code to verify as it probably isn't an issue for most people, since normally one shouldn't use the --seed option. I have not tested on Linux.

That's quite likely what is happening.

Alright, ran into something odd here. I did a merge on some of my tables for len7, and the grtmerge process says it wrote out 296,226,124 chains after the merge. But GRTAnalyze and GRTVerify are showing only 91,703,871 chains

The header has the correct number of chains listed, and it looks like mmap may be using a uint for the definition of std::size_t, which is the type for file_status.st_size in the mmap call. This is on Windows.

Did the math and the file size of the merged table (6220748604 bytes) works out that with that value overflowing a uint you would have 1925789501 bytes.

(1925789501 - 8192) / 21 = 91703871.85 => 91703871 chains

*sigh* Yeah. Sounds likely. I'll look into large file support for Windows - my first guess was the printf overflowing, but it sounds like it's most likely an issue with the file size based on your maths.

Seriously, Windows is such a pain in the ass.

Are you comfortable coding? For some of this stuff, it might be nice to have you take a stab at fixing it & sending me patches. I really don't do much testing on Windows, because all my "heavy lifting" boxes are Linux. I will verify that it's not an issue on Linux, but I'm fairly certain it's not, as I've dealt with 300GB files on Linux.

I could setup a vm for VS 2008 and try to get the dependencies all setup to first see if I can get an environment where I can make a successful build as is. I would just have reservations that I am definitely not as familiar with the code as you are.

Got VS 2008 setup and boost all built up for 64 bit, still had the issue. Next I changed file_status to be a _stat64 instead of a stat struct in GRTTableSearchV2::SetTableFilename

Rebuilt GRTAnalyse and now it sees all the records correctly.

So far I am patching it by adding the following at line 12 on GRTTableSearchV2.cpp

Code: Select all

#if _MSC_VER >= 1000
#define stat    _stat64
#endif


Not sure how clean that would be to how you are setting up the code. Probably also need the same in the V1 file as well? I havn't used any V1 tables to test against.

Awesome. I'll get that patch submitted in the next day or two!

How interested are you in developing/debugging? I could get you SVN commit access over at sourceforge, or you could just send me patches like this.

I definitely appreciate the time you've taken to work on the Windows code - I really only know 64-bit Posix, and Windows is an awful lot of guessing to me.