blazer wrote:BTW NTLM step rate is at about 1.1 Bln for me on the GTX470. hehe just under half of FRTs current one.
Wait, the FRTU CUDA client is pushing 2B steps/sec on your 470? HOW?
blazer wrote:BTW NTLM step rate is at about 1.1 Bln for me on the GTX470. hehe just under half of FRTs current one.
blazer wrote:how can you make perfect tables without making a lot of chains and going through them? can you somehow predict the merges and goto the next start point?
...
OH wait i think i got it. Can you simply not "discard" the data between the start and end point and perform a massive lookup each link which will allow you to prematurely terminate generating that chain if it is going to merge. Probably not a great idea ur going to have to search through allooot of data. Though you could only keep the reduced data each step i guess, to cut down on space.
Bitweasil wrote:Sc00bz: With the reduction function/table index issues, simply incrementing index values by more than the chain length would fix it with the current code, yes?
So for chain length 200,000 use indexes 0, 300000, 600000, 900000 ?
blazer wrote:Wouldn't it be wise though to say build them like this.
For a chain length of 200 000
50 000, 30 000, 29 000, 25 000, 22 000, 20 000 .......
Eg we decrease the number chain length extensions each step for the generation to counter the merges rather than use a constant extension.
blazer wrote:Does that mean we need to start off with like 10x the amount of chains <<Bad idea i think
or
Do we do a chunk of chains per time
blazer wrote:for a chain length of 200 000 you designate for example 8 checkpoints
at 50 000, 40 000, 35 000, 25 000, 20 000, 15 000, 10 000, 5 000 <-- Hopefully that adds up
Now you generate say (using sequential start points)
1/ 10 000 000 chains @ 50 000 chainL
Perfect Sort + Make a copy
2/ partial perfect chains +40, 000 increment chainL
etc etc
3/ Reach last checkpoint and we should have a bunch of chains remaining at chain length 200 000 + we should have 8 checkpoint files containing end points at the different positions
Now you use the next lot of sequential start points
Repeat, BUT since we now have the data for the above 8 check points we can perfect our next chains taking into account all those partial chains at the different checkpoints and we also up the number of starting chains to 15 000 000 since there will obviously be more merges.
Numbers were randomly picked so probably don't add up or bad use of numbers, but just was curious about this concept
blazer wrote:btw, what happens if you don't increment the index greater than chainL ?
.....................
.....................
.....................
.....................
Bitweasil wrote:The issue there is the memory requirements needed. To do this "merge reduction" bit, you basically have to generate the *entire tableset* to 50,000 then check/remove duplicates, and do the same thing as you go forward. It doesn't work nearly as well distributed, unless there's something about it I'm missing.
When talking about tables on the order of several TB, you'd need to do this locally with a cluster, I think. Something with a lot of bandwidth. The size of a partial table is the same as the size of a full table.
Users browsing this forum: No registered users and 1 guest