New design paradigm discussion: Massively Multiprocess

[foobar2342]

I feel the sudden need to dump my ideas here, of which about 60% are already implemented in my code, which is unfortunately:( windows-specific.

--Extension support. By using dynamic loading of dll's with a common interface, you can extend the code to support virtually any architecture. Is there any way to port the use of dll's to *nix?

man dlopen

[vampyr]

About controlling cheating, Lets go the slightly botnet-ish approach shall we?

About the network, the tracker keeps track of uptime of the peers.
Once in a while (1-2 hours) the tracker promotes random nodes to supernodes. Every previous supernode will no longer be a supernode.
These supernodes are marked in the peer list. ONLY SUPERNODES can send work to nodes. Nodes request work from their nearest supernode.
This allows for grouping of hashes etc. Nodes reply back to the supernode with a result, supernode checks it, if neccasary updates the tracker with cheaters,
and sends the result back to the client requesting the cracking of the hashes.

There's a few things we'd need for this to be successful tho.
--A SIMPLE API, in essence we only require 3 api calls.
@@-ProcessWork(work)
@@-AcceptingWork(workType)
@@-NumberOfDevices(workType)
this would make it easy for anyone to extend the cracker with their own algorithms.

--a GUI (yea i know that sounds shitty to code, and it IS, but we're going to need a GUI coder, or at least a simple interface)
--some algorithms to start with (i think that's sorted already)
--a network architecture (c++) --Someone's going to be coding this (And i wouldn't mind doing so)
--a tracker architecture (LAMP)-- Yep, thats right, trackers. (And if you want SQLi bugs, yeah you should probably ask me to do this;) )

So essentially we'd need 2-3 coders for this. It's a BIG project tho, so we'll first have to see which code we can borrow (libraries;) )
The GUI could probably be made in c#, as long as the extensions are in c.
The network architecture could be added as an extension itself, plus speed is kind of essential, so i recommend c++ (c# would probably do too).
Tracker's by far the easiest, nothing special about it.

[edit]Also, a massively distributed cracker like this would make rainbow tables practically obsolete. [/edit]

[neinbrucke]

i somehow fail the see a reason why you would want to do this peer2peer wise. Brute forcing jobs don't take a whole lot of bandwidth, as you will need trackers anyway, you can just as well use those as central servers that distribute work.

and rainbow tables will never be useless... some hashes are not meant to be distributed on public networks.

[vampyr]

Except if those jobs contain +- 4k hashes, and your splitting up work for say 1-8 mixedalpha-numeric into 5 minute pieces.
Yes you *could* use the trackers to distribute the work, but 4k*20=80k for one task. Now imagine having 250 nodes, yeah it's kind of like that.
The _simpler_ the tracker is, the more people can host one, strengthening the network.
Plus that it would not require much extra code to pick x ip adresses from the tracker, and ask them if they can spare some processing time for your work.
Oh, that and that trackers, if done your way, would have to synchronize with each other. Assuming a supernode can perform the same tasks as a normal node + some,
synchronization between trackers would not be necessary.

So yes, it would probably save you a little time coding (not a terrible lot tho, probably a day or two), but it'd make your network more vulnerable to DOS (How are you going to submit hashes to the server, use a CAPTCHA? How are you going to handle proxies?), and as with one central server there IS no redundancy, well if there's an outage your screwed. Plus that ping times generally greatly benefit from using a server near you;)

[neinbrucke]

i liked the idea better when the focus was on personal cracking networks with a possible future option for public online crackers. (btw, with peer2peer you also have the trouble that people should adjust their firewalls and such.)

[foobar2342]

About controlling cheating, Lets go the slightly botnet-ish approach shall we?

About the network, the tracker keeps track of uptime of the peers.
Once in a while (1-2 hours) the tracker promotes random nodes to supernodes. Every previous supernode will no longer be a supernode.
These supernodes are marked in the peer list. ONLY SUPERNODES can send work to nodes. Nodes request work from their nearest supernode.
This allows for grouping of hashes etc. Nodes reply back to the supernode with a result, supernode checks it, if neccasary updates the tracker with cheaters,
and sends the result back to the client requesting the cracking of the hashes.

There's a few things we'd need for this to be successful tho.
--A SIMPLE API, in essence we only require 3 api calls.
@@-ProcessWork(work)
@@-AcceptingWork(workType)
@@-NumberOfDevices(workType)
this would make it easy for anyone to extend the cracker with their own algorithms.

--a GUI (yea i know that sounds shitty to code, and it IS, but we're going to need a GUI coder, or at least a simple interface)
--some algorithms to start with (i think that's sorted already)
--a network architecture (c++) --Someone's going to be coding this (And i wouldn't mind doing so)
--a tracker architecture (LAMP)-- Yep, thats right, trackers. (And if you want SQLi bugs, yeah you should probably ask me to do this;) )

So essentially we'd need 2-3 coders for this. It's a BIG project tho, so we'll first have to see which code we can borrow (libraries;) )
The GUI could probably be made in c#, as long as the extensions are in c.
The network architecture could be added as an extension itself, plus speed is kind of essential, so i recommend c++ (c# would probably do too).
Tracker's by far the easiest, nothing special about it.

[edit]Also, a massively distributed cracker like this would make rainbow tables practically obsolete. [/edit]

why should rainbow tables be obsolete? what was the advantage of brute forcing again? heating your house?
if you make a distributed rainbow table you are still going to be x times faster than with distributed brute forcers.

[foobar2342]

i liked the idea better when the focus was on personal cracking networks with a possible future option for public online crackers. (btw, with peer2peer you also have the trouble that people should adjust their firewalls and such.)

his proposal is not p2p. peers never have to talk to each other. it is a client redundant-server architecture. and since
clients only connect to a server there is no need for firewall traversal.

[vampyr]

why should rainbow tables be obsolete? what was the advantage of brute forcing again? heating your house?
if you make a distributed rainbow table you are still going to be x times faster than with distributed brute forcers.

Apart from having a 1000 watt heater (4*487x2), it requires little cpu time on the client side, so more people could join. It requires little bandwidth on account of the peers. A multitude of hashes can be searched simultaneously, 4k being a fairly reasonable amount. Oh, that and that salted hashes can be cracked.
I somehow seriously doubt that making a distributed rainbow table is going to be faster, for a LOT of the searching will be done on one single server.

[edit] Also, private trackers would obviously be possible And making a LAN plugin using the extension mechanism probably even simpler. [/edit]

[foobar2342]

why should rainbow tables be obsolete? what was the advantage of brute forcing again? heating your house?
if you make a distributed rainbow table you are still going to be x times faster than with distributed brute forcers.

Apart from having a 1000 watt heater (4*487x2), it requires little cpu time on the client side, so more people could join. It requires little bandwidth on account of the peers. A multitude of hashes can be searched simultaneously, 4k being a fairly reasonable amount. Oh, that and that salted hashes can be cracked.
I somehow seriously doubt that making a distributed rainbow table is going to be faster, for a LOT of the searching will be done on one single server.

[edit] Also, private trackers would obviously be possible And making a LAN plugin using the extension mechanism probably even simpler. [/edit]

with a distributed rainbow table i mean that every client has a table with a different round function.
the amount of communication is obviously the same (hashes -> client -> plaintexts) and the burden on
the cpu is negligible (control the gpu and make some disk lookups).
With a rainbow table you can also search for many hashes simultaneously, so that argument does not count.
salted hashes are an argument indeed, but still this does not make rainbow tables obsolete since tables never
competed in this domain.

[vampyr]

why should rainbow tables be obsolete? what was the advantage of brute forcing again? heating your house?
if you make a distributed rainbow table you are still going to be x times faster than with distributed brute forcers.

Apart from having a 1000 watt heater (4*487x2), it requires little cpu time on the client side, so more people could join. It requires little bandwidth on account of the peers. A multitude of hashes can be searched simultaneously, 4k being a fairly reasonable amount. Oh, that and that salted hashes can be cracked.
I somehow seriously doubt that making a distributed rainbow table is going to be faster, for a LOT of the searching will be done on one single server.

[edit] Also, private trackers would obviously be possible And making a LAN plugin using the extension mechanism probably even simpler. [/edit]

with a distributed rainbow table i mean that every client has a table with a different round function.
the amount of communication is obviously the same (hashes -> client -> plaintexts) and the burden on
the cpu is negligible (control the gpu and make some disk lookups).
With a rainbow table you can also search for many hashes simultaneously, so that argument does not count.
salted hashes are an argument indeed, but still this does not make rainbow tables obsolete since tables never
competed in this domain.

Which sounds awesome, in theory, but in practice leads to a few concerns:
1: reliability. If say half the nodes are lost, would the network still function? What if a node and it's backup are lost? Surely not having the tables for those functions would impair the network's ability to find plaintexts.
2: As you need to send a request to EVERY node this way (if each node contains it's own table) traffic is indeed an issue.
3: Storage? I do not know many people willing to spare say up to 100gb of disk space on their private machines.
4: Checking? Who knows if the table the node has is corrupted. Certainly as your not storing the table on a central server, there is essentially no way of doing this. Yes you could salt with a set of random hashes, but rainbow tables aren't guaranteed to contain those.