Cryptohaze on EC2
Contents |
Why?
The GPUs in the EC2 GPU instances are very slow compared to modern AMD GPUs. However, they're fairly cheap to run, especially in spot instances, and you can have them in 10 minutes - not the few days it takes to build a system. People keep showing interest in this, and it would be silly of me to deny people what they want!
I'm not using an AMI (Amazon Machine Image). I thought about it, and realized that it was going to be too difficult to keep updated. Especially with the new multiforcer, it's in active development. So, I'm moving to a new method that works well with Amazon's cloud setup.
Instead of doing this, I'm providing a script that can be copied into the user-data section of the EC2 instance setup that will automatically configure the host and optionally start the New Multiforcer pointing at a remote server (for, say, spot instance use).
If you haven't set instances up in a while, please come get a new copy of the script. It's likely to have URLs changing at various points in time.
The Quick Steps
I'm going to assume a slight familarity with EC2 instances, login keys, and security groups here. If you're not familiar with these, feel free to add info on them. I know my way around them, so I'll just assume other people do too - they're EC2 tech, not my stuff.
- Log into EC2. Bring up your EC2 Instances page
- Click "Launch Instance"
- Select the "Classic Wizard"
- Scroll down to the "Cluster GPU Amazon Linux AMI 2012.03" (or whatever the current Cluster GPU Linux is) and select it
- For "Instance Type", select "Cluster GPU (cg1.4xlarge)" - this is the GPU instance you probably want.
- If you want spot instances, this is where you select them. I do support spot instances!
- The Advanced Options is where you paste the script from below. BE SURE TO CHANGE YOUR REMOTE HOSTNAME! You can also change the shutdown behavior to "Terminate" if this is a spot instance or remote only node. This will destroy the whole instance when it shuts down, which is fine for a pure remote compute node.
- The default storage config is fine.
- Name your instances if you so desire - add a value to the "Name" key line.
- Either pick an existing key pair or choose a new one.
- Choose the quicklaunch security group, or create a new one. It just needs to let SSH in and other traffic out.
- If everything looks good, go ahead & launch the instance!
It may take a few minutes for the instance to start, install the packages, and connect. It also seems to take roughly forever for the nVidia cards to come online. If you see rates of a few hundred million per second, this is because the GPUs have not yet chimed in. I think Amazon is doing something funky with virtual device routing, or otherwise making it take far longer than usual to connect to the GPUs.
Debugging
Once an instance has started, you can connect to it as normal.
The startup script log is in /var/log/cloud-init.log - you can watch it, and if it's not gotten to the Multiforcer yet, it's still installing. It typically takes 3-5 minutes after boot to finish installing everything.
It may take much longer for the nVidia GPUs to chime in. I don't know why - thoughts?
Stopping the Multiforcer
If it's running, and you want to stop it (and the wrapper script that will keep it running), log in and then:
sudo killall -9 part-000 New-Multiforcer
Note that if you just turn the instance off, without killing the Multiforcer, a hanging TCP connection will stick around for a long time. This is a known bug, and I'll fix it at some point.
The Script
Copy, paste, CHANGE YOUR HOSTNAME. If you don't want it to launch the multiforcer, you can change RUN_REMOTE to false and then log in.
Everything is installed in /root - the Cryptohaze suite lives in /root/Cryptohaze-Build-Latest and you should be able to run everything. If not, it's a bug, and please tell me.
#!/bin/bash REMOTE_HOST=freecracking.cryptohaze.com RUN_REMOTE=true cd /root sed -i '/Defaults*[requiretty,visiblepw]/ s/^/#/' /etc/sudoers wget http://prdownloads.sourceforge.net/argtable/argtable2-13.tar.gz tar -zxvf argtable2-13.tar.gz cd argtable2-13 ./configure && make -j 24 && make install cd .. wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.bz2 tar -xvjf protobuf-2.4.1.tar.bz2 cd protobuf-2.4.1 ./configure && make -j 24 && make install cd .. echo "/usr/local/lib" > /tmp/cryptohaze.conf mv /tmp/cryptohaze.conf /etc/ld.so.conf.d/ ldconfig wget http://us.download.nvidia.com/XFree86/Linux-x86_64/302.17/NVIDIA-Linux-x86_64-302.17.run chmod 755 NVIDIA-* yum install kernel-devel-`uname -r` kernel-headers-`uname -r` -y ./NVIDIA-Linux-x86_64-302.17.run -s wget http://developer.amd.com/Downloads/AMD-APP-SDK-v2.7-lnx64.tgz tar -zxvf AMD-APP-SDK-v2.7-lnx64.tgz ./Install-AMD-APP.sh wget https://www.cryptohaze.com/ec2/Cryptohaze-Build-Latest.tar.bz2 tar -xvjf Cryptohaze-Build-Latest.tar.bz2 cd Cryptohaze-Build-Latest while $RUN_REMOTE; do rmmod nvidia modprobe nvidia ./New-Multiforcer --nocpu --remoteip=$REMOTE_HOST --daemon done
The script, commented
This is a commented walkthrough, for the few of you who don't quite trust copy-paste. Realistically, you're downloading a binary from my site as root and running it, so... if you don't trust me, this section is probably useless. However, it's also comments for my future use.
Initial setup. This will run as a cloud-init script when run in user data. It will also point people who don't change the hostname
#!/bin/bash REMOTE_HOST=freecracking.cryptohaze.com RUN_REMOTE=true
Change to /root (as we start in / ), and disable the "sudo requires a tty" flags. These interfere with the AMD APP SDK install script - it does some stuff with sudo that I don't care to weed out. I simply comment out two lines in /etc/sudoers here, in place.
cd /root sed -i '/Defaults*[requiretty,visiblepw]/ s/^/#/' /etc/sudoers
Install argtable2. There are no packages for it for some reason, and we just build from source and install.
wget http://prdownloads.sourceforge.net/argtable/argtable2-13.tar.gz tar -zxvf argtable2-13.tar.gz cd argtable2-13 ./configure && make -j 24 && make install cd ..
Install Protobufs. Same as the above.
wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.bz2 tar -xvjf protobuf-2.4.1.tar.bz2 cd protobuf-2.4.1 ./configure && make -j 24 && make install cd ..
Both argtable and protobuf install into /usr/local/lib - this is not part of the default ld path. Add a config file to the ld path and update the loader.
echo "/usr/local/lib" > /tmp/cryptohaze.conf mv /tmp/cryptohaze.conf /etc/ld.so.conf.d/ ldconfig
I live on the bleeding edge (a quarter mile at a time?) and use the latest nVidia drivers - not the long term ones. To install the nVidia drivers, I need kernel sources.
wget http://us.download.nvidia.com/XFree86/Linux-x86_64/302.17/NVIDIA-Linux-x86_64-302.17.run chmod 755 NVIDIA-* yum install kernel-devel-`uname -r` kernel-headers-`uname -r` -y ./NVIDIA-Linux-x86_64-302.17.run -s
To effectively use the CPU, I want to treat it as an OpenCL device. On the GPU instances, the AMD SDK works best (and installs as OpenCL platform 0). Download and install it. This installer script is the reason for the sudo muckery above.
wget http://developer.amd.com/Downloads/AMD-APP-SDK-v2.7-lnx64.tgz tar -zxvf AMD-APP-SDK-v2.7-lnx64.tgz ./Install-AMD-APP.sh
Download the latest Cryptohaze build (this is probably more recent than what's in SVN, and is built for the EC2 nodes - cutting edge stuff!)
wget https://www.cryptohaze.com/ec2/Cryptohaze-Build-Latest.tar.bz2 tar -xvjf Cryptohaze-Build-Latest.tar.bz2 cd Cryptohaze-Build-Latest
Finally, if instructed to do so, run the Multiforcer. As things do occasionally crash in GPU land, this is a tight loop that will just relaunch it. I'm not sure if removing the nVidia module and reinserting it actually does anything, but it seemed to clear up some weird errors I was having where only one GPU would show up. Alternately, that was a virtualization glitch.
while $RUN_REMOTE; do rmmod nvidia modprobe nvidia ./New-Multiforcer --nocpu --remoteip=$REMOTE_HOST --daemon done
Enjoy!
