SSH Keys and multi machine processing
A lot of my processing can be parallelised effectively by just splitting up the data and running on many machines. To accomplish this I use SSH keys so I can start jobs over the network from one bash script. Works nicely and is quite effective – the only downside is normally having to send data over NFS.
Anyway, crucial to this is the ability to install SSH keys, this can be done quite easily.
Firstly make sure that you have ssh’d from the machines you want to set this up on before. Simply just ssh into another machine, this will create a .ssh in your home directory with the proper permissions.
Now on your main machine run:
ssh-keygen -t dsa
This will generate a key. It will ask you for a passphrase. If you are hoping to spawn loads of jobs over many machines then don’t put anything here (otherwise you’ll be asked for the passphrase instead of your password!). Of course this is a security risk – so make sure you trust your network etc.
Now copy this key over to the other machines, in this case I’m using a virtual machine at 192.168.56.101, so:
scp ~/.ssh/id_dsa.pub 192.168.56.101:.ssh/authorized_keys2
Now on the machine you just copied it over (in my case ssh 192.168.56.101) run:
ssh-agent sh -c ‘ssh-add < /dev/null && bash'
This will add the key and allow you to effectlviely have passwordless login – and allow for some nice multi-machine processing – we do this for lots of the Arecibo data we have to deal with, allows us to effectively run on 100 cores without any fancy software just a small bash script that loops over an array of machines.