1

Running parallel bash jobs on a HPC cluster using GNU parallel

view story
linux-howto

http://unix.stackexchange.com – On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck doing the other. permute1.sh: PROCS=144 permuations=1000 seq 1 $permuations | parallel -j $PROCS sh permute2.sh {} permute2.sh (taking 100 random lines from a file and performs some actions on it for permutation) id=$1 randomlines=100 awk 'BEGIN{srand();} {a[NR]=$0} END{for(I=1;I<='$randomlines';I++){x=int(rand()*NR);print a[x];}}' FILE.txt > results/randomlines.$id.txt # do stuff wit (HowTos)