1

Spreading stdin to parallel processes

view story
linux-howto

http://unix.stackexchange.com – I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of these processes, then dispatch work to whichever ones are not busy. There are several different commandline tools that almost do what I want, I've narrowed it down to two almost working options: find . -type f | split -n r/24 -u --filter="myjob" find . -type f | parallel --pipe -u -l 1 myjob The problem is that split does a pure round-robin, so one of the processes gets behind and stays behind (HowTos)