\input texinfo
@setfilename Learn_GNU_Parallel_in_5_minutes.info

@documentencoding utf-8

@settitle Learn GNU Parallel in 5 minutes

@node Top
@top Learn GNU Parallel in 5 minutes

@menu
* Learn GNU Parallel in 5 minutes::
* Learn GNU Parallel in an hour::
* Advanced usage::
@end menu

@node Learn GNU Parallel in 5 minutes
@chapter Learn GNU Parallel in 5 minutes

You just need to run commands in parallel. You do not care about fine
tuning.

To get going please run this to make some example files:

@verbatim
  # If your system does not have 'seq', we will use 'jot' instead
  if ! seq 1 2>/dev/null; then alias seq=jot; fi

  seq 5 | parallel 'seq {} > example.{}'
@end verbatim

@menu
* Input sources::
* Building the command line::
* Controlling the output::
* Controlling the execution::
* Pipe mode::
* That's it::
@end menu

@node Input sources
@section Input sources

GNU @strong{parallel} reads values from input sources. One input source is
the command line. The values are put after @strong{:::} :

@verbatim
  parallel echo ::: 1 2 3 4 5
@end verbatim

This makes it easy to run the same program on some files:

@verbatim
  parallel wc ::: example.*
@end verbatim

If you give multiple @strong{:::}s, GNU @strong{parallel} will make all combinations:

@verbatim
  parallel wc ::: -l -c ::: example.*
@end verbatim

GNU @strong{parallel} can also read the values from stdin (standard input):

@verbatim
  seq 5 | parallel echo
@end verbatim

@node Building the command line
@section Building the command line

The command line is put before the @strong{:::}. It can contain contain a
command and options for the command:

@verbatim
  parallel wc -l ::: example.*
@end verbatim

The command can contain multiple programs. Just remember to quote
characters that are interpreted by the shell (such as @strong{;}):

@verbatim
  parallel echo counting lines';' wc -l ::: example.*
@end verbatim

The value will normally be appended to the command, but can be placed
anywhere by using the replacement string @strong{@{@}}:

@verbatim
  parallel echo counting {}';' wc -l {} ::: example.*
@end verbatim

When using multiple input sources you use the positional replacement
strings:

@verbatim
  parallel echo count {1} in {2}';' wc {1} {2} ::: -l -c ::: example.*
@end verbatim

@node Controlling the output
@section Controlling the output

The output will be printed as soon as the command completes. This
means the output may come in a different order than the input:

@verbatim
  parallel sleep {}';' echo {} done ::: 5 4 3 2 1
@end verbatim

You can force GNU @strong{parallel} to print in the order of the values with
@strong{--keep-order}/@strong{-k}. This will still run the commands in parallel.
The output of the later jobs will be delayed, until the earlier jobs
are printed:

@verbatim
  parallel -k sleep {}';' echo {} done ::: 5 4 3 2 1
@end verbatim

@node Controlling the execution
@section Controlling the execution

If your jobs are compute intensive, you will most likely run one job
for each core in the system. This is the default for GNU @strong{parallel}.

But sometimes you want more jobs running. You control the number of
job slots with @strong{-j}. Give @strong{-j} the number of jobs to run in
parallel:

@verbatim
  parallel -j50 \
    wget http://ftpmirror.gnu.org/parallel/parallel-{1}{2}22.tar.bz2 \
    ::: 2012 2013 2014 2015 2016 \
    ::: 01 02 03 04 05 06 07 08 09 10 11 12
@end verbatim

@node Pipe mode
@section Pipe mode

GNU @strong{parallel} can also pass blocks of data to commands on stdin
(standard input):

@verbatim
  seq 1000000 | parallel --pipe wc
@end verbatim

This can be used to process big text files. By default GNU @strong{parallel}
splits on \n (newline) and passes a block of around 1 MB to each job.

@node That's it
@section That's it

You have now learned the basic use of GNU @strong{parallel}. This will
probably cover most cases of your use of GNU @strong{parallel}.

The rest of this document will go into more details on each of the
sections and cover special use cases.

@node Learn GNU Parallel in an hour
@chapter Learn GNU Parallel in an hour

In this part we will dive deeper into what you learned in the first 5 minutes.

To get going please run this to make some example files:

@verbatim
  seq 6 > seq6
  seq 6 -1 1 > seq-6
@end verbatim

@menu
* Input sources 1::
* Building the command line 1::
* Copying environment::
* Controlling the output 1::
* Controlling the execution 1::
* Remote execution::
* Pipe mode 1::
* That's it 1::
@end menu

@node Input sources 1
@section Input sources

On top of the command line, input sources can also be stdin (standard
input or '-'), files and fifos and they can be mixed. Files are given
after @strong{-a} or @strong{::::}. So these all do the same:

@verbatim
  parallel echo Dice1={1} Dice2={2} ::: 1 2 3 4 5 6 ::: 6 5 4 3 2 1
  parallel echo Dice1={1} Dice2={2} ::: <(seq 6) ::: <(seq 6 -1 1)
  parallel echo Dice1={1} Dice2={2} :::: seq6 seq-6
  parallel -a seq6 -a seq-6 echo Dice1={1} Dice2={2}
  parallel -a seq6 echo Dice1={1} Dice2={2} :::: seq-6
  parallel echo Dice1={1} Dice2={2} ::: 1 2 3 4 5 6 :::: seq-6
  cat seq-6 | parallel echo Dice1={1} Dice2={2} :::: seq6 -
@end verbatim

If stdin (standard input) is the only input source, you do not need the '-':

@verbatim
  cat seq6 | parallel echo Dice1={1}
@end verbatim

You can link multiple input sources with @strong{:::+} and @strong{::::+}:

@verbatim
  parallel echo {1}={2} ::: I II III IV V VI :::+ 1 2 3 4 5 6
  parallel echo {1}={2} ::: I II III IV V VI ::::+ seq6
@end verbatim

@node Building the command line 1
@section Building the command line

@menu
* The command::
@end menu

@node The command
@subsection The command

The command can be a script, a binary or a Bash function if the
function is exported using @strong{export -f}:

@verbatim
  # Works only in Bash
  my_func() {
    echo in my_func "$1"
  }
  export -f my_func
  parallel my_func ::: 1 2 3
@end verbatim

@node Copying environment
@section Copying environment

env_parallel

@menu
* The replacement strings::
@end menu

@node The replacement strings
@subsection The replacement strings

GNU @strong{parallel} has some replacement strings to make it easier

@node Controlling the output 1
@section Controlling the output

@menu
* parset::
@end menu

@node parset
@subsection parset

@strong{parset} is a shell function to get the output from GNU @strong{parallel}
into shell variables.

@strong{parset} is fully supported for @strong{Bash/Zsh/Ksh} and partially supported
for @strong{ash/dash}. I will assume you run @strong{Bash}.

To activate @strong{parset} you have to run:

@verbatim
  . `which env_parallel.bash`
@end verbatim

(replace @strong{bash} with your shell's name).

Then you can run:

@verbatim
  parset a,b,c seq ::: 4 5 6
  echo "$c"
@end verbatim

or:

@verbatim
  parset 'a b c' seq ::: 4 5 6
  echo "$c"
@end verbatim

If you give a single variable, this will become an array:

@verbatim
  parset arr seq ::: 4 5 6
  echo "${arr[1]}"
@end verbatim

@strong{parset} has one limitation: If it reads from a pipe, the output will
be lost.

@verbatim
  echo This will not work | parset myarr echo
  echo Nothing: "${myarr[*]}"
@end verbatim

Instead you can do this:

@verbatim
  echo This will work > tempfile
  parset myarr echo < tempfile
  echo ${myarr[*]}
@end verbatim

sql
cvs

@node Controlling the execution 1
@section Controlling the execution

--dryrun -v

@node Remote execution
@section Remote execution

For this section you must have @strong{ssh} access with no password to 2
servers: @strong{$server1} and @strong{$server2}.

@verbatim
  server1=server.example.com
  server2=server2.example.net
@end verbatim

So you must be able to do this:

@verbatim
  ssh $server1 echo works
  ssh $server2 echo works
@end verbatim

It can be setup by running 'ssh-keygen -t dsa; ssh-copy-id $server1'
and using an empty passphrase. Or you can use @strong{ssh-agent}.

@menu
* Workers::
* --transferfile::
* --return::
* --cleanup::
@end menu

@node Workers
@subsection Workers

@node --transferfile
@subsection --transferfile

@strong{--transferfile} @emph{filename} will transfer @emph{filename} to the
worker. @emph{filename} can contain a replacement string:

@verbatim
  parallel -S $server1,$server2 --transferfile {} wc ::: example.*
  parallel -S $server1,$server2 --transferfile {2} \
     echo count {1} in {2}';' wc {1} {2} ::: -l -c ::: example.*
@end verbatim

A shorthand for @strong{--transferfile @{@}} is @strong{--transfer}.

@node --return
@subsection --return

@node --cleanup
@subsection --cleanup

A shorthand for @strong{--transfer --return @{@} --cleanup} is @strong{--trc @{@}}.

@node Pipe mode 1
@section Pipe mode

--pipepart

@node That's it 1
@section That's it

@node Advanced usage
@chapter Advanced usage

parset fifo, cmd substtition, arrayelements, array with var names and cmds, env_parset

env_parallel

Interfacing with R.

Interfacing with JSON/jq

4dl() @{
  board="$(printf -- '%s' "$@{1@}" | cut -d '/' -f4)"
  thread="$(printf -- '%s' "$@{1@}" | cut -d '/' -f6)"
  wget -qO- "https://a.4cdn.org/$@{board@}/thread/$@{thread@}.json" |
    jq -r '
      .posts
      | map(select(.tim != null))
      | map((.tim | tostring) + .ext)
      | map("https://i.4cdn.org/'"$@{board@}"'/"+.)[]
    ' |
      parallel --gnu -j 0 wget -nv
@}

Interfacing with XML/?

Interfacing with HTML/?

@menu
* Controlling the execution 2::
* Remote execution 1::
@end menu

@node Controlling the execution 2
@section Controlling the execution

--termseq

@node Remote execution 1
@section Remote execution

seq 10 | parallel --sshlogin 'ssh -i "key.pem" a@@b.com' echo 

seq 10 | PARALLLEL_SSH='ssh -i "key.pem"' parallel --sshlogin a@@b.com echo 

seq 10 | parallel --ssh 'ssh -i "key.pem"' --sshlogin a@@b.com echo 

ssh-agent

The sshlogin file format

Check if servers are up

@bye
