<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter>
    <title>Cluster Component Usage</title>
    <sect1>
        <title>Synopsis</title>
        <para>The Cluster component allows submission of jobs to a cluster.</para>
    </sect1>
    <sect1>
        <title>Rationale</title>
        <para>The Cluster component submits jobs described by a given job script to the queuing
            system of a cluster. It allows to upload directories beforehand and to download
            directories after the job is terminated.</para>
        <para>To check if jobs are finished, the Cluster component polls the queuing system every
            minute and asks for their states. The connection to the cluster is established via SSH.
            For the submission, a directory (sandbox-[uuid]) is created on the cluster in the user’s
            home directory. It will serve as the current working directory for all remote command
            line calls.</para>
        <para>The remote directory structure is as
            follows:<programlisting>/sandbox-[id]
    /iteration-0
        /cluster-job-0
            /input
            /output
        /cluster-job-1
            /input
            /output
        …
        /cluster-job-shared-input
    /iteration-1
        /cluster-job-0
            /input
            /output
        …
        / cluster-job-shared-input
    …
    job </programlisting>The
            job script is uploaded to /sandbox-[id]/job. The job submission is done from
            /sandbox-[id]/iteration-[n]/cluster-job-[n]/.</para>
        <para>If the job failed and the Cluster component should be marked as failed, a file named
            job_failed must be created in /sandbox-[id]/iteration-[n]/cluster-job-[n]/output. The
            content of the file is used as error message. The output directories are not downloaded
            for the failed job and all remaining jobs terminated afterwards.</para>
    </sect1>
    <sect1>
        <title>Usage</title>
    <para>The Cluster component is configured as follows:</para>
                <sect2>
                    <title>Cluster tab</title>
                    <para> In the Cluster tab define the information needed to connect to a cluster
                        via SSH. Define host IP or resolvable host name, port number, etc. The
                        working directory root is the folder, where the sandbox folder mentioned
                        above is created. Also define the queuing system running on the cluster. In
                        some cases, the queuing system console commands like qsub, qstat, etc. are
                        not known within a non-interactive SSH shell on the cluster. For that, you
                        can optionally define the absolute paths to the required commands
                        explicitly. If you don’t know them, just type ‘which qsub’ etc. on a
                        cluster’s console and you will get them.</para>
                    <para><inlinegraphic fileref="../../images/image001.png"/></para>
                </sect2>
        <sect2>
            <title>Inputs/Outputs tab</title>
            <para>In the tab Inputs/Outputs you see the inputs and outputs of the Cluster
                component. The inputs and outputs are static and cannot be modified except
                the scheduling behavior.</para>
            <para>
                <orderedlist>
                    <listitem>
                        <para>Job count: The count of jobs to submit on each iteration</para>
                    </listitem>
                    <listitem>
                        <para>Job inputs: Input directories which are uploaded before each iteration
                            to /sandbox-[id]/iteration-n/cluster-job-n/input (in the order as they
                            arrive, 0 for first directory, 1 for second, etc.)</para>
                    </listitem>
                    <listitem>
                        <para>Shared job input: Input directory which is uploaded before
                            each iteration to /sandbox-[id]/iteration-n/shared-input</para>
                    </listitem>
                    <listitem>
                        <para>Job outputs: Output directory which is download after each
                            iteration from
                            sandbox-[id]/iteration-n/cluster-job-n/output</para>
                    </listitem>
                </orderedlist>
            </para>
            <para><inlinegraphic fileref="../../images/image003.png"/></para>
        </sect2>
                <sect2>
                    <title>Job tab</title>
                    <para>The job itself is either described in the Job tab or is provided within
                        each input directory (/sandbox-[id]/iteration-[#]/cluster-job-[#]/input).
                        Select the check box accordingly. If it is provided within each input
                        directory, the name of the script must be: run_cluster_job.sh</para>
                    <para><inlinegraphic fileref="../../images/image002.png"/></para>
            <para>To see the native standard out and error of the job submission see the Workflow
                console.</para>
        <para><inlinegraphic fileref="../../images/image004.png"/></para>
                </sect2>
    </sect1>
</chapter>
