The Cluster component is configured as follows:
In the Cluster tab define the information needed to connect to a cluster via SSH. Define host IP or resolvable host name, port number, etc. The working directory root is the folder, where the sandbox folder mentioned above is created. Also define the queuing system running on the cluster. In some cases, the queuing system console commands like qsub, qstat, etc. are not known within a non-interactive SSH shell on the cluster. For that, you can optionally define the absolute paths to the required commands explicitly. If you don’t know them, just type ‘which qsub’ etc. on a cluster’s console and you will get them.
In the tab Inputs/Outputs you see the inputs and outputs of the Cluster component. The inputs and outputs are static and cannot be modified except the scheduling behavior.
-
Job count: The count of jobs to submit on each iteration
-
Job inputs: Input directories which are uploaded before each iteration to /sandbox-[id]/iteration-n/cluster-job-n/input (in the order as they arrive, 0 for first directory, 1 for second, etc.)
-
Shared job input: Input directory which is uploaded before each iteration to /sandbox-[id]/iteration-n/shared-input
-
Job outputs: Output directory which is download after each iteration from sandbox-[id]/iteration-n/cluster-job-n/output
The job itself is either described in the Job tab or is provided within each input directory (/sandbox-[id]/iteration-[#]/cluster-job-[#]/input). Select the check box accordingly. If it is provided within each input directory, the name of the script must be: run_cluster_job.sh
To see the native standard out and error of the job submission see the Workflow console.
