This section describes the basics of workflows in RCE.
RCE is designed to execute automated, distributed workflows. Workflows consist of so called workflow components which can be coupled among each other. Loops are supported, even multi-nested ones.
To get started with workflows in RCE it is recommended to both read the following sections about workflows and walk through the example workflows provided in RCE. The sections here refer to the workflow examples where it is useful and vice versa.
Workflows in RCE are encapsulated in so called projects. To create the workflow examples project go to: File->New->Workflow Examples Project. A dialog appears. Leave the default project name or give it a name of your choice and confirm by clicking 'Finish'. In the 'Project Explorer' on the left-hand side, the newly created project is shown. The example workflows are grouped in sub folders. It is recommended to walk through the workflows following the prefix starting with '01_01_Hello_World.wf'.
Workflow components are either tools that are integrated by users or are provided by RCE supplying multi-purpose functionality. The following list shows workflow components provided by RCE grouped by purpose (workflow components that are deprecated (i.e., they are removed soon) or that are not recommended to use anymore are left out):
Data:Database
Data Flow:Input Provider, Output Writer, Joiner, Switch
Evaluation:Optimizer, Design of Experiments, Parametric Study, Converger, Evaluation Memory
Execution:Script, Cluster, Excel
XML:XML Loader, XML Merger, XML Values
CPACS:TiGL Viewer, VAMPzero Initializer
The example workflows in sub folder '02_Component Groups' introduce some of the workflow components provided. Additionally, there is a documentation for each workflow component available in RCE. To access it, you can either rightclick on a component in a workflow and select "Open Help" or press "F1". A help view opens on the right-hand side. Moreover there is an entry "Help Contents" in the "Help" menu where you can navigate to the component help you require.
The XML and CPACS components are able to read or extract data from an Xml-file via dynamic in- or outputs. The XPathChooser is a feature that provides help selecting the item, which shall be read or extracted. Add an in- or output and press the "XPath choosing..." button to open a window , where you can select the Xml-file, which contains the item that shall be selected. After choosing the file, the XPathChooser opens containing a tree, symbolizing the Xml-File. By selecting an element, the text below is updated and displays the current path. The last two columns are used to choose attributes. The attribute name can be selected via the column “Attributes”. In the column “Values”, the proper value can be selected. Use a double-click on an element to expand or fold the tree. The chosen XPath will be written in the text field of the window, in which the XPathChooser has been opened originally. Using this text field, new paths can be created. Add a slash and the name of the node that shall be created to the existing path. The new path will be added during the workflow run.
New XPaths can only be generated within the inputs tab. Using the outputs tab will cause an error.
A workflow component can send data to other workflow components. Therefore, a so called connection needs to be created between the sending workflow component and the receiving one. For that purpose, workflow components can have so called inputs and outputs. A connection is always created between an output and an input. You can think of a connection as a directed data channel. Data is sent as atomic packages which are not related to each other (there is no data streaming between workflow components). Supported data types are:
Primitive data types:
Short Text:A short text (up to 140 characters)
Integer:Integer number
Float:Floating point number
Boolean:Boolean value (true or false)
Referenced data types (The actual data is stored in RCE's data management and only a reference is transfered):
File:File
Directory:Directory
Other data types
Small Table:Table restricted to values of type Short Text, Integer, Float, Boolean (primitive data types) (up to 100.000 cells)
Vector:one-dimensional "Small Table" (one column) restricted to values of type Float
Matrix:Small Table restricted to values of type Float
Not all of the workflow components support all of the data types listed. A connection can be created between an output and an input if:
The data type of the output is the same as or convertible to the data type of the input.
The input is not already connected to another output.
Note that data from an output can be sent to multiple inputs, but an input can just receive data from a single output.
The following table shows which data types are convertible to which other types:
Table 3.1. Data Type Conversion Table
| To | Boolean | Integer | Float | Vector | Matrix | Small Table | Short Text | File | Directory |
| From | |||||||||
| Boolean | x | x | x | x | x | ||||
| Integer | x | x | x | x | |||||
| Float | x | x | x | ||||||
| Vector | x | x | |||||||
| Matrix | x | ||||||||
| Small Table | |||||||||
| Short Text | x | ||||||||
| File | |||||||||
| Directory |
The execution of workflows is data-driven. As soon as all of the desired input data is available, a workflow component will be executed. Which input data is desired, is defined by the component developer (for RCE's default workflow components), the tool integrator, and/or the workflow creator. The workflow component developer and tool integrator decide which options are allowed for a particular workflow component. The workflow creator can choose between those options at workflow design time. The following options exist:
Input handling:
Constant: The value won't be consumed during execution and will be reused in the next iteration (if there is any loop in the workflow). The workflow will fail if there is more than one value received, except for nested loops: All inputs of type Constant are resetted within nested loops, after the nested loop has been finished.
Single (Consumed): The input value will be consumed during execution and won't be reused in the next iteration (if there is any loop in the workflow). Queuing of input values is not allowed. If another value is received before the current one was consumed, the workflow will fail. This can guard against workflow design errors. E.g., an optimizer must not receive more than one value at one single input within one iteration.
Queue (Consumed): The input value will be consumed during execution and won't be reused in the next iteration (if there is any loop in the workflow). Queuing of input values is allowed.
Execution constraint:
Required: The input value is required for execution. Thus, the input must be connected to an output.
Required if connected: The input value is not required for execution (e.g., if a default value will be used as fall back within the component). Thus, the input doesn't need to be connected to an output. But if it is connected to an output, it will be handled as an input of type Required.
Not required: The input value is not required for execution. Thus, the input doesn't need to be connected to an output. If it is connected to an output, the input value will be passed to the component if there is a value available at the time of execution. Values at inputs of type Not required cannot trigger component execution except if it is the only input defined for a component. Note: With this option, non-deterministic workflows can be easily created. Use this option carefully. If in doubt, leave it out.
Note: With RCE 6.0.0 the scheduling options changed. Below is the migration path:
Initial was migrated to Constant and Required.
Required was migrated to Single (Consumed) and Required.
Optional was migrated to Single (Consumed) and Required if connected.
If you encounter any problems with workflows created before RCE 6.0.0, it is likely, that it affects the migration to Single (Consumed) instead of to Queue (Consumed). We decided to migrate conservatively to not hide any existing workflow design errors. So, if queuing of input values is allowed for an input, just change the input handling option to Queue (Consumed) after the workflow was updated. Another issue can affect the migration of Optional. If it affects an input of the script component, check the option, which let the script component execute on each new value at any of its inputs. Also check Not required as an alternative execution constraint option.
Workflow components can be coupled to loops. A loop must always contain a so called driver workflow component. Driver workflow components (group "Evaluation") are: Optimizer, Design of Experiments, Parametric Study, Converger (see the example workflow "02_02_Evaluation_Drivers"). The responsibilities of a driver workflow component in a loop are:
Send values to the loop and receive the result values.
Finish the loop based on some certain criteria.
If a loop contains another loop, we speak of the latter as a nested loop. A nested loop can contain again another loop and so on. To create workflows with nested loops (see example workflows in "03_Workflow_Logic"), some certain concepts behind nested loops must be understood:
Loop level: If a loop contains another loop, that loop is considered as a nested loop with a lower loop level. From the perspective of the nested loop, the other loop is considered as a loop with an upper loop level.
If a driver workflow component is part of a nested loop, you need to check the checkbox in the "Nested Loop" configuration tab
Data exchange between loops of different loop levels is only allowed via a driver workflow component. Thereby, only particular inputs and outputs of driver workflow components are allowed to be connected to inputs and outputs of the next upper loop level and particular ones to inputs and outputs of the same loop level. For example, if a 'same loop level' output is connected to a loop with an upper loop level, the workflow won't succeed or might even get stuck. Below you find tables of inputs and outputs for each driver workflow component and whether they must be connected to the same loop level or to the next upper loop level. (Note: In the inputs and outputs tables of driver workflow components (in 'Inputs/Outputs' properties tab), the loop level requirement is present in a particular column for each input and ouptut.)
Table 3.2. Inputs of Optimizer
| Input | Loop Level |
|---|---|
| * - lower bounds - start value | To next upper loop level |
| * - upper bounds - start value | To next upper loop level |
| * - start value | To next upper loop level |
| * (Objective functions) | To same loop level |
| * (Constraints) | To same loop level |
| d*.d* (Gradients) | To same loop level |
Table 3.3. Outputs of Optimizer
| Output | Loop Level |
|---|---|
| *_optimal | To next upper loop level |
| Done | To next upper loop level |
| * (Design variables) | To same loop level |
| Gradient request | To same loop level |
| Iteration | To same loop level |
Table 3.4. Inputs of Design of Experiments
| Input | Loop Level |
|---|---|
| *_start | To next upper loop level |
| * | To same loop level |
Table 3.5. Outputs of Design of Experiments
| Output | Loop Level |
|---|---|
| Done | To same loop level |
| * | To same loop level |
Table 3.6. Inputs of Parametric Study
| Input | Loop Level |
|---|---|
| *_start | To next upper loop level |
| * | To same loop level |
Table 3.9. Outputs of Converger
| Output | Loop Level |
|---|---|
| Converged | To next upper loop level |
| Converged absolute | To next upper loop level |
| Converged relative | To next upper loop level |
| *_converged | To next upper loop level |
| Done | To same loop level |
| * | To same loop level |
Workflow components of a loop can fail. There are two kind of failures:
A workflow component fails gracefully, i.e. it couldn't compute any results for the inputs received but works normally. In this case, it sends a value of type "not-a-value" with the specified cause to its outputs which finally are received by the driver workflow components as results.
A workflow component fails, i.e. it crashes for an unexpected reason. In this case, the workflow engine sends values of type "not-a-value" with the specified cause as results to the driver workflow component.
In the "Fault Tolerance" configuration tab of workflow driver components, it can be configured how to handle failures in loops, for both kind of failures separately.
After the execution of an integrated tool, the results are sent via outputs to the next workflow component (e.g. to the next integrated tool). By default, this is done in an automated manner without any user interaction. If the data should be verified by a person responsible for the tool before they are sent further, manual verification of tool results must be enabled in the tool integration wizard in the 'Verification' tab of the 'Inputs and Outputs' page.
In case manual verification of tool results is enabled, the results are hold after each tool execution and the corresponding workflow component remains in state "Waiting for approval". Then, there are two options:
Approve tool results: The tool results are sent via the outputs to the next workflow component and the workflow continues normally.
Reject tool results: The tool results are not sent via the outputs to the next workflow component and the workflow is cancelled.
To apply one of the options, a so called verfication key is required. The verification key is generated by RCE after each tool execution and is written to a file on the file system of the machine which executed the tool. (The location is specified in the 'Verification' tab of the 'Inputs and Outputs' page in the tool integration wizard.) Optionally, the verification key can also be sent via e-mail if e-mail support is configured for the RCE instance where the tool is integrated. (E-mail support can only be configured using the Configuration UI as described in Section 2.3.2, “Mail: SMTP server configuration”) E-mail delivery can be enabled and the recipients can be defined in the 'Verification' tab of the 'Inputs and Outputs' page in the tool integration wizard.
Once the verification key is known (either from the file or an e-mail), perform follwing steps to approve or reject the tools results:
Start an RCE instance with a graphical user interface. (Your tool must be available, i.e. it must appear in the palette of the workflow editor.)
In the menu bar at the top, go to Run -> Verify tool results...
A dialog appears that guides you through the verification process.