<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<section id="tool_integration">
    <title>User-Defined Components</title>
    <para>RCE comes with a number of components that already allow you to create rather large and
        complex workflows. It is technically possible to construct a workflow using only the
        components that come with RCE, since you can call external tools via the Script-component.
        Maintaining such a workflow will, however, prove quite cumbersome. Moreover, every instance
        of the Script-component in an RCE workflow must be configured individually, since there
        exists no possibility to share configuration between component instances. Finally, while
        components can be shared among a network of RCE instances (cf. <xref
            linkend="publishing_authorization"/>) this sharing does not extend to configuration
        values such as the script of a Script-component. To simplify the construction, sharing, and
        maintenance of workflows containing calls to external tools, RCE allows for the integration
        of such external tools as user-defined components.</para>
    <section>
        <title>Integrating External Tools as Components</title>
        <para>If you want to integrate an external tool, the tool must</para>
        <itemizedlist>
            <listitem>
                <para>be callable via command line,</para>
            </listitem>
            <listitem>
                <para>have a non-interactive mode which is called via command line, and</para>
            </listitem>
            <listitem>
                <para>have its input provided through environment variables, command line arguments,
                    or files</para>
            </listitem>
        </itemizedlist>
        <para>If these requirements are fulfilled, a tool can be integrated into RCE. An
                <emphasis>integration file</emphasis> describes the "interface" of the tool to RCE.
            This interface consists of, among others, its inputs, outputs, as well as how the tool
            is executed. The integration file can be found in the profile directory (cf. <xref
                linkend="configuration"/>) in the subdirectory <code>integration/common/&lt;tool
                name></code>, where <code>&lt;tool name></code> is the name under which your tool
            will be available as a component in RCE. You can find an example of such an integration
            file by importing the Workflow Examples Project (via <emphasis>File -> New -> Workflow
                Examples Project</emphasis>) and opting to integrate an example tool during the
            import.</para>
        <para>If you use RCE with a graphical user interface you can integrate a tool via a wizard
            which guides you through the settings. This wizard can be found in the menu <emphasis
                role="italic">Tool Integration -> Integrate Tool...</emphasis>. Required fields are
            marked with an asterisk (*). When the wizard is finished and if everything is correct,
            the integrated tool will automatically show up in the Workflow Editor palette.</para>
        <para>
            <note>
                <para>The wizard has a dynamic help, which is shown by clicking on the question mark
                    on the bottom left or by pressing F1. It will guide you through the pages of the
                    wizard.</para>
            </note>
        </para>
        <para>One major part of the tool integration consists of the definition of a
                <emphasis>pre-</emphasis>, an <emphasis>execution-</emphasis>, and a <emphasis>post
                script</emphasis>. Pre- and post script define how incoming data from RCE will be
            passed to the tool, and how outgoing data from the tool will be passed back to RCE,
            respectively. These scripts are written in Python and executed using Jython, a Java
            implementation of Python 2. Please make sure that your pre- and post scripts have the
            desired behavior under this version of Python. (cf. <xref
                linkend="tool_integration_known_issues"/>) . The execution script determines how the
            tool is called and is given in either cmd or sh, depending on whether the tool is
            executed on Windows or Linux.</para>
        <para>During integration, you can specify tool properties. These can be used to, e.g.,
            switch between different execution modes of a tool, such as fast or precise computation.
            While the use of properties allows the eventual user of the component a great amount of
            flexibility, they also easily lead to inadvertent security issues. Consider, e.g., a
            tool that copies some data to a configurable directory and removes that directory after
            its computation as part of cleanup. Malicious users may set the configuration directory
            to <code><phrase os="windows">C:\Windows</phrase><phrase os="linux">/</phrase></code>
            and cause the tool to remove vital system directories on termination. To prevent users
            from creating such security issues by accident, RCE does not allow the use of property
            values containing <code>\"</code>, ASCII-characters in the range
                <code>0x00</code>-<code>0x1f</code>, <code>\\</code>, <code>/</code>,
                <code>\*</code>, <code>\?</code><phrase os="windows">, or
                <code>%</code></phrase><phrase os="linux">, <code>`</code>, or
                <code>\$</code></phrase>.</para>
        <para>If you would like to allow your users to supply a configuration file or a
            configuration directory, please add this file as an explicit input to the component.
            Other options include, e.g., fixing a "whitelist" of safe configuration options and
            allowing the user a choice of these configuration options via properties.</para>
        <section>
            <title>Directory Structure for Integrated Tools</title>
            <para>When executing an integrated tool, a certain directory structure is created in the
                chosen working directory. This structure depends on the options you have chosen in
                the integration wizard. The two options that matter are "Use a new working directory
                each run" and "Tool copying behavior".</para>
            <para><inlinegraphic fileref="images/Folderstructure.png" width="400px"/></para>
            <para><emphasis role="bold">Root Working Directory</emphasis>: This is the directory you
                choose in the "Tool Integration Wizard" as "Working Directory" on the "Launch
                Settings" page.</para>
            <para><emphasis role="bold">Config Directory</emphasis>: In this directory, the
                configuration file that may be created by the tool integration will be created by
                default. The configuration files can be created from the properties that are defined
                for the tool on the "Tool Properties" page.</para>
            <para><emphasis role="bold">Input Directory</emphasis>: All inputs of type "File" and
                "Directory" will be copied here. They will have a subdirectory that has the same
                name as the name of the input (e.g. the input "x" of type "File" will be put into
                "Input Directory/x/filename").</para>
            <para><emphasis role="bold">Output Directory</emphasis>: All outputs of type "File" and
                "Directory" can be written into this directory. After that, you can use the
                placeholder for this directory to assign these outputs to RCE outputs in the post
                execution script. To write, e.g., the output directory into an output "x" of type
                "Directory" the following line in the post execution script would be required:
                    <code>${out:x} = "${dir:output}"</code></para>
            <para><emphasis role="bold">Tool Directory</emphasis>: This is the directory where the
                actual tool is located. If the tool should not be copied, it will be exactly the
                same directory that you choose, otherwise it will be the same as the chosen
                directory but copied to the working directory.</para>
            <para><emphasis role="bold">Working Directory</emphasis>: A working directory is always
                the location, where all the other directories will be created. If the option "Use a
                new working directory on each run" is disabled, this will always be the same as the
                "Root Working Directory". Otherwise, a new directory is created each run (the name
                will be the run number) and is the working directory for the run.</para>
        </section>
        <section>
            <title>Copying of Integrated Tools</title>
            <para>When a component is created in the integration wizard, a configuration file is
                created.</para>
            <para>All configuration files from the tool integration are stored in the directory
                    <code>&lt;profile folder>/integration/tools/</code></para>
            <para>In this directory, there is a separation between different kinds of integration
                realized through one subdirectory for each. The <code>common</code> folder always
                exists.</para>
            <para>In these subdirectories, the integrated tools are stored, again separated through
                into a subdirectory for each. The name of the directory is the name of integration
                of the tool. </para>
            <para>If an integrated tool is copied to another RCE instance or another machine, the
                directory of the tool must be copied, containing a <code>configuration.json</code>
                and some optional files. It must be put in the equivalent integration type directory
                of the target RCE instance. After that, RCE automatically reads the new folder and
                if everything is valid, the tool will be integrated right away.</para>
            <para>
                <note>
                    <para>If you want to delete a tool folder that contains some documentation, this
                        can cause an error. If you have this problem, first empty the documentation
                        folder and delete the empty folder the documentation folder at first (it
                        must be empty), afterwards you can delete the tool folder.</para>
                </note>
            </para>
            <section>
                <title>Tool Execution Return Codes</title>
                <para>The tools are executed by using a command line call on the operating system
                    via the execution script. When the tool finished executing (with or without
                    error), its exit code is handed back to the execution script and can be analyzed
                    in this script. If in the script nothing else is done, the exit code is handed
                    back to RCE. When there is an exit code that is not "0", RCE assumes that the
                    tool crashed and thus lets the component crash without executing the post
                    script. Using the option "Exit codes other than 0 is not an error" prevents the
                    component from crashing immediately. With this option enabled, the post script
                    wil be executed in any way and the exit code from the tool execution can be read
                    by using the placeholder from <emphasis>Additional Properties</emphasis>. In
                    this case, the post script can run any post processing and either not fail the
                    component, so the workflow runs as normal, or let the component crash after some
                    debugging information was written using the Script API
                        <code>RCE.fail("reason")</code>.</para>
            </section>
        </section>
        <section>
            <title>Integration of CPACS Tools</title>
            <section>
                <title>Additional concepts of CPACS Tool Integration</title>
                <para>Extending the common Tool Integration concept, the CPACS Tool Integration has
                    some additional features.<itemizedlist>
                        <listitem>
                            <para>Parameter Input Mapping (optional): Substitutes single values in
                                the incoming CPACS content, based on an XPath configured at workflow
                                design time as a dynamic input of the component</para>
                        </listitem>
                        <listitem>
                            <para>Input Mapping: Generates the tool input XML file as a subset of
                                the incoming CPACS file XML structure, specified by a mapping file
                            </para>
                        </listitem>
                        <listitem>
                            <para>Tool Specific Input Mapping (optional): Adds tool specific data to
                                the tool input file, based on a mapping file and a data XML
                                file</para>
                        </listitem>
                        <listitem>
                            <para>Output Mapping: Merges the content of the tool output XML file
                                into the origin incoming CPACS file, based on a mapping file</para>
                        </listitem>
                        <listitem>
                            <para>Parameter Output Mapping (optional): Generates output values as
                                single values of the CPACS result file, based on an XPath configured
                                at workflow design time as a dynamic output of the component</para>
                        </listitem>
                        <listitem>
                            <para>Execution option to only run on changed input: If enabled, the
                                integrated tool will only run on changed input. Therefore the
                                content of the generated tool input file is compared to the last
                                runs content. Additionally the data of the static input channels are
                                compared to the previous ones.</para>
                        </listitem>
                    </itemizedlist></para>
                <para>All the features listed above can be configured in the tool integration wizard
                    on the dedicated <emphasis role="italic">CPACS Tool Properties</emphasis>
                    page.</para>
                <para>The mappings can be specified by XML or XSLT as shown in the following
                    examples. RCE differentiates between these methods in accordance to the
                    corresponding file extension (.xml or .xsl).</para>
                <para>For XML mapping, the following mapping modes are supported (see the mapping
                    mode definitions in the mapping examples below):<itemizedlist>
                        <listitem>
                            <para>append: Elements in the target path that have no equivalent in the
                                source path are retained and are not deleted. Otherwise the elements
                                in the target path are replaced by the corresponding elements in the
                                source path. Two elements in the source and target path are
                                considered to be the same if they have the same element name, the
                                same number of attributes and the same attributes with the same
                                values.</para>
                        </listitem>
                        <listitem>
                            <para>delete: Before copying, all elements that are described by the
                                target path are deleted in the target XML file. This is also the
                                standard behavior if no mapping mode is explicitly set in a mapping
                                rule.</para>
                        </listitem>
                        <listitem>
                            <para>delete-only: All elements that are described by the target path
                                are deleted in the target XML file.</para>
                        </listitem>
                    </itemizedlist></para>
                <para>If a target element described by the target path is not available in the XML
                    file, it is created including all of its parent elements.</para>
                <para>Example for an input or tool specific XML mapping :</para>
                <para>
                    <programlisting>&lt;?xml version="1.0" encoding="UTF-8"?>
&lt;map:mappings xmlns:map="http://www.rcenvironment.de/2015/mapping" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    &lt;map:mapping mode="append">
        &lt;map:source>/path/to/your/element&lt;/map:source>
        &lt;map:target>/toolInput/data/var1&lt;/map:target>
    &lt;/map:mapping>
    
    &lt;map:mapping mode="delete">
        &lt;map:source>/path/to/your/element&lt;/map:source>
        &lt;map:target>/toolInput/data/var2&lt;/map:target>
    &lt;/map:mapping>

    &lt;map:mapping mode="delete-only">
        &lt;map:target>/toolInput/data/var3&lt;/map:target>
    &lt;/map:mapping>

    &lt;map:mapping>
        &lt;map:source>/path/to/your/element&lt;/map:source>
        &lt;map:target>/toolInput/data/var4&lt;/map:target>
    &lt;/map:mapping>
    
    &lt;xsl:for-each select="$sourceFile/result/cases/case">
        &lt;map:mapping mode="delete">
            &lt;map:source>/path/to/your/case[&lt;xsl:value-of select="position()" />]/element&lt;/map:source>
            &lt;map:target>/toolInput/data/condition[&lt;xsl:value-of select="position()" />]/var&lt;/map:target>
        &lt;/map:mapping>
    &lt;/xsl:for-each>

&lt;/map:mappings></programlisting>
                </para>
                <para>Input or tool specific XSLT mapping:</para>
                <para>
                    <programlisting>&lt;?xml version="1.0" encoding="utf-8"?>
&lt;xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="cpacs_schema.xsd">
 &lt;xsl:output method="xml" media-type="text/xml" />
 &lt;xsl:template match="/">
  &lt;toolInput>
   &lt;data>
    &lt;var1>
     &lt;xsl:value-of select="/path/to/your/element" />
    &lt;/var1>
   &lt;/data>
  &lt;/toolInput>
 &lt;/xsl:template>
&lt;/xsl:stylesheet></programlisting>
                </para>
                <para>Example of an output XML mapping:</para>
                <para>
                    <programlisting>&lt;?xml version="1.0" encoding="UTF-8"?>
&lt;map:mappings xmlns:map="http://www.rcenvironment.de/2015/mapping" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    &lt;map:mapping>
        &lt;map:source>/toolOutput/data/result1&lt;/map:source>
        &lt;map:target>/path/to/your/result/element&lt;/map:target>
    &lt;/map:mapping>

&lt;/map:mappings></programlisting>
                </para>
                <para>And output XSLT mapping:</para>
                <para>
                    <programlisting>&lt;?xml version="1.0" encoding="UTF-8"?>
&lt;xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" exclude-result-prefixes="xsi">
  &lt;xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
  &lt;!--Define Variable for toolOutput.xml-->
  &lt;xsl:variable name="toolOutputFile" select="'./ToolOutput/toolOutput.xml'"/>
  &lt;!--Copy complete source file to result file -->
  &lt;xsl:template match="@* | node()">
    &lt;xsl:copy>
      &lt;xsl:apply-templates select="@* | node()"/>
    &lt;/xsl:copy>
  &lt;/xsl:template>
  &lt;!--Modify a value of an existing node-->
    &lt;xsl:template match="/path/to/your/result">
    &lt;element>  
    &lt;xsl:value-of select="document($toolOutputFile)/toolOutput/data/result1"/>
    &lt;/element>
    &lt;/xsl:template>
&lt;/xsl:stylesheet></programlisting>
                </para>
                <para>Please ensure to use the proper namespace for map
                    (xmlns:map="http://www.rcenvironment.de/2015/mapping") in XML mapping files and
                    the proper namespace for xsl (xmlns:xsl="http://www.w3.org/1999/XSL/Transform")
                    in both types of mapping files.</para>
                <para>The figure below illustrates how the additional features are used in the run
                    process of an user-integrated CPACS tool.</para>
                <figure>
                    <title>Run process of an user-integrated CPACS Tool</title>
                    <mediaobject>
                        <imageobject>
                            <imagedata fileref="images/ug_CPACS_ToolRunProcess.png"
                                contentwidth="390"/>
                        </imageobject>
                    </mediaobject>
                </figure>
            </section>
            <section>
                <title>Integrate a CPACS Tool into a Client Instance</title>
                <para>
                    <orderedlist>
                        <listitem>
                            <para>Start RCE as Client</para>
                        </listitem>
                        <listitem>
                            <para>Open the <emphasis role="italic">Tool Integration
                                    Wizard</emphasis> by clicking the <emphasis role="italic"
                                    >Integrate Tool...</emphasis> in the <emphasis role="italic"
                                    >File</emphasis> menu.<note>
                                    <para>You will always find further help by clicking the
                                            <emphasis role="italic">?</emphasis> on the bottom left
                                        corner on each page of the wizard or by pressing <emphasis
                                            role="italic">F1</emphasis>.</para>
                                </note>
                            </para>
                        </listitem>
                        <listitem>
                            <para>Choose the option <emphasis role="italic">Create a new tool
                                    configuration from a template</emphasis>. <note>
                                    <para>The CPACS templates delivered with RCE are designed to
                                        match the conventions of the old CPACS tool wrapper
                                        (respectively ModelCenter tool wrapper). Most of the
                                        properties are preconfigured and do not need to be
                                        changed.</para>
                                </note></para>
                        </listitem>
                        <listitem>
                            <para>Select one of the <emphasis role="italic">CPACS</emphasis>
                                templates. <?linebreak?>Click <emphasis role="italic"
                                    >Next</emphasis>.</para>
                        </listitem>
                        <listitem>
                            <para>Fill in the <emphasis role="italic">Tool Description</emphasis>
                                page. <?linebreak?>Click <emphasis role="italic"
                                >Next</emphasis>.</para>
                        </listitem>
                        <listitem>
                            <para>On the <emphasis role="italic">Inputs and Outputs</emphasis> page
                                you will find preconfigured static in- and outputs, that will match
                                the old tool wrapper conventions. If your tool needs additional in-
                                or outputs, feel free to configure.<?linebreak?>Click <emphasis
                                    role="italic">Next</emphasis>.</para>
                        </listitem>
                        <listitem>
                            <para>Skip the page<emphasis role="italic"> Tool Properties</emphasis>
                                by clicking <emphasis role="italic">Next</emphasis> since it is not
                                relevant for tools that match the conventions of the old CPACS tool
                                wrapper.</para>
                        </listitem>
                        <listitem>
                            <para>Add a launch setting for the tool by clicking the <emphasis
                                    role="italic">Add</emphasis> button on the <emphasis
                                    role="italic">Launch Settings</emphasis> page. Configure the
                                path of the CPACS tool and fill in a version, click <emphasis
                                    role="italic">OK</emphasis>. If you would like to allow users of
                                your tool to choose that the temp directory won’t be deleted at all
                                after workflow execution, check the property <emphasis role="italic"
                                    >Never delete working directory(ies)</emphasis>. Not to delete
                                the working directory can be very useful for users for debugging
                                purposes, at least if they have access to the server’s file system.
                                But this option can result in disc space issues as the amount
                                required grows continuously with each workflow execution. It is
                                recommended to check that option during integrating the tool and
                                uncheck it before publishing the tool. <?linebreak?>Click <emphasis
                                    role="italic">Next</emphasis>.</para>
                        </listitem>
                        <listitem>
                            <para>The <emphasis role="italic">CPACS Tool Properties</emphasis> are
                                preconfigured to match the folder structure defined for the old
                                CPACS tool wrapper. In most cases you do not have to change this
                                configuration. If you are using XSLT mapping, please select the
                                corresponding mapping files. If your tool does not work with static
                                tool specific input, please deselect this property.
                                <?linebreak?>Click <emphasis role="italic">Next</emphasis>.</para>
                        </listitem>
                        <listitem>
                            <para>In the <emphasis role="italic">Execution command(s)</emphasis> tab
                                on the <emphasis role="italic">Execution</emphasis> page, you need
                                to define your execution command itself as well as optional pre and
                                post commands. Commands will be processed sequentially line by line.
                                An example for a typical Windows command including pre and post
                                commands will look like the following:</para>
                            <para>
                                <programlisting>rem pre-command
pre.bat

rem tool-execution
YourTool.exe ToolInput/toolInput.xml ToolOutput/toolOutput.xml

rem post-command
post.bat</programlisting>
                            </para>
                        </listitem>
                        <listitem>
                            <para>Click <emphasis role="italic">Save and activate</emphasis> and
                                your tool will appear immediately in the palette and is be ready to
                                use.</para>
                        </listitem>
                        <listitem>
                            <para>If not already done, do not forget to publish your tool (cf. <xref
                                    linkend="publishing_authorization"/> ) after testing it locally.
                                To check if your tool is successfully published to the RCE network
                                open the tab <emphasis role="italic">Network View</emphasis> at the
                                bottom and checkout <emphasis role="italic">Published
                                    Components</emphasis> after expanding the entry of your RCE
                                instance.</para>
                        </listitem>
                    </orderedlist>
                </para>
            </section>
            <section>
                <title>Integrate a CPACS Tool into a Server Instance in Headless Mode</title>
                <para>The way to integrate a CPACS tool on a server running RCE in headless mode is
                    as follows: Perform the steps to integrate a CPACS tool on a client instance and
                    make sure that the path of the CPACS tool configured on the <emphasis
                        role="italic">Launch Settings</emphasis> page (step 8) matches the absolute
                    tool path on your server system. Afterwards, you will find the configuration
                    files inside your rce profile folder at the following location:</para>
                <para><code>/integration/tools/cpacs/[YourToolName]</code></para>
                <para>Copy the folder <code>[YourToolName]</code> to the same location inside the
                    profile folder running with your headless server instance. Use the "auth"
                    commands (cf. <xref linkend="publishing_authorization"/> ) to publish your tool.
                    If the server instance is already running, your tool will be available
                    immediately after publishing.</para>
            </section>
        </section>
        <section id="tool_integration_known_issues">
            <title>Known Issues</title>
            <para>As noted above, pre- and postscripts are executed using a Java implementation of
                Python 2. One particular caveat is Python 2's handling of unicode strings, which
                requires prefixing strings containing non-ASCII-characters with <code>u</code>.
                Please refer to, e.g., <ulink
                    url="https://python.readthedocs.io/en/v2.7.2/howto/unicode.html">this
                    tutorial</ulink> on handling unicode in Python 2 for further information.</para>
            <para>Under Windows, the execution script is eventually handed to the Java method
                    <code>Runtime#eval</code>. This method in turn relies on internal Windows APIs
                which have known issues with handling unicode. Please make sure that your execution
                script works as expected when executed via RCE with unicode strings.</para>
        </section>
    </section>
    <section>
        <title>Integrating Workflows as Components (Experimental)</title>
        <para>In this section we describe how to integrate a workflow containing multiple components
            as a component itself. This feature is currently experimental and <emphasis>not
                recommended for productive use</emphasis>.</para>
        <para>Consider a disciplinary tool that computes the value of some function f_c(x) for some
            parameter c and some input value x and assume that the user has already integrated this
            tool as the component <code>DiscComp</code>. Furthermore assume that in multiple
            workflows the user would like to fix some value for c and find a minimum of f_c. She
            implements this use case via the structure shown in the following figure.<figure>
                <title>Workflow for determining the optimal input for the function f_c(x).</title>
                <mediaobject>
                    <imageobject>
                        <imagedata fileref="images/InnerWorkflow.png"/>
                    </imageobject>
                </mediaobject>
            </figure></para>
        <para>In that workflow, the user opted to provide the parameter c via an input provider,
            while she used an optimizer to determine the optimal value of x. That optimal value is
            then written via an output writer. The user now wants to use this workflow as part of
            other, more complex workflows.</para>
        <para>One approach would be to simply copy the part of the workflow that implements the
            actual computation (i.e., the components <code>Optimizer</code> and
                <code>DiscComp</code>) and paste it whenever it needs this functionality in
            other workflows. This approach, however, is neither scalable nor maintainable: While this
            example requires only copying of two componentes, one can easily imagine situations in
            which the functionality to be copied is implemented via dozens of components, which
            leads to severe cluttering of the workflows in which the functionality is used.
            Furthermore, if the user changes the original workflow, e.g., if she uses another
            algorithm for the optimization, she would have to re-copy the changed parts to all
            workflows that use the original parts.</para>
        <para>Instead of manually copying and pasting, the user may instead opt to integrate the
            workflow shown in the above figure as a tool to be used in other workflows. This allows
            her to hide the details of the implementation (i.e., the use of an optimizer and of
                <code>DiscComp</code>) from users of her component and to easily update that
            implementation.</para>
        <para>In the following, we first show how to integrate an existing workflow as a component
            before detailing the technical backgrounds of executing a workflow as a component.
            Finally, we discuss caveats and common questions about this feature. In all these
            sections, we will refer to an "inner" workflow and an "outer" workflow. These refer to
            the workflow that is integrated as a component and to the workflow in which that
            component is used later on, respectively.</para>
        <section>
            <title>Integrating a Workflow via command console</title>
            <para>Before integrating the workflow shown above, we assume that you have already
                constructed a workflow that implements the behavior that you want to provide to
                other users as a component. Moreover, we assume that this workflow contains some
                input providers that feed initial data into the workflow and some output writers
                that persist the results of the computation implemented by the workflow. In the
                figure above, these input providers and output writers are situated to the left of
                the component <code>DiscComp</code> and to the right of the optimizer, respectively.
                Finally, the workflow to be integrated must not contain any placeholders (cf. <xref linkend="placeholder_files"/> ). Otherwise user input would be required at
                execution time in order to assign values, which would prevent automated execution of
                the integrated workflow.<note>
                    <para>You can easily determine whether your workflow contains placeholders by
                        opening the workflow execution wizard (either via the green arrow in the
                        upper bar in the GUI or via the shortcut Ctrl + Shift + X). If there exist
                        any placeholders that are to be assigned values before the start of the
                        execution, the wizard will show a second page that displays all such
                        placeholders. If no such page exists, the workflow does not contain
                        placeholders and is ready for integration as a component.</para>
                </note></para>
            <para>Integrating a workflow consists of nothing more than determining endpoints of
                components in the inner workflow that will be exposed to the outer workflow by the
                resulting component. In this case, we opt to expose the input <code>c</code> of
                    <code>DiscComp</code> as well as the output <code>x_output</code> of
                    <code>Optimizer</code>. In general, inputs will be exposed as inputs on the
                component in the outer workflow, while outputs will be exposed as outputs. It is not
                possible to expose an input of a component in the inner workflow as an output to the
                outer workflow, or vice versa.</para>
            <para>In order to integrate the above workflow as a component, we first remove the input
                providers and output collectors that handle the inputs and outputs that are to be
                passed into the inner workflow by the outer workflow. In the example above, we
                simply deactivate the two components (e.g. via the keyboard shortcut Ctrl + D) and
                obtain the workflow shown in the following figure.</para>
            <figure>
                <title>Workflow from the above figure prepared for integration as a
                    component.</title>
                <mediaobject>
                    <imageobject>
                        <imagedata fileref="images/InnerWorkflowPrepared.png"/>
                    </imageobject>
                </mediaobject>
            </figure>
            <para>While previously, all endpoints of all components were connected, now there exist
                two unconnected endpoints: The input c of <code>DiscComp</code> as well as the
                output <code>x_optimal</code> of <code>Optimizer</code>. The workflow is now ready
                for integration as a component.</para>
            <para>Integration of workflows is performed via the command console and, in particular,
                via the command <code>wf integrate</code>. This command has the following general
                form:<programlisting>wf integrate [-v] &lt;component name> &lt;absolute path to .wf file> [&lt;exposed endpoint definition>...]</programlisting></para>
            <para>The optional argument <code>-v</code> enables verbose mode. If this parameter is
                set, the command outputs detailed information about the endpoints that are exposed
                to the calling workflows. This does not change the behavior of the command.</para>
            <para>The parameter <code>component name</code> determines the name of the component
                that is integrated, i.e., the name that will appear in the pallet and in the
                workflow editor. Since in our example the purpose of the new component in our
                example is to determine some optimal parameter x, we opt to call the component
                    <code>FindOptimalX</code>.</para>
            <para>The parameter <code>absolute path to .wf file</code> is self-explanatory and
                denotes the path on your local file system where the workflow file describing the
                workflow to be integrated is located. In our example we assume that the workflow
                file is located at <phrase os="windows"><code>C:\Users\user\workflow.wf</code></phrase><phrase os="linux"><code>/home/user/workflow.wf</code></phrase>.<note>
                    <para>Recall that you can obtain the absolute path to any workflow file in the
                        project explorer via a right click on the workflow and selecting
                            <emphasis>Copy Full Path</emphasis>.</para>
                    <para>Furthermore, recall that parameters in the command console are separated
                        by spaces unless the parameter is surrounded by quotation marks . Hence, if
                        the path to your workflow contains spaces, enclose it in quotation marks.
                        Finally, recall that backslashes must be escaped, i.e., the path <code>C:\My
                            Folder</code> would have to be entered as <code>"C:\\My
                        Folder"</code>.</para>
                </note></para>
            <para>Each succeeding parameter is interpreted as the definition of an exposed endpoint.
                Each such definition is of the following
                form:<programlisting>--expose &lt;component name>:&lt;internal endpoint name>:&lt;exposed endpoint name></programlisting></para>
            <para>Here, <code>component name</code> refers to the name of the component in the inner
                workflow whose endpoint is to be exposed. The parameter <code>internal endpoint
                    name</code> denotes the name of the endpoint of the component that is to be
                exposed, while the parameter <code>exposed endpoint name</code> determines the name
                of the endpoint on the resulting component. Make sure that each <code>exposed
                    endpoint name</code> is unique within the context of the resulting component, as
                the behavior of a component with multiple inputs or outputs of the same name is undefined.<note>
                    <para>Instead of the names of the component and the endpoints that are displayed
                        in the workflow editor, you may instead use the internal identifiers of
                        these nodes and endpoints, respectively. These are not currently shown in
                        the GUI of RCE but can, e.g., be determined by inspecting the workflow file
                        via some text editor. While this should not be necessary when integrating
                        workflows manually, it may prove useful when automating the creation and
                        integration of workflows. </para>
                </note></para>
            <para>Recall that you do not need to specify whether the endpoint is exposed as an input
                or as an output, but that the underlying endpoint determines the configuration of
                the endpoint on the resulting component: Inputs are only ever exposed as inputs,
                whereas outputs are only ever exposed as outputs. This principle extends to the
                configuration of inputs: If the endpoint on the component in the inner workflow is,
                e.g., configured be required for component execution and to only expect a constant
                value, then the endpoint on the resulting component is configured
                analogously.</para>
            <para>Furthermore recall that we want to expose the input <code>c</code> of
                    <code>DiscComp</code> as well as the output <code>x_output</code> of
                    <code>Optimizer</code>. We want the former input to retain its original name,
                while we want to expose the latter input as <code>optimalX</code>. In order to
                integrate the example workflow prepared above as a component, we thus issue the
                following
                command:<programlisting>wf integrate FindOptimalX "<phrase os="windows">C:\Users\user\workflow.wf</phrase><phrase os="linux">/home/user/workflow.wf</phrase>"--expose DiscComp:c:c --expose Optimizer:x_optimal:optimalX</programlisting></para>
            <para>When enabling verbose mode via the switch <code>-v</code>, RCE writes the
                following
                output<programlisting>Input Adapter : c --[Float,Constant,Required]-> c @ 03b5b758-3b44-4a53-b832-be9991321285
Output Adapter: x_optimal @ 402cac5e-2206-48cc-a62f-803bd320a15a --[Float]-> x_opt</programlisting></para>
            <para>where <code>03b5b758-3b44-4a53-b832-be9991321285</code> and
                    <code>402cac5e-2206-48cc-a62f-803bd320a15a</code> denote the IDs of the
                component <code>DiscComp</code> and of <code>Optimizer</code>, respectively.</para>
            <para>Once the execution of the command has finished, a new component named
                    <code>FindOptimalX</code> with a single input named <code>c</code> and a single
                output named <code>optimalX</code> will be available for use in all other
                workflows.</para>
        </section>
        <section>
            <title>Workflow Integration Editor</title>
            <para> With RCE 10.4 we provide the Workflow Integration Editor to support the workflow
                integration via the graphical user interface of RCE. There are two options to open
                the editor:<itemizedlist>
                    <listitem>
                        <para>Select the workflow you want to integrate as a component from the
                            Project Explorer and go to either the main menu <emphasis role="italic"
                                >Intgration > Integrate Workflow File</emphasis> or use the quick
                            acess via the icon bar.</para>
                    </listitem>
                    <listitem>
                        <para>Select a (sub-)workflow in the Workflow Editor and choose from the
                            context menu <emphasis role="italic">Integrate selected Components as
                                Component</emphasis></para>
                    </listitem>
                </itemizedlist></para>
            <para>Required fields are marked with an asterisk (*). Press <emphasis role="italic"
                    >Integrate Workflow</emphasis> to integrate the workflow as a component. The
                component will automatically show up in the Workflow Editor palette.</para>
            <para>
                <note>
                    <para>The workflow integration editor has a dynamic help, which is shown by
                        clicking on the question mark on the bottom left or by pressing F1. It will
                        guide you through the tabs of the workflow integration editor.</para>
                </note>
            </para>
            <para>To edit an already integrated workflow component use the <emphasis role="italic"
                    >Intgration > Edit Workflow Integration</emphasis> from the main menu or use the
                context menu in either the Workflow Editor or the Palette after selecting the
                desired workflow component.</para>
            <figure>
                <title>Workflow Integration Editor</title>
                <mediaobject>
                    <imageobject>
                        <imagedata fileref="images/WorkflowIntegrationEditor.png" width="400px"/>
                    </imageobject>
                </mediaobject>
            </figure>
            <para>One major part of the workflow integration consists of the input/output mapping.
                On the Mapping tab all unconnected endpoints of your selected (sub-)workflow are
                displayed. Here you can select endpoints to serve as inputs and outputs for the
                integrated workflow component. <note>
                    <para> Inputs with the constraint <emphasis role="italic">Required</emphasis>
                        are mandatory. </para>
                </note></para>
            <para>You can assign custom names for your endpoints by editing the Mapped Name cell in
                the table.</para>
            <figure>
                <title>Mapping Page</title>
                <mediaobject>
                    <imageobject>
                        <imagedata fileref="images/MappingPage.png" width="300px"/>
                    </imageobject>
                </mediaobject>
            </figure>
        </section>
        <section>
            <title>Technical Details of Executing an Integrated Workflow</title>
            <para>In this section some technical details are given for interested users how exactely
                an integrated workflow used as a workflow component in another workflow is executed.
                If you are interested in more conceptual details or encounter problems while running
                an integrated workflow, this section may be helpful.</para>
            <para>Recall that each workflow that RCE executes is controlled by some particular
                instance, i.e., by the workflow controller. Since executing an integrated workflow
                executes the underlying workflow, RCE requires a workflow controller for doing so.
                That workflow controller may or may not be the same as the one executing the outer
                workflow. Currently, the instance publishing the component serves as the workflow
                controller for the execution of the inner workflow. We call it the "inner workflow
                controller" to distinguish it from the workflow controller that runs the outer
                workflow. This "inner workflow controller" runs a copy of the workflow in the exact
                state in which it was integrated, i.e., changes made to the workflow after
                integration will have no effect on the behavior of the integrated component.</para>
            <para>Futhermore, since the publishing instance serves as "inner workflow controller"
                for the execution of the integrated workflow, the execution of the integrated
                workflow will show up in the Workflow Data Browser of the publishing instance under
                the name <code>&lt;workflow component name> running as component '&lt;component
                    name>' of workflow '&lt;outer workflow>'</code>. The <code>&lt;workflow
                    component name></code> denotes the name as which the publishing instance
                published the integrated workflow, <code>&lt;component name></code> denotes the name
                under which the workflow component is used in the outer workflow, and
                    <code>&lt;outer workflow></code> denotes the name under which the outer workflow
                is stored in the Workflow Data Browser of its workflow controller.<note>
                    <para>Nesting workflows, i.e., integrating workflows as components that already
                        contain workflows integrated as components, can easily lead to unreadable
                        names of workflow executions that are stored in the Workflow Data Browser.
                        This may significantly inhibit manual inspection of the resulting data. Keep
                        this in mind when designing workflows.</para>
                </note></para>
            <para>Technically, before starting the integrated workflow, the instance controlling the
                inner workflow injects two additional components into the workflow, one so-called
                input adapter and one so-called output adapter. These components are not accessible
                by the user when constructing workflows. They are only used to transport data from
                the inputs of the workflow component to the exposed inputs as well as data from the
                exposed outputs to the outputs of the workflow component, respectively.</para>
            <para>Upon execution of the integrated component in the outer workflow, the instance
                publishing the component first injects the input and the output adapter as described
                above. It subsequently executes the workflow and collects the results via the output
                adapter. </para>
        </section>
        <section>
            <title>Limitations, Caveats, and FAQ</title>
            <para>Since the integration of workflows as components is currently under development
                and only released as a beta feature, there are some caveats and known issues that
                you should be aware of. We have alluded to these limitations and caveats throughout
                this section, but briefly list them here again for the sake of readability.</para>
            <itemizedlist>
                <listitem>
                    <para>Workflow files are "frozen" at integration time. Changes to an integrated
                        workflow file after integration do not change the behavior of the component.
                        If you want to apply changes to the workflow file to the component, you will
                        have to re-integrate the workflow.</para>
                </listitem>
                <listitem>
                    <para>Currently, no placeholder files (cf. <xref linkend="placeholder_files"/> )
                        are supported, i.e., the integrated workflow must contain no placeholders.
                        Moreover, the workflow is not checked for containing placeholders at
                        integration time, but instead the execution of the the component will fail
                        at execution time.</para>
                </listitem>
                <listitem>
                    <para>Via the command console, the user cannot specify a version of the
                        integrated component. If there is demand, we will add the command line
                        switch <code>--version</code> in order to allow the user to have multiple
                        versions of the same workflow integrated simultaneously. Also, the user can
                        currently not specify an individual icon to be used for the integrated
                        component. This may also be added in future versions.</para>
                </listitem>
                <listitem>
                    <para>If some adapted output is written to multiple times during a single run of
                        the integrated workflow, only the final values written to that output are
                        forwarded to the outer workflow.</para>
                </listitem>
                <listitem>
                    <para>Due to this new implementation, there is doubled functionality between the
                        command <code>wf integrate</code> and the command <code>ra-admin
                            wf-publish</code>. After the full release of the integration of
                        workflows as component, the latter command will be deprecated and its output
                        replaced by a message asking the user to use <code>wf integrate</code>
                        instead.</para>
                </listitem>
                <listitem>
                    <para>If the underlying workflow is paused during execution, this pause state is
                        not reflected in the calling workflow. Instead the component is shown as
                        running. Similarly, if the integrated workflow includes some result
                        verification and the results are rejected, the component simply fails
                        instead of indicating the rejection of results.</para>
                </listitem>
                <listitem>
                    <para>Component names passed to the command <code>wf integrate</code> are not
                        checked to satisfy the rules on component names. This will be fixed before
                        release and integration of a component with an invalid name will be refused
                        with an informative error message.</para>
                </listitem>
            </itemizedlist>
            <para>Furthermore, there are some common questions that may occur in the context of
                integrating a workflow as a component. We collect and answer these questions here
                again for the sake of readability.<variablelist>
                    <varlistentry>
                        <term>Where is the integration folder of my new component?</term>
                        <listitem>
                            <para>The integration of a workflow as a component is stored in a
                                profile in the folder <phrase os="windows"
                                        ><code>integration\tools\workflow</code></phrase><phrase
                                    os="linux"
                                ><code>integration/tools/workflow</code></phrase>.</para>
                        </listitem>
                    </varlistentry>
                    <varlistentry>
                        <term>Can I move the folder containing the integration of a workflow to
                            other instances, similarly to the integration of common tools?</term>
                        <listitem>
                            <para>Yes, this is possible, since the integration folder contains a
                                copy of the workflow file which was produced at integration time.
                                Also, you can publish integrated workflows to other instances just
                                as you can publish common tools.</para>
                        </listitem>
                    </varlistentry>
                    <varlistentry>
                        <term>What happens if an integrated workflow uses some remote components
                            that are not available? </term>
                        <listitem>
                            <para>In that case the component is still available as long as the
                                instance publishing it is available. The availability of the
                                components contained in the integrated workflow is only checked at
                                execution time. If a component is unavailable at that time, the
                                execution of the component fails.</para>
                        </listitem>
                    </varlistentry>
                </variablelist></para>
        </section>
    </section>
    
</section>
