<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="http://docbook.org/xml/5.0/rng/docbook.rng" schematypens="http://relaxng.org/ns/structure/1.0"?>
<?xml-model href="http://docbook.org/xml/5.0/rng/docbook.rng" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
<section xmlns="http://docbook.org/ns/docbook" 
    xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0">
    <title>Integrated Test Script Runner</title>
    <para>The integrated "Test Script Runner" (TSR) was created to automate test sequences on one or
        more RCE standalone installations. These installations are automatically set up using an RCE
        feature called "Instance Management" (IM), which is still under development, and therefore
        not fully documented yet. However, the instructions below should be sufficient to configure
        this feature as needed for the TSR.</para>
    <para>As of RCE 9.0.0, the Test Script Runner is included in the standard application release,
        as well as the standard Eclipse checkout. Therefore, very little configuration is required
        to use it.</para>
    <section>
        <title>Configuration</title>
        <para>The only configuration that is required for using the TSR is adding an Instance
            Management configuration to the RCE instance that will execute the test scripts. This
            defines the root directory where work files and directories of managed RCE test
            installations will be stored. Locate the profile directory that is being launched and
            edit its <code>configuration.json</code> file. In this file, add this configuration
            block on the root JSON level, and adjust the settings as necessary:</para>
        <para>
            <literallayout><code>"instanceManagement": {
  "dataRootDirectory": "<phrase os="windows">C:\\MyWorkdir\\rce-im-dat</phrase><phrase os="linux">/tmp/rce-im-dat</phrase>",
  "installationsRootDirectory": "<phrase os="windows">C:\\MyWorkdir\\rce-im-inst</phrase><phrase os="linux">/tmp/rce-im-inst</phrase>"
}</code></literallayout>
        </para>
        <para>There are additional Instance Management configuration parameters available. These
            are, however not usually needed for using the TSR.</para>
        <note>
            <para>It is <emphasis>strongly</emphasis> recommended to use a short full filesystem
                path as the <code>dataRootDirectory</code> and the
                    <code>installationsRootDirectory</code>, as RCE installations will be placed
                inside of it, and long filesystem paths are known to cause problems with these. The
                maximum lenght will be determined, documented, and maybe also automatically checked
                in the future.</para>
        </note>
    </section>
    <section>
        <title>Test Definitions</title>
        <para>We use the language <link xlink:href="https://cucumber.io/docs/gherkin/"
                >Gherkin</link>, a structured natural language, to define our tests and execute them
            using <link xlink:href="https://cucumber.io/">Cucumber</link>. In the nomenclature of
            Gherkin, a single test is called a <emphasis>scenario</emphasis>, while multiple
            scenarios make up a <emphasis>feature</emphasis>. The tests are defined in
                <code>*.feature</code> files in the directory
                <code>/de.rcenvironment.supplemental.testscriptrunner.scripts/resources/scripts</code>.<note>
                <para>If you would like to edit <code>.feature</code>-files directly from Eclipse,
                    we recommend using the <emphasis>Cucumber Eclipse Plugin</emphasis>, which can
                    be found in the Eclipse Marketplace. </para>
            </note>We show one test (or scenario)
            below:<programlisting>@Start01
@DefaultTestSuite
Scenario: Concurrent headless instance starting and stopping

  Given instances "Node1, Node2, Node3, Node4, Node5" using the default build
  When  starting all instances concurrently
  Then  instances "Node1, Node2, Node3, Node4, Node5" should be running
  
  When  stopping all instances concurrently
  Then  the log output of all instances should indicate a clean shutdown with no warnings or errors</programlisting></para>
        <para>Each test has one or more <emphasis>test IDs</emphasis> denoted by the annotation-like
            tags above the indivual test scenarios, e.g. <code>@Start01</code> or
                <code>@DefaultTestSuite</code>. These IDs serve to later refer to that test for
            execution. The same ID may also be assigned to multiple tests. Thus, the test script
            runner can, e.g., be asked to execute all tests with a test ID of
                <code>DefaultTestSuite</code>. The values of the test IDs can be chosen almost
            arbitrarily. The only reserved IDs are <code>@Disabled</code> and
            <code>@disabled</code>, which prevent the test from being executed at all. </para>
        <para>In order to execute a test written in Gherkin, Cucumber requires an implementation of
            each individual line in the test. Each line is called a <emphasis>test step</emphasis>,
            while the code implementing the desired behavior is called a <emphasis>test step
                definition</emphasis>. In RCE, the test step definitions are location in the bundle
                <code>de.rcenvironment.supplemental.testscriptrunner</code> in the package
                <code>de.rcenvironment.extras.testscriptrunner.definitions.impl</code>. Please refer
            to those implementations for the canonical overview over available test steps. In the
            following, we list some of the more commonly used test steps. We denote placeholders
            using angular brackets.</para>
        <para>
            <variablelist>
                <varlistentry>
                    <term>Given running instance &lt;instance id> using the default build</term>
                    <listitem>
                        <para>Starts an instance of RCE without a GUI. The instance id is used to
                            refer to this instance in later test steps.</para>
                    </listitem>
                </varlistentry>
                <varlistentry>
                    <term>When executing workflow &lt;workflow id> on node &lt;instance id></term>
                    <listitem>
                        <para>Executes the given workflow on the given instance and waits for the
                            termination of that workflow. The workflow id must correspond to the
                            basename of a workflow file in the directory
                                <code>de.rcenvironment.supplemental.testscriptrunner.scripts/resources/scripts/workflows</code>,
                            i.e., the file name without the suffix <code>.wf</code>.</para>
                    </listitem>
                </varlistentry>
                <varlistentry>
                    <term>Then that workflow run should be identical to &lt;golden master id></term>
                    <listitem>
                        <para>Can only be used after using the test step "When executing workflow
                            &lt;workflow id> on node &lt;instance id>". Asserts that the workflow
                            execution is identical to that stored in the directory
                                <code>de.rcenvironment.supplemental.testscriptrunner.scripts/resources/scripts/golden_masters</code>.
                            The golden master id must correspond to the basename of a file in that
                            directory, i.e., to the file name without the suffix <code>.json</code>.
                            You can export a workflow execution to serve as a golden master via the
                            command <code>tc export_wf_run</code> in a running RCE instance.</para>
                    </listitem>
                </varlistentry>
            </variablelist>
        </para>
    </section>
    <section>
        <title>Executing Tests</title>
        <para>The TSR is invoked by a single RCE console command (<code>run-test</code>), with an
            alias for readability (<code>run-tests</code>). The general syntax is:</para>
        <para><code>run-test[s] [--format pretty|json] &lt;comma-separated list of test ids>|--all
                &lt;build id></code></para>
        <para>By default, the result of the test is printed in human-readable format. If you would
            like output in the JSON-format instead, you may use the option <code>--format</code>,
            which requires either <code>pretty</code> or <code>json</code> as its only
            parameter.</para>
        <para>There are three typical scenarios for calling this command: <itemizedlist>
                <listitem>
                    <para>from within an RCE instance launched from Eclipse during development,
                        usually using the GUI workflow console</para>
                </listitem>
                <listitem>
                    <para>from within a standalone RCE instance, also usually using the GUI workflow
                        console</para>
                </listitem>
                <listitem>
                    <para>as a CLI batch run (<code>rce --batch "...")</code> using a standalone
                        instance.</para>
                </listitem>
            </itemizedlist></para>
        <para>The RCE installation to be tested is defined by the <replaceable>&lt;build
                id></replaceable> parameter in the above command. One important aspect to understand
            is that this installation is generally independent of the installation being used to
            execute the TSR command. The latter is, in a sense, only the "host" of test scripts.
            There are three ways of specifying the build to test: <orderedlist>
                <listitem>
                    <para>A  build download id, which corresponds to a certain part of the standard
                        download URL, for example <code>snapshots/trunk</code> or
                            <code>releases/9.0.0</code>. The structure should be self-explanatory.
                        (The major release tree to use for snapshot builds is one of the optional
                        Instance Management settings mentioned above; the default is to use the
                        current major version, ie 9.x.)</para>
                </listitem>
                <listitem>
                    <para>A path to an <emphasis>unpacked</emphasis> local standalone (product)
                        build, which can, for example, result from a local build run or from
                        unpacking a downloaded product zip file. The syntax for this is
                                <code>local:<replaceable>&lt;local installation
                            path></replaceable></code>. This directory can be either writable or
                        read-only. For example, it is also possible to test a (read-only) .deb or
                        .rpm package installation this way.</para>
                    <para>Note that this path must point to an already-unpacked RCE build, unlike
                        the first approach, which downloads zipped release packages and unpacks them
                        automatically.</para>
                </listitem>
                <listitem>
                    <para>As it is a frequent use case when testing standalone builds to execute the
                        test command the installation itself, there is a convenient shortcut for
                        this. By specifying <code>:self</code> as the build id, the test scripts are
                        executed on the installation of the instance used to run the test
                        command.</para>
                    <para>Note that due to technical limitations, however, this shortcut is not
                        possible when launching RCE from Eclipse, as the test scripts require a
                        standard product build to execute.</para>
                </listitem>
            </orderedlist></para>
        <para>Recall that each test has one or more test ids, denoted by annotation-like tags. These
            test ids can be specified in the command <code>run-test[s]</code> with or without the
                <code>@</code> character. "<code>--all</code>" executes all available test
            scenarios. </para>
    </section>
    <section>
        <title>Examples</title>
    </section>
    <section>
        <para>
            <itemizedlist>
                <listitem>
                    <para><code>run-test Test02,Test04 snapshots/trunk</code> - runs two specific
                        tests on the latest snapshot build</para>
                </listitem>
                <listitem>
                    <para><code>run-test --format pretty Test02,Test04 snapshots/trunk</code> -
                        equivalent to the command above</para>
                </listitem>
                <listitem>
                    <para><code>run-test --format json Test02,Test04 snapshots/trunk</code> - runs
                        two specific tests on the latest snapshot build and outputs the result in
                        JSON format</para>
                </listitem>
                <listitem>
                    <para><code>run-test DefaultTestSuite :self</code> - runs the default collection
                        of tests on the current installation</para>
                </listitem>
                <listitem>
                    <para><code>run-test --all local:/tmp/local-rce-build</code> - runs all
                        available tests on a local build</para>
                </listitem>
                <listitem>
                    <para><code>rce --batch "run-test DefaultTestSuite :self"</code> - the full
                        command line for the standard self-test of an installation</para>
                </listitem>
            </itemizedlist>
        </para>
    </section>
</section>
