The only configuration that is required for using the TSR is adding an Instance Management configuration to the RCE instance that will execute the test scripts. This defines the root directory where work files and directories of managed RCE test installations will be stored. Locate the profile directory that is being launched and edit its configuration.json file. In this file, add this configuration block on the root JSON level, and adjust the settings as necessary:

There are additional Instance Management configuration parameters available. These are, however not usually needed for using the TSR.

We use the language Gherkin, a structured natural language, to define our tests and execute them using Cucumber. In the nomenclature of Gherkin, a single test is called a scenario, while multiple scenarios make up a feature. The tests are defined in *.feature files in the directory /de.rcenvironment.supplemental.testscriptrunner.scripts/resources/scripts.

We show one test (or scenario) below:

@Start01
@DefaultTestSuite
Scenario: Concurrent headless instance starting and stopping

  Given instances "Node1, Node2, Node3, Node4, Node5" using the default build
  When  starting all instances concurrently
  Then  instances "Node1, Node2, Node3, Node4, Node5" should be running
  
  When  stopping all instances concurrently
  Then  the log output of all instances should indicate a clean shutdown with no warnings or errors

Each test has one or more test IDs denoted by the annotation-like tags above the indivual test scenarios, e.g. @Start01 or @DefaultTestSuite. These IDs serve to later refer to that test for execution. The same ID may also be assigned to multiple tests. Thus, the test script runner can, e.g., be asked to execute all tests with a test ID of DefaultTestSuite. The values of the test IDs can be chosen almost arbitrarily. The only reserved IDs are @Disabled and @disabled, which prevent the test from being executed at all.

In order to execute a test written in Gherkin, Cucumber requires an implementation of each individual line in the test. Each line is called a test step, while the code implementing the desired behavior is called a test step definition. In RCE, the test step definitions are location in the bundle de.rcenvironment.supplemental.testscriptrunner in the package de.rcenvironment.extras.testscriptrunner.definitions.impl. Please refer to those implementations for the canonical overview over available test steps. In the following, we list some of the more commonly used test steps. We denote placeholders using angular brackets.

The TSR is invoked by a single RCE console command (run-test), with an alias for readability (run-tests). The general syntax is:

run-test[s] [--format pretty|json] <comma-separated list of test ids>|--all <build id>

By default, the result of the test is printed in human-readable format. If you would like output in the JSON-format instead, you may use the option --format, which requires either pretty or json as its only parameter.

There are three typical scenarios for calling this command:

The RCE installation to be tested is defined by the <build id> parameter in the above command. One important aspect to understand is that this installation is generally independent of the installation being used to execute the TSR command. The latter is, in a sense, only the "host" of test scripts. There are three ways of specifying the build to test:

Recall that each test has one or more test ids, denoted by annotation-like tags. These test ids can be specified in the command run-test[s] with or without the @ character. "--all" executes all available test scenarios.