<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
                         "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter>
    <title>Optimizer Component Usage</title>
    <sect1>
        <title>Synopsis</title>
        <para>The Optimizer component allows the optimization of design variables in a
            workflow.</para>
    </sect1>
    <sect1>
        <title>Rationale</title>
        <para>The Optimizer component uses a black-box optimization software library. By starting
            the component, an input file for the selected optimization algorithm is created and the
            software will be started in the background. Different optimization packages can be
            installed.</para>
        <para>One package that is delivered with RCE on Windows x64 and Linux x64 machines is
            Dakota. Dakota was tested on the following distributions: Windows 10, Windows Server
            2019, Ubuntu 18.04, and CentOS 8. </para>
        <para>For more information about the Dakota Package see the <ulink
                url="https://dakota.sandia.gov/">Dakota Project page</ulink>.</para>
    </sect1>
    <sect1>
        <title>Usage</title>
        <para>To use the optimizer component you need to do the following steps:</para>
        <itemizedlist>
            <listitem>
                <para>In the algorithm tab you can choose an optimization algorithm that fits your
                    problem. There are several algorithms from the Dakota package available.</para>
                <para><inlinegraphic fileref="../../images/image001.png"/></para>
                <para>There are properties for each algorithm. For editing, click on the 'Algorithm
                    Properties' button. The appearing dialog shows the properties of the chosen
                    algorithm. The properties differ from algorithm to algorithm. For more
                    information about the properties see the documentation of the package.</para>
                <para>If you have an operating system, on which the default Dakota does not work,
                    you have the option to choose a custom Dakota binary by checking the box 'Use
                    custom dakota binary'. You will be asked for the Dakota executable path at
                    workflow start. This can be either a downloaded version from the dakota website
                    or a self compiled binary with the source code from the Dakota website. For more
                    information about compiling Dakota see: <ulink
                        url="https://software.sandia.gov/trac/dakota/wiki/Developer"
                        >https://software.sandia.gov/trac/dakota/wiki/Developer</ulink></para>
            </listitem>
            <listitem>
                <para>The next step is to define the inputs and outputs for the component. There are
                    three types of data you can configure.</para>
                <para>
                    <orderedlist>
                        <listitem>
                            <para>The objective function variables are the one to be optimized. For
                                each variable you can specify if it should be minimized, be
                                maximized or be searched for a specific value (solve for). If you
                                have more than one objective function, you can define their weight
                                in the optimization process. If there is only one objective
                                function, the weight will be ignored. Note that some algorithms
                                support single- and multi-objective optimization.</para>
                            <para>If you have defined some design variables, you also can choose if
                                the objective function you create has gradients or not. If you
                                select this, new inputs will appear in the connection editor, which
                                are intended for the values from the gradients. Note that for every
                                design variable you have, a new input for the objective function
                                exists.</para>
                        </listitem>
                        <listitem>
                            <para>The constraint variables are used to bound particular variables to
                                a region or value. If a solution is found but it causes a constraint
                                variable to be out of bounds, the solution is not valid. Again, the
                                constraints can have gradients.</para>
                        </listitem>
                        <listitem>
                            <para>The design variables are the values that are modified by the
                                algorithm to find an optimal solution. For each variable you have to
                                define a startvalue which will be the initial value for the
                                optimization algorithm. You also have to define the lower and upper
                                bound for each variable.</para>
                        </listitem>
                    </orderedlist>
                </para>
                <para>All data have to be either a float or a vector data type.</para>
                <para>
                    <note>
                        <para>Some Dakota algorithms do not support discrete optimization. For
                            discrete design variables the following algorithms are available:<itemizedlist>
                                <listitem>
                                    <para>Dakota Coliny Evolutionary Algorithm</para>
                                </listitem>
                                <listitem>
                                    <para>Dakota Multi Objective Genetic Algorithm</para>
                                </listitem>
                                <listitem>
                                    <para>Dakota NOMAD</para>
                                </listitem>
                                <listitem>
                                    <para>Dakota Single Objective Genetic Algorithm</para>
                                </listitem>
                            </itemizedlist> All other Dakota methods will ignore discrete design
                            variables during optimization. </para>
                    </note>
                </para>
                <para>There are three more tables for endpoints. They are just read-only and can not
                    be modified. They are created automatically when configuring the
                    Optimizer.</para>
                <para>
                    <orderedlist>
                        <listitem>
                            <para>The 'Start value inputs' table shows which design variables need
                                start values before running. Start values can be the starting value
                                for this design variable, if the option 'Has start value' in the
                                design variable dialog is not chosen.</para>
                            <para>Other possible start values are the lower and upper bounds of a
                                design variable, if the option 'Has unified bounds' is not
                                selected.</para>
                        </listitem>
                        <listitem>
                            <para>In the 'Optimal design variables outputs' table, the outputs for
                                the values of the optimized point are shown.</para>
                        </listitem>
                        <listitem>
                            <para>The 'Gradients' table shows which objective functions should have
                                gradient inputs as well. This is chosen in the dialog for the
                                objective functions ('Has gradient').</para>
                        </listitem>
                    </orderedlist>
                </para>
            </listitem>
        </itemizedlist>
        <para>After these steps the optimizer component is ready to start. In a running workflow,
            you are able to see the output from the Dakota optimizer in the Workflow console.</para>
        <para>By double clicking the Optimizer component in the runtime view of the workflow, you
            will get the values the optimizer produces and the possibility to export these values to
            an Excel file. You are also able to plot a graph with the given results in the diagram
            tab. </para>
        <para><inlinegraphic fileref="../../images/image005.png"/></para>
        <para>For help concerning nested and fault-tolerant loop settings, see the section
            'Usage/Workflows' in the user guide.</para>
        </sect1>
        <sect1>
        <title>Optimization Algorithm API</title>
        <para>Manual on how to use the Optimization Algorithm API. </para>
        <sect2>
            <title>Basic Concept</title>
            <para>The RCE Optimization Algorithm API enables the user to integrate their own optimization algorithms into RCE and use them in the common Optimizer Component. The
                API is based on Python, so the user's algorithm must be callable from phyton, e.g. through a system call.</para>
        </sect2>
        <sect2>
            <title>How to integrate an algorithm into RCE</title>
            <para>The location where the API looks for integrated algorithms is in RCE's profile folder:
                <code>&lt;profile&gt;/integration/optimizer/</code>.
                Below this path, every subfolder must have a specific structure in order to be recognized as an optimizer integration. Each subfolder must contain two folders with the names 
                "gui_properties_configuration" and "source". 
                An example integration "example_algorithm" is available at the installation path of RCE in the subfolder
                "examples/optimization_algorithm_api/optimizer". Copy it to your profile and you can use this algorithm in RCE.</para>
            <sect3>
                <title>GUI Properties Definition</title>
                <para>Within the "gui_properties_configuration" folder, the GUI of the Optimizer Component must be configured for the integrated algorithms. At
                    first, there has to be a file named "algorithms.json". In this file, RCE looks for the algorithms to
                    be defined. The file is structured as follows:</para>
                <programlisting>{
"Name of algorithm":"name of json file for algorithm"
}</programlisting>
                <para> For example:</para>
                <programlisting>{
"Name of method1" : "method1",
"Name of method2" : "method2"
}</programlisting>
                <para>where "method1.json" and "method2.json" exist in the same directory. </para>
                <para>The method files also have to be in a certain format which looks like this:</para>
                <programlisting>{
    "methodName":"Name of method",
    "optimizerPackage":"generic",
    "specificSettings":{
        "propertyName":{
            "GuiName":"Name shown in optimizer GUI",
            "dataType": ["Int" | "Real" | "String" | "Bool"],
            "SWTWidget": ["Text" | "Combo" | "Check"],
            "DefaultValue": "",
            "Value":"",
            "Validation":""
        }
    }
}</programlisting>
                <para>The "optimizerPackage" must always have the value "generic" and the
                    "methodName" must have the same value as defined in the "algorithms.json". In
                    the section "specificSettings", properties can be defined in order to make them
                    configurable in the RCE GUI. You can choose between three different types of GUI
                    widgets and four different data types. The properties will be displayed when you
                    open the "Algorithm properties..." view in the RCE's Algorithm section of the
                    Optimizer component. Three categories are available to organize the properties
                    on different tabs: "commonSettings","specificSettings" and
                    "responsesSettings".</para>
                <para>Every property must have the following fields:</para>
                <para><emphasis>GuiName:</emphasis> The name that is displayed in the "Algorithm properties..." view and
                    describes the property.</para>
                <para><emphasis>dataType:</emphasis> The data type of the current property. Valid values are:</para>
                <itemizedlist>
                    <listitem><para><emphasis role="bold">Int</emphasis>: an integer number </para></listitem>
                    <listitem><para><emphasis role="bold">Real</emphasis>: a float number </para></listitem>
                    <listitem><para><emphasis role="bold">String</emphasis>: a text </para></listitem>
                </itemizedlist>
                <para><emphasis>SWTWidget:</emphasis> This value defines what kind of
                    GUI element is used for the property. Valid values are:</para>
                <itemizedlist>
                    <listitem><para><emphasis role="bold">Text</emphasis>: a text box where the user can enter any string</para></listitem>
                    <listitem><para><emphasis role="bold">Combo</emphasis>: a dropdown menu with pre defined values. When using
                            the Combo, you have to define the values to be shown, using: <itemizedlist>
                                <listitem>
                                    <para><emphasis role="bold">Choices</emphasis>: A comma
                                        separated list of the values, e.g. "Option 1, Option 2"
                                    </para>
                                </listitem>
                            </itemizedlist></para></listitem>
                    <listitem><para><emphasis role="bold">Check</emphasis>: a checkbox to select an option </para></listitem>
                </itemizedlist>
                <para><emphasis>DefaultValue:</emphasis> The default value that is chosen
                    if the user does not manually enter a value for the property. For Combos, this must
                    be one of the "Choices".</para>
                <para><emphasis>Value:</emphasis> The value must always be an empty string ("").</para>
                <para><emphasis>Validation:</emphasis> For Int or Real data types you can add a validation. Possible validations are:</para>
                <itemizedlist>
                    <listitem><para>&gt;, &gt;=, &lt;, &lt;= followed by a number, e.g. "&gt;=0"</para></listitem>
                    <listitem><para>"required" or "optional" if a value must be entered or can be empty</para></listitem>
                    <listitem><para>empty string "" if no validation is required </para></listitem>
                </itemizedlist>
                <para>All required and configurable properties for the integrated algorithm should
                    be defined in a json file using the format described above. Apart from that, no
                    further adjustments are necessary in the "gui_properties_configuration"
                    folder.</para>
                <para>You will find a more detailed example json file at the end of this manual. (cf. <xref linkend="example"/>)</para>
            </sect3>
            <sect3>
                <title>Source Folder</title>
                <para>In the "source" folder, the algorithm logic must be defined. Two files are mandatory, which will be
                    the entry point for the Optimizer Component. One file must be named "python_path", which only contains one single line that points to the executable of a python installation. 
                    The other file must be named "generic_optimizer.py". This script must call your own optimizer method. 
                    In this script you can use the Optimizer Algorithm API. The API contains three modules. Import the modules as follows:</para>
                <programlisting>from RCE_Optimizer_API import configuration
from RCE_Optimizer_API import evaluation
from RCE_Optimizer_API import result</programlisting>
                <para><emphasis role="bold">Module Description</emphasis>:</para>
                <itemizedlist>
                    <listitem><para><emphasis role="italic">configuration</emphasis>: This module contains all information that is
                            needed to configure the optimization method. You can get the design
                            variables names and counts and the objective names and weigths.
                            Furthermore you can access the property values configured by the user in
                            the GUI.</para></listitem>
                    <listitem><para><emphasis role="italic">evaluation</emphasis>: Use this module you start an evaluation run in
                            RCE and get the result of each evaluation and end the optimizer.</para></listitem>
                    <listitem><para><emphasis role="italic">result</emphasis>: If an evaluation is done, it generates a new result
                            object. It contains objective and constraint values, their gradients and
                            the failed inputs from RCE and provides methods to access them. The
                            result object is lost at the next evaluation unless it is explicitly
                            saved somewhere.</para>
                        <para>For detailed information on the modules and the included methods see
                            Section <xref linkend="module_description"/></para></listitem>
                </itemizedlist>
            </sect3>
            <sect3 id="example">
                <title>Example GUI configuration json</title>
                <programlisting>{
    "methodName" : "Name of method",
    "optimizerPackage" : "generic",
    "commonSettings" : {
        ...
    },
    "specificSettings" : {
        "textExample" : {
            "GuiName" : "Enter Value:",
            "GuiOrder" : "1",
            "dataType" : "Real",
            "SWTWidget" : "Text",
            "DefaultValue" : "1.0",
            "Value" : "",
            "Validation" : "&lt;=1.0"
        },
        "comboExample" : {
            "GuiName" : "Select parameter",
            "GuiOrder" : "2",
            "dataType" : "String",
            "SWTWidget" : "Combo",
            "Choices" : "choice1,choice2,choice3",
            "DefaultValue" : "choice1",
            "Value" : ""
        },
        "checkboxExample" : {
            "GuiName" : "Any flag:",
            "GuiOrder" : "3",
            "dataType" : "Bool",
            "SWTWidget" : "Check",
            "DefaultValue" : "false",
            "Value" : ""
        }
    },	
    "responsesSettings" : {
        ...
    }
}</programlisting>
                <note>
                    <para>Property names must be unique on each tab. Otherwise the last configuration is used.</para>
                </note>
                <note><para>The field "GuiOrder" is optional. Use this field to specify or change easily the order of the widgets in the GUI.</para></note>
            </sect3>
        </sect2>
            <sect2 id="module_description">
        <title>Module Description</title>
        <para>
            <table frame="all">
                <title>configuration.py</title>
                <tgroup cols="2">
                    <colspec colname="c1" colnum="1" colwidth="1*"/>
                    <colspec colname="c2" colnum="2" colwidth="1.64*" align="left"/>
                    <thead>
                        <row>
                            <entry>Method</entry>
                            <entry>Description</entry>
                        </row>
                    </thead>
                    <tbody>
                        <row>
                            <entry>def get_algorithm()</entry>
                            <entry>Returns the selected algorithm</entry>
                        </row>
                        <row>
                            <entry>def get_design_variable_count()</entry>
                            <entry>Returns the number of design variables</entry>
                        </row>
                        <row>
                            <entry>def get_design_variable_names()</entry>
                            <entry>Returns a list of variable names</entry>
                        </row>
                        <row>
                            <entry>def get_design_variable_max_values()</entry>
                            <entry>Returns a dictionary of the variables and their corresponding
                                upper bound</entry>
                        </row>
                        <row>
                            <entry>def get_design_variable_min_values()</entry>
                            <entry>Returns a dictionary of the variables and their corresponding
                                lower bound</entry>
                        </row>
                            <row>
                                <entry>def get_design_variable_max_value(variable_name)</entry>
                                <entry>Returns the upper bound of the given variable
                                    "variable_name"</entry>
                            </row>
                            <row>
                                <entry>def get_design_variable_min_value(variable_name)</entry>
                                <entry>Returns the lower bound of the given variable
                                    "variable_name"</entry>
                            </row>
                        <row>
                            <entry>def get_start_values()</entry>
                            <entry>Returns a dictionary of the variables and their corresponding
                                start values</entry>
                        </row>
                        <row>
                            <entry>def get_start_value(variable_name)</entry>
                            <entry>Returns the start value of the given variable
                                "variable_name"</entry>
                        </row>
                        <row>
                            <entry>def get_step_values()</entry>
                            <entry>Returns a dictionary of the variables and their corresponding
                                start values</entry>
                        </row>
                        <row>
                            <entry>def is_discrete_variable(variable_name)</entry>
                            <entry>Returns whether the given variable "variable_name" is discrete or
                                not</entry>
                        </row>
                        <row>
                            <entry>def get_constraint_names()</entry>
                            <entry>Returns a list of constraint names</entry>
                        </row>
                        <row>
                            <entry>def get_constraint_max_values()</entry>
                            <entry>Returns a dictionary of the constraints and their corresponding
                                upper bound</entry>
                        </row>
                        <row>
                            <entry>def get_constraint_min_values()</entry>
                            <entry>Returns a dictionary of the constraints and their corresponding
                                lower bound</entry>
                        </row>
                            <row>
                                <entry>def get_constraint_max_value(constraint_name)</entry>
                                <entry>Returns the upper bound of the given constraint
                                    "constraint_name"</entry>
                            </row>
                            <row>
                                <entry>def get_constraint_min_value(constraint_name)</entry>
                                <entry>Returns the lower bound of the given constraint
                                    "constraint_name"</entry>
                            </row>
                        <row>
                            <entry>def get_objective_names()</entry>
                            <entry>Returns a list of objectives names</entry>
                        </row>
                        <row>
                            <entry>def get_objective_weights()</entry>
                            <entry>Returns a dictionary of the objectives and their corresponding
                                weight</entry>
                        </row>
                        <row>
                            <entry>def get_optimization_targets()</entry>
                            <entry>Returns a dictionary of the objectives and their corresponding
                                optimization target</entry>
                        </row>
                        <row>
                            <entry>def get_optimization_target(name)</entry>
                            <entry>Returns the optimization target of the given objective "name" or
                                None if  an objective "name" does not exists</entry>
                        </row>
                        <row>
                            <entry>def get_properties()</entry>
                            <entry>Returns a dictionary of the properties and their corresponding
                                values</entry>
                        </row>
                        <row>
                            <entry>def get_property(name)</entry>
                            <entry>Returns the value of the given property "name" or None if a
                                property "name" does not exists</entry>
                        </row>
                        <row>
                            <entry>def get_property_keys()</entry>
                            <entry>Returns a list of property keys</entry>
                        </row>
                    </tbody>
                </tgroup>
            </table>
        </para>
        <para>
            <table frame="all">
                <title>evaluation.json</title>
                <tgroup cols="2">
                        <colspec colname="c1" colnum="1" colwidth="1*"/>
                        <colspec colname="c2" colnum="2" colwidth="1.64*"/>
                        <thead>
                            <row>
                                <entry>Method </entry>
                                <entry>Description</entry>
                            </row>
                        </thead>
                        <tbody>
                            <row>
                                <entry>def evaluate(number_evaluation, design_variables,
                                    grad_request = False)</entry>
                                <entry>Starts the evaluation run with the given run number, designs
                                    variables and a boolean whether gradients are requested or not
                                    (default value is False). The result object of the current run
                                    is returned. <para>Note: The design variables are handed over in
                                        an alphabetically sorted list as displayed in the Properties
                                        view of RCE's GUI. Be aware that uppercase is treated before
                                        lowercase variable names. Vectors are passed entry-wise.
                                        </para><para>Example: Given the following design variables
                                        "Var1", "Vec" and "var2" with the values 1, [2,3,4] and 2 in
                                        evaluation run number 5. Start the evaluation run with
                                            <emphasis role="italic"
                                            >evaluate(5,[1,2,3,4,2])</emphasis>.</para></entry>
                            </row>
                            <row>
                                <entry>def finalize(optimal_evaluation_number)</entry>
                                <entry>Ends the Optimization with the given run number as the
                                    optimal run. The optimal values are written to the
                                    outputs.</entry>
                            </row>
                        </tbody>
                    </tgroup>
            </table>
        </para>
        <para>
            <table frame="all">
                <title>result.py</title>
                <tgroup cols="2">
                        <colspec colname="c1" colnum="1" colwidth="1*"/>
                        <colspec colname="c2" colnum="2" colwidth="1.64*"/>
                        <thead>
                            <row>
                                <entry>Method</entry>
                                <entry>Description</entry>
                            </row>
                        </thead>
                        <tbody>
                            <row>
                                <entry>def get_constraint_gradient(constraint_name)</entry>
                                <entry>Returns the gradient value of the given constraint
                                    "constraint_name" or None if the constraint does not exists or
                                    no gradient is defined</entry>
                            </row>
                            <row>
                                <entry>def get_constraint_value(constraint_name)</entry>
                                <entry>Returns the value of the given constraint "constraint_name"
                                    or None if  a constraint "constraint_name" does not
                                    exists</entry>
                            </row>
                            <row>
                                <entry>def get_failed()</entry>
                                <entry>Returns a list of the failed optimization runs</entry>
                            </row>
                            <row>
                                <entry>def get_objective_gradient(objective_name)</entry>
                                <entry>Returns the gradient value of the given objective
                                    "objective_name" or None if the objective does not exists or no
                                    gradient is defined</entry>
                            </row>
                            <row>
                                <entry>def get_objective_value(objective_name)</entry>
                                <entry>Returns the value of the given objective "objective_name" or
                                    None if the objective does not exists</entry>
                            </row>
                            <row>
                                <entry>def has_gradient(name)</entry>
                                <entry>Returns True if the given input "name" has a gradient
                                    defined, False otherwise</entry>
                            </row>
                        </tbody>
                    </tgroup>
            </table>
        </para>
    </sect2>
        </sect1>
</chapter>
