The RCE Event Log provides a high-level summary of events that are relevant to users and administrators of RCE instances.
Unlike a low-level technical log file, the Event Log is meant to aggregate events as much as possible. Where a technical log may list several steps related to an action or event, the Event Log is meant to focus on semantic results and final events, and condense them into a minimal number of entries.
For example, a technical log would typically represent the related events "user requested pause", followed by "pausing", and then "paused" as separate entries, often with other events mixed in-between.
The Event Log, in contrast, would represent this example as a single "paused" event. The details surrounding this event are collected and represented as attributes. For example, the "paused" event could have a "reason" attribute with a value of "initiated by user", and an optional "initiated by" attribute providing the user’s identity information. This design allows users to quickly understand what is happening in the system without associating multiple log events.
Of course, not all events can be merged this way. For example, the start of a network connection should be logged once it is established, as it is unknown when it will terminate. In these such cases, providing one or more unique association id(s) as event attribute(s) is recommended. If the event log is consumed using specialized clients, events can be automatically associated using these ids. Even if the event log is consumed without such tools, these association ids can still be easily found by using text-based search or filtering.
In the current implementation, all event entries are appended to two files in the profile directory.
The events-readable.log file is meant for human inspection, and provides more explicit titles for event types and attributes.
This file can be read in RCE by opening it via "File > Open", or with any external text file reader just like a .txt file.
The events-compact.log file, on the other hand, is meant for automated parsing.
It uses technical ids for the event types and their attributes, and represents each event as a single line of UTF-8 JSON.
Event timestamps are represented differently in those files.
In events-readable.log, times are formatted in a human-readable way, in the form of <YYYY>-<MM>-<DD>T<hh>:<mm>:<ss>.<mmm>.
This format is based on ISO 8601, with milliseconds accuracy, using "unqualified local time", i.e., no time zone information but using local time.
The latter choice was made to improve human readability when inspecting raw event data, at the cost of potential "jumps in time" on changes in the local UTC offset, e.g., "winter" time vs. "summer" time/DST.
In events-compact.log, on the other hand, timestamps are encoded in the "_ts" JSON field as milliseconds from 1970-01-01T00:00:00Z.
Unlike other RCE log files, these files are intentionally not cleared on the restart of an instance, in order to provide a long-term protocol of system activity. For example, if an RCE instance is running as a system daemon, even repeated restarts of the host system will not erase older event data.
Note that both file formats are preliminary, and the scope of event types and attributes is not fully implemented yet. Notably, most workflow and component/tool execution events are not being logged yet.
This section lists the specified event types and their possible attributes.
The human-readable titles of the types and attributes (listed first) are what appears in the events-readable.log file.
Their technical ids (shown in a smaller font and parantheses) are used in the events-compact.log file, with the intention of more long-term stability.
| Event Type (Title / Id / Description) | Attribute (Title / Id) | Attribute Description |
|---|---|---|
Application Starting The application is starting and has passed basic initialization, including profile selection. | User home directory (user_home) | The "home" directory of the user running the application. |
User system id (user_name) | The technical/system name of the user running the application. | |
Profile location (profile_location) | The file system location of the profile that the application is using. | |
Working directory (work_dir) | The selected "working directory" when starting the application. | |
RCE version id (rce_version) | The full RCE version string. | |
OS name and version (os_name) | The name of the underlying operating system, as reported by the JRE. | |
JVM version (jvm_version) | The Java version, as reported by the JRE. | |
Application Shutdown Requested A user or technical process has requested this instance to shut down, e.g. via GUI, console command, or system shell command. If available, information about the cause should be logged as "Shutdown trigger"/"method". | Shutdown trigger (method) | The kind of event that initiated/requested the shutdown. Currently not strictly specified, for informational use only. Current examples are "console command", "CLI/network signal", and null (unspecified) |
Part of a restart (is_restart) | "yes" if the shutdown request is part of a restart request, i.e., a new application session will start after shutdown. | |
Application Terminating A marker event logged at the latest possible time during a regular shutdown of the application. | - (no attributes) | |
Server Port Opened A network server port (typically TCP) was opened by the application. | Port type (type) | The technical type and/or protocol of the server port. |
IP port number (port) | The IP port number. | |
IP bind address (bind_ip) | The IP address the port is bound to, which affects from which network interfaces it can be accessed. | |
Server Port Closed A network server port (typically TCP) was closed by the application. | Port type (type) | The technical type and/or protocol of the server port. |
IP port number (port) | The IP port number. | |
IP bind address (bind_ip) | The IP address the port was bound to. | |
Incoming Connection Accepted An incoming connection has been successfully established. This should be logged as late as reasonably possible to avoid "accepted" events where the connection is immediately closed again due to a validation or version mismatch error. Mutually exclusive with the "connection.incoming.refused" event, but may be reached after one or more "connection.incoming.auth.failed" events if the server allows multiple login attempts. | Connection type (type) | The technical type and/or protocol of the connection. Usually equal to the server port’s type. |
Connection id (connection_id) | An association id for this connection. The only assumptions that should be made about its content is that it is a string of "reasonable" length, not empty, suitable for log output, and unique within the application’s session. | |
Remote node id (remote_node_id) | The id of the remote RCE node, if available from the protocol type’s connection process. This id can by any of the four supported id types, but will typically be an "instance session id". | |
Login/user name (login_name) | The user/login/account identifier successfully used for authentication/authorization, if applicable. | |
Authentication method (auth_method) | The kind of authentication/authorization (e.g. passphrase or private key) that was successfully used, if applicable. | |
Failure count (auth_failure_count) | The number of failed authentication/authorization attempts for this connection, if applicable. | |
Remote IP address (remote_ip) | The remote IP address of the incoming connection. | |
Remote IP port (remote_port) | The remote IP port number of the incoming connection. | |
Server IP port (server_port) | The local server IP port number that the incoming connection is connected to. | |
Incoming Connection Failed To Authenticate As part of establishing an incoming connection, an authentication/authorization attempt was made but failed. Whether this failure is fatal for the overall connection attempt or not is implementation-specific. If this is ultimately followed up by a successful authentication/authorization attempt (in case another attempt was allowed in the first place), this event should be succeeded by a "connection.incoming.accepted" event. Otherwise, this should always be succeeded by a "connection.incoming.refused" event. | Connection type (type) | The technical type and/or protocol of the connection. Usually equal to the server port’s type. |
Connection id (connection_id) | An association id for this connection. The only assumptions that should be made about its content is that it is a string of "reasonable" length, not empty, suitable for log output, and unique within the application’s session. | |
Login/user name (login_name) | The user/login/account identifier used for the failed authentication/authorization attempt. | |
Authentication method (auth_method) | The kind of authentication/authorization (e.g. passphrase or private key) that was attempted. | |
Failure reason (auth_failure_reason) | The reason why authentication/authorization failed (e.g., wrong password or unknown user). | |
Failure count (auth_failure_count) | The number of failed authentication/authorization attempts for this connection. | |
Remote IP address (remote_ip) | The remote IP address of the incoming connection. | |
Remote IP port (remote_port) | The remote IP port number of the incoming connection. | |
Server IP port (server_port) | The local server IP port number that the incoming connection is connected to. | |
Incoming Connection Refused An incoming connection has failed to complete its login process or has been refused for some other reason. Mutually exclusive with the "connection.incoming.accepted" and "connection.incoming.closed" events. Both incorrect authentication attempts as well as authentication timeouts (e.g. when an SSH client makes no authentication attempt at all) are both represented by this event. These sub-types can be distinguished by the "last_auth_failure_reason" attribute. | Connection type (type) | The technical type and/or protocol of the connection. Usually equal to the server port’s type. |
Connection id (connection_id) | An association id for this connection. The only assumptions that should be made about its content is that it is a string of "reasonable" length, not empty, suitable for log output, and unique within the application’s session. | |
Reason (close_reason) | The human-readable reason why this connection was refused. | |
Last login/user name (last_login_name) | The user/login/account identifier used for the last failed authentication/authorization attempt, if applicable. | |
Last auth. failure (last_auth_failure_reason) | The last reason why authentication/authorization failed (e.g., wrong password or unknown user), if applicable. | |
Last auth. method (last_auth_method) | The last kind of authentication/authorization (e.g. passphrase or private key) that was attempted, if applicable. | |
Auth. failure count (auth_failure_count) | The number of failed authentication/authorization attempts for this connection, if applicable. | |
Remote IP address (remote_ip) | The remote IP address of the incoming connection. | |
Remote IP port (remote_port) | The remote IP port number of the incoming connection. | |
Server IP port (server_port) | The local server IP port number that the incoming connection is connected to. | |
Duration (duration) | The duration (in msec) that this connection was open/established for. The precise start and end times for this calculation are implementation- and type-dependent. | |
Incoming Connection Closed An incoming connection has been closed, either by the client, the server, or a network event. This event is always preceded by a "connection.incoming.accepted" event. Mutually exclusive with the "connection.incoming.refused" event. | Connection type (type) | The technical type and/or protocol of the connection. Usually equal to the server port’s type. |
Connection id (connection_id) | An association id for this connection. The only assumptions that should be made about its content is that it is a string of "reasonable" length, not empty, suitable for log output, and unique within the application’s session. | |
Remote node id (remote_node_id) | The id of the remote RCE node, if available from the protocol type’s connection process. This id can by any of the four supported id types, but will typically be an "instance session id". | |
Reason (close_reason) | The human-readable reason why this connection was closed. | |
Remote IP address (remote_ip) | The remote IP address of the incoming connection. | |
Remote IP port (remote_port) | The remote IP port number of the incoming connection. | |
Server IP port (server_port) | The local server IP port number that the incoming connection is connected to. | |
Duration (duration) | The duration (in msec) that this connection was open/established for. The precise start and end times for this calculation are implementation- and type-dependent. | |
Incoming Uplink Connection Accepted After the login credentials for an incoming SSH connection were accepted, this subsequent event indicates successful completion of the Uplink protocol handshake, too. This includes protocol compatibility validation and the successful assignment of an unused Uplink namespace. | Uplink session id (session_id) | The Uplink-specific session id. |
SSH connection id (connection_id) | The SSH connection id for correlation with "connection.incoming.*" events. | |
Client software id (client_version) | The software version of the Uplink client; includes information about the client software used (e.g., "rce/…"). | |
Protocol version (protocol_version) | The Uplink protocol version being used for the connection/session, based on the initial client-server handshake. | |
Effective login name (login_name) | The final/effective login name used. Due to namespace mapping constraints, this may be different from the original login name; see "original_login_name". | |
Original login name (original_login_name) | The login name requested by the client, before any modifications by the server. Omitted if equal to the effective login name (see "login_name"). | |
Effective client id (client_id) | The final/effective client id used. Due to namespace mapping constraints, this may be different from the original client id; see "original_client_id". | |
Original client id (original_client_id) | The client id requested by the client, before any modifications by the server. Omitted if equal to the effective client id (see "client_id"). | |
Assigned namespace (namespace) | The namespace assigned to this client for Uplink destination address mapping. Typically related to the session id. | |
Incoming Uplink Connection Refused After the login credentials for an incoming SSH connection were accepted, this subsequent event indicates failure of the Uplink protocol handshake. This may be due to a version incompatibility or the desired namespace already being used. The next event logged after this should be "connection.incoming.closed", as the SSH connection was already "accepted". Mutually exclusive with "uplink.incoming.accepted" and "uplink.incoming.closed". | Uplink session id (session_id) | The Uplink-specific session id. |
SSH connection id (connection_id) | The SSH connection id for correlation with "connection.incoming.*" events. | |
Client software id (client_version) | The software version of the Uplink client; includes information about the client software used (e.g., "rce/…"). | |
Protocol version (protocol_version) | The protocol version requested by the client. May or may not be a version that this server supports. | |
Original login name (login_name) | The login name used by the client, without any mapping modification that would have been made on success. | |
Original client id (client_id) | The client id sent by the client, without any mapping modification that would have been made on success. | |
Reason (reason) | The human-readable reason for refusing the Uplink session. | |
Incoming Uplink Connection Closed After a previous "uplink.incoming.accepted" event, this indicates the end of the application-level Uplink session. This event should be logged for any kind of connection termination, from graceful disconnect to low-level connection errors. Mutually exclusive with "uplink.incoming.refused". | Uplink session id (session_id) | The Uplink-specific session id. |
SSH connection id (connection_id) | The SSH connection id for correlation with "connection.incoming.*" events. | |
Final connection state (final_state) | The final (technical) state of the Uplink connection; indicates the reason for terminating the session. | |
Network Node Discovered/Named Either a new node in the logical RCE network has been discovered, or an existing node has changed its title. Relevant for determining which technical node was behind a given display name at a certain time. Besides general logging, this is also a basic security trail for potential node "impersonation" attempts. | Persistent instance id (instance_id) | The persistent part (the "instance node id") of the logical RCE-specific network address. |
Instance session id (session_id) | The per-session suffix of the of the logical RCE-specific network address. Used to distinguish runs of the same instance, i.e., the session id always changes on an instance’s restart. | |
Logical sub-node id (logical_sub_node_id) | The logical "sub-node" selector within the logical RCE-specific network address. If absent, then this event represents a change of the "root" name of an instance/node. If present, then a logical "sub-node" has changed its specific name, while the "root" name was unchanged. | |
Announced name (name) | The title/name of the RCE instance, as defined by its user or administrator. | |
Is local/own node (is_local_node) | \"yes\" if the observed node is the local node. This is a convenience property to simplify event filtering. | |
Workflow Request Initiated (Request Sent) A workflow run was initiated from the local node. The designated workflow controller may be either the local or a remote node. | Workflow run id (workflow_run_id) | The id string assigned to this workflow run. Can be considered globally unique for all practical purposes. |
Workflow controller node (workflow_controller_node) | The logical node id of the node designated to run the workflow controller. | |
Controller is local (workflow_controller_is_local_node) | A convenience attribute designating whether the workflow controller node is the local node. | |
Workflow file path (local_workflow_file) | The path of the local workflow file that was submitted for execution, if applicable; OTHERWISE absent. | |
Workflow metadata (WIP) (workflow_metadata) | TODO specify contents; structured data | |
success (success) | Whether the workflow was successfully initiated, i.e., whether the request was made and accepted by the designated execution node. | |
Workflow Execution Requested (Request Received) A workflow run was requested, and an attempt was made to initialize its workflow controller on the local node. If controller initialization fails, this event MUST still be logged to make the request visible, as the ".initiated" event may have been logged be on a remote node. | Workflow run id (workflow_run_id) | The id string assigned to this workflow run. Can be considered globally unique for all practical purposes. |
Start timestamp (start_ts) | On success, the canonical start time of the workflow, which will usually be close, but not necessarily equal to the timestamp of this event; OTHERWISE absent. (TODO specify format) | |
Initiator node (initiator_node) | The logical session node id of the node that initiated this workflow run. Note that in RCE 10.x and earlier, this value is NOT strongly verified, and should be considered informational only. | |
Initiator is local (initiator_is_local_node) | A convenience attribute designating whether the node that initiated this workflow run is the local node. Note that in RCE 10.x and earlier, this value is NOT strongly verified, and should be considered informational only. | |
Workflow metadata (WIP) (workflow_metadata) | TODO specify contents; structured data | |
success (success) | Whether the workflow was successfully initialized. | |
Workflow Execution Completed The end of a local workflow controller’s execution. (TODO clarify whether this may be logged on a failed ".requested" event or not.). | Workflow run id (workflow_run_id) | The id string assigned to this workflow run. Can be considered globally unique for all practical purposes. |
End timestamp (end_ts) | The canonical end time of the workflow, which will usually be close, but not necessarily equal to the timestamp of this event. (TODO specify format) | |
Duration (duration) | A convenience attribute specifying the duration of this workflow run. (TODO specify format) | |
Final workflow state (final_state) | The final state of the workflow, as defined by the workflow engine. | |
Workflow Request Completed The final event of a workflow run that was initiated by the local node. (TODO clarify whether this may be logged on a failed ".initiate" event or not.). | Workflow run id (workflow_run_id) | The id string assigned to this workflow run. Can be considered globally unique for all practical purposes. |
Final workflow state (final_state) | The final state of the workflow, as defined by the workflow engine. | |
Component/Tool Execution Initiated (Request Sent) A workflow component run was initiated from the local node, which is always the workflow controller. The node controlling the component’s execution may be the local or a remote one. | Workflow run id (workflow_run_id) | The id string assigned to this workflow run. Can be considered globally unique for all practical purposes. |
Component run id (component_run_id) | The id string assigned to this component run. Unique within the scope of the associated workflow. | |
Executing node (execution_controller_node) | The logical session node id of the node designated to be this component run’s execution controller. | |
Executing node is local (execution_controller_is_local_node) | A convenience attribute designating whether this component’s run execution controller node is the local node. | |
Component/Tool Execution Requested A workflow component run was requested, and an attempt was made to initialize its controller on the local node. If controller initialization fails, this event MUST still be logged to make the request visible, as the ".initiated" event may have been logged be on a remote workflow controller node. | Workflow run id (workflow_run_id) | The id string assigned to this workflow run. Can be considered globally unique for all practical purposes. |
Component run id (component_run_id) | The id string assigned to this component run. Unique within the scope of the associated workflow. | |
Workflow controller node (workflow_controller_node) | The logical session node id of the node running the workflow controller. Note that in RCE 10.x and earlier, this value is NOT strongly verified, and should be considered informational only. | |
Workflow contr. is local (workflow_controller_is_local_node) | A convenience attribute designating whether the workflow controller node is the local node. Note that in RCE 10.x and earlier, this value is NOT strongly verified, and should be considered informational only. | |
Start timestamp (start_ts) | On success, the canonical start time of the component run, which will usually be close, but not necessarily equal to the timestamp of this event; OTHERWISE absent. (TODO specify format) | |
Component/Tool Execution Completed The end of a local workflow controller’s execution. (TODO clarify whether this may be logged on a failed ".requested" event or not.). | Workflow run id (workflow_run_id) | The id string assigned to this workflow run. Can be considered globally unique for all practical purposes. |
Component run id (component_run_id) | The id string assigned to this component run. Unique within the scope of the associated workflow. | |
End timestamp (end_ts) | The canonical end time of the component run, which will usually be close, but not necessarily equal to the timestamp of this event. (TODO specify format) | |
Duration (duration) | A convenience attribute specifying the wall-clock duration of this component run. (TODO specify format) | |
Final component state (final_state) | The final state of the component, as defined by the workflow engine. | |
Component/Tool Execution Request Completed The final event of a component run that was initiated by a workflow controller running on the local node. (TODO clarify whether this may be logged on a failed ".initiate" event or not.). | Workflow run id (workflow_run_id) | The id string assigned to this workflow run. Can be considered globally unique for all practical purposes. |
Component run id (component_run_id) | The id string assigned to this component run. Unique within the scope of the associated workflow. | |
End timestamp (end_ts) | The canonical end time of the component run, which will usually be close, but not necessarily equal to the timestamp of this event. (TODO specify format) | |
Duration (duration) | A convenience attribute specifying the wall-clock duration of this component run. (TODO specify format) | |
Final component state (final_state) | The final state of the component, as defined by the workflow engine. | |
Login/Account Data Initialized A set of accounts has been initialized, typically at application or subsystem startup. | Account type (type) | The type of accounts that were initialized, e.g. "ssh". |
New number of accounts (number_of_accounts) | Indicates the total number of registered accounts after initialization. | |
Account data origin (origin) | Indicates the data source from which the initial account data was read. Absent if not applicable for the current account type. | |
Login/Account Data Updated A set of accounts has been updated at application runtime, after being already initialized. | Account type (type) | The type of accounts that were updated, e.g. "ssh". |
New number of accounts (number_of_accounts) | Indicates the total number of registered accounts after the update (NOT the number of changes!). | |
Account data origin (origin) | Indicates the data source from which the updated account data was read. Absent if not applicable for the current account type. | |
System Monitoring Initialized Indicates that the system monitoring subsystem was initialized and logs static system information. | Java process id (jvm_pid) | The PID of the main process running the JVM. Depending on the operating system, this process may either be the RCE executable or a separate Java process. |
Processor count (JVM) (jvm_processor_count) | The number of processors, as reported by the JVM. | |
Heap limit (JVM) (jvm_heap_limit) | The configured heap (Java RAM) limit, as reported by the JVM. | |
System RAM (native) (system_total_ram) | The total system RAM in bytes, as reported by the system monitoring library. | |
Logical CPUs (native) (system_logical_cpus) | The number of "logical CPUs", as reported by the system monitoring library. | |
Custom Event A custom event type, allowing extensions or plugins to make use of the event logging system. This event type is special in the way that its list of attributes is not pre-defined, except for "type". This allows custom events to log all kinds of extra attributes. Consequently, any event log validation code must have a special rule to accept those attributes. Note that while custom attribute keys are supported, there are still certain rules for them. For now, they must start with a-z, end with a-z or a digit, and not exceed the maximum length defined above. It is also recommended to keep them lowercase and dot-separated for consistency. | Custom event type id (type) | A custom type id describing the logged event. It should generally follow the naming pattern of the standard events, and plugins/extensions should strive to make those ids collision-free. |
It is possible that event types not listed here were/are/will be logged by past, current, or future RCE versions. Therefore, any automated code for parsing event log data MUST tolerate event types not listed here.