1. Documentation
2. Plug-Ins & Storage

4. Plug-Ins & Storage

4.1 Plug-ins & Server Files

All RapidContext server functionality is provided by plug-ins. The plug-in ZIP files are stored in the plugin/ directory on the server. The figure below illustrates the directory structure:

• plugin/
  • cmdline.zip — The command-line plug-in
  • http.zip — The HTTP plug-in
  • jdbc.zip — The JDBC plug-in
  • local/ — A plug-in with local modifications and additions
    • connection/ — Connections added or modified
    • lib/ — JAR files added (JDBC drivers)
    • role/ — Roles added or modified
    • session/ — Currently active sessions
    • user/ — Users added or modified
  • system.zip — The system or core platform plug-in

As seen above, most plug-ins are stored as packed ZIP files in the plugin/ directory. Only the local plug-in is stored in unpacked form by default. There are two special plug-ins that cannot be removed or unloaded:

• system — A plug-in that provides all the platform built-in functionality. In essence, a platform update consists of replacing this plug-in with a new version. This plug-in is always searched last, meaning that any file in it may be “shadowed” by a file with the same name from another plug-in.
• local — A plug-in that provides the local modifications and additions to RapidContext. It is the only read-write plug-in (all others are read-only) and is stored unpacked to enable direct modification to the files. This plug-in is always searched first, meaning that any file in it will always “shadow” a file with the same name in another plug-in.

The list of plug-ins to load on startup is stored in the config.properties file (in the plugin/local/ directory). The plug-in order in that file also controls the order in which plug-ins are loaded.

4.2 Storage Overview

The plug-ins are managed by the RapidContext storage subsystem. The server storage provides a unified view of all objects in the system, similar to a virtual file system. It has a number of important features:

• Object Identification — All objects can be located and fetched using a unique object identifier, similar to an absolute file name.
• Browsing & Lookup — The hierarchical structure of the storage makes it possible to browse and provide a RESTful view of its content. Any intermediate directory levels are automatically created when objects are stored (and removed if deleted).
• Meta-data & Introspection — A basic level of object meta-data is provided for all objects. This makes it possible to introspect and control many aspects of the running system.
• Persistence & Serialization — Standard object serialization automatically handles Java object creation and destruction, as well as persistence to disk. The Java .properties file format also makes low-level inspection and modification easy.
• Memory Cache — Live objects are cached in memory as long as the objects require. Background serialization to disk for modified objects is also provided.
• Virtual Objects — The storage system also manages virtual objects that are only in server memory with no persistent representation. This allows easy introspection into some aspects of the running system.
• Permissions & Access Control — A role-base access control system handles user permissions for search, read and write operations.

4.3 Storage Tree

The storage tree can be browsed and inspected directly (requires admin role) from the rapidcontext/storage/ path on the server. The directory tree is structured into directories based on the object types, as seen below:

All plug-ins are mounted to the storage tree, under the storage/ directory. Active (loaded) plug-ins are also overlaid onto the root storage tree. Each plug-in therefore shares some parts of the overall directory layout (outlined above). The plug-in storages may also “shadow” objects in the tree from other plug-ins, which is used when modifying objects.

The built-in storage mount points are structured as shown in the figure below:

• storage/ — Mount-points for all storage providers
  • cache/ — Storage caches for mounted overlay storages
    • plugin/ — Storage caches for plug-in storages
  • plugin/ — Mount-points for plug-in storage providers
    • cmdline/ — Mount-point for the cmdline plug-in
    • http/ — Mount-point for the http plug-in
    • jdbc/ — Mount-point for the jdbc plug-in
    • local/ — Mount-point for the local plug-in
    • system/ — Mount-point for the system plug-in

4.4 Storage Data Types

The objects retrieved from storage all have a storage data type. The data types can be divided into three categories that cover everything:

• Index — An object that provides a directory listing of the current storage path. The index objects are automatically generated from the storage content and cannot be modified directly.
• Object — A set of key-value pairs that can be inspected and serialized to a number of formats (JSON, XML, etc). An object may contain nested data structures consisting of arrays or objects.
• Binary — An opaque object that is only available as a binary data stream. This is used for serving static images, HTML files and similar. The object MIME type is based on the file extension, i.e. the characters after the first . in the storage path.

The first two categories of objects expose their storage data type in the type property, e.g. connection/jdbc for a JDBC connection object. The data types themselves are also objects that can be retrieved from the type/[type identifier] storage path. The example below shows parts of the JDBC connection data type (in YAML format):

id: connection/jdbc
type: type
description: >-
    The JDBC connection type. JDBC connections allows execution of SQL queries
    and statements to any JDBC data source. Connections may be pooled for
    maximum resource utilization.
initializer: org.rapidcontext.app.plugin.jdbc.JdbcConnection
property:
  - name: url
    description: |-
        The JDBC connection URL. (...)
    required: true
  ...
  

The object type initializer property links a data object to a Java object. Whenever such an object is retrieved from storage, the corresponding Java object will be automatically created and put into the storage cache. The Java objects in the storage cache may remain there indefinitely, but roughly every 30 seconds a cache cleanup job will destroy any objects reporting an inactive status.

4.5 Hidden & Computed Properties

Sensitive data such as passwords or access tokens can be stored in a hidden property which is prefixed by a . character. Such properties are filtered out whenever an object is returned via a procedure or similar. They remain available for Java implementations of connections, procedures, etc.

The example below shows how the password property is hidden for user objects by default:

id: johndoe
type: user
name: John Doe
description: A simple example user
enabled: true
realm: RapidContext
.password: secret
  

Some properties are calculated or transient and should not be written to storage. These computed properties are prefixed by a _ character. Any operation that writes to storage will filter these out.

4.6 Value Expansion, Vaults & Secrets

Secrets or external property values can be auto-expanded when loaded from storage. Value expansion uses a ${secret-key} syntax, and is supported for all serialization formats (properties, json, yaml, xml).

The value lookup for the expansion keys are handled via one or more of the configured vaults. By default, the env and prop vaults are available (for looking up values via system environment variables or Java system properties). When expanding variables, a specific vault can be specified instead of a global lookup; e.g. ${env!SECRET_TOKEN}.

Default values for missing keys can be specified after the key, e.g. ${prop!key:default}. If no default is specified, any missing reference will be expanded to an empty string.