4.1. Framework Subsystems

Fig. 4.1 depicts the relationships between the framework subsystems. A subsystem is a cohesive set of components that together provide a well-defined set of functionalities to the rest of the framework. Each subsystem has a set of responsibilities and a well-defined interface to be used by the other subsystems and plugins that need to use the provided functionality. In the figure, the elliptical nodes represent subsystems, and the arrows indicate what other subsystems each subsystem depends upon. The subsystems have no circular dependencies. The diagram shows a box that identifies the core framework subsystems, another that identifies the external subsystems, and a third one that shows examples of plugins.

digraph finite_state_machine { fontname="Helvetica,Arial,sans-serif" compound=true node [fontname="Helvetica,Arial,sans-serif", style=filled, fillcolor=darksalmon]; logging [label="Logging"]; configuration [label="Configuration"]; subgraph cluster_core_subsystems { label = "Core framework subsystems"; algo_description [label="Algorithm\ndescription"]; application [label="Framework\nApplication"]; data_modeling [label="Data\nmodeling"]; error_handler [label="Error\nhandler"]; io [label="IO"]; data_product_management [label="Data-product\nmanagement"]; metadata_management [label="Metadata\nmanagement"]; monitoring [label="Monitoring\nand\nreporting"]; plugin_management [label="Plugin\nmanagement"]; registration [label="Registration"]; random_number [ label="Random\nnumbers"]; resource_management [label="Resource\nmanagement"]; signal_handler [label="Signal\nhandler"]; task_management [label="Task\nmanagement"]; { rank=same; algo_description; task_management; } algo_description -> registration; application -> algo_description; application -> error_handler; application -> monitoring; application -> plugin_management; application -> signal_handler; application -> task_management; data_modeling -> monitoring; data_modeling -> registration; io -> data_modeling; metadata_management -> data_modeling; metadata_management -> registration; plugin_management -> monitoring; plugin_management -> registration; resource_management -> monitoring; task_management -> error_handler; task_management -> io; task_management -> data_product_management; task_management -> metadata_management; task_management -> monitoring; task_management -> random_number; task_management -> resource_management; task_management -> signal_handler; data_product_management -> metadata_management; data_product_management -> data_modeling; } subgraph cluster_external { label="External subsystems"; edge [style=invis] node [fillcolor=lightyellow]; style="filled"; fillcolor="lightgray"; Build; CI; Documentation; event_display [label="Event\ndisplay"]; workflow [label="Workflow\nmanagement"]; } subgraph cluster_plugins { label="Plugins"; edge [style=invis] node [shape=box,fillcolor=lightblue]; ranksep="0.05"; style="filled"; fillcolor="lightgray"; Algo [label="User algorithms"] ROOT [label="IO backends"] RUCIO [label="Data management"] ellipsis [label="etc.", fillcolor=none, peripheries=0]; Algo -> ROOT -> RUCIO -> ellipsis; } algo_description -> configuration [ltail=cluster_core_subsystems]; error_handler -> logging [ltail=cluster_core_subsystems]; RUCIO -> logging [ltail=cluster_plugins]; ROOT -> configuration [ltail=cluster_plugins]; workflow -> application [ltail=cluster_external, lhead=cluster_core_subsystems, fontname="Helvetica,Arial,sans-serif", label=" Makes use of", style=dotted, minlen=3] }

Fig. 4.1 Phlex subsystems

The figure organizes all Phlex subsystems into three groups, with two standalone cross-cutting nodes.

The 14 core subsystems and their dependency arrows show the following structure:

  • Framework Application provides the executables run by the user. It is responsible for creating and executing the workflow graph.

  • Task Management provides the runtime scheduler and executor. It depends on oneTBB Flow Graph. It provides the nodes that are used to represent CHOFs in the workflow. This includes computational, provider, translator and preserver nodes. It also provides the types used to define drivers, as well as framework-supplied drivers It also provides the types that are used to represent the workflow graph as a whole.

  • Data Modeling provides the mechanism by which users define data-product types, and the tools used to translate between in-memory representations of data-product concepts.

  • Data-Product Management provides the mechanisms to organize and identify data products. This includes the types used to represent data cells, data-product families, and data layers. It also provides the mechanisms by which the lifetime of data products are managed. It provides the mechanisms used for data-product lookup (based on data-product metadata).

  • Metadata Management provides the mechanisms by which framework-defined metadata are associated with data products, data cells, data-product families, data layers, and jobs. It also provides the facilities by which users can define their own metadata types and associate them with the appropriate entities, listed above. It also provides the mechanisms by which the metadata associated with one of these items can be retrieved.

  • IO provides the mechanisms by which data products and relevant metadata are read and written to persistent storage. It defines an interface implemented by IO plugins. Providers and perservers use this interface to read and write data products and metadata.

  • Plugin Management provides tools to dynamically load libraries that implement plugins. Supported plugin types include:

    1. modules that create computational nodes,

    2. sources that create provider nodes,

    3. libraries that provide translator nodes,

    4. libraries that provide preserver nodes,

    5. libraries that provide drivers, and

    6. libraries that provide IO backends.

  • Resource Management is responsible for initializing and configuring TBB. It also provides mechanisms to initialize and manage GPU runtime systems.

  • Monitoring and Reporting provides the mechanisms by which the framework’s performance and behavior can be monitored and reported.

  • Registration provides the mechanisms by which users can register their own algorithms, data-product types, providers, drivers, and plugins.

The following subsystems will be specified when the requirements are more complete.

  • Algorithm Description

  • Error Handler

  • Signal Handler

  • Random Numbers

Two cross-cutting subsystems sit outside all clusters because every other subsystem may depend on them:

  • Configuration — provides configurability for both the core subsystems and plugins.

  • Logging — initializes the logging mechanism that can be used by both the core subsystems and plugins.

Three representative plugin categories are shown — User Algorithms, IO Backends, and Data Management (e.g., RUCIO) — with an etc. node indicating extensibility. As a group, plugins depend on Configuration (to configure each plugin) and Logging (to emit messages), but are otherwise decoupled from core internals.

IO Backends are plugins that implement the IO interface provided by the IO subsystem. One such implementation is FORM, which provides the ability to read and write from ROOT and HDF5 files. Other IO technologies may be supported by implementing another IO Backend plugin.

Five externally-managed systems — Build, CI, Documentation, Event Display, and Workflow Management — are grouped separately. Of these, only Workflow Management has a cross-edge: a dotted “Makes use of” arrow from the workflow manager to the core subsystems cluster. This indicates a workflow system that drives framework application execution but is not architecturally part of the framework itself.