3. Conceptual Design¶

Phlex adopts the data-flow approach discussed in Section 2.6.1. Instead of expressing scientific workflows as monolithic functions to be executed, workflows are factorized into composable algorithms that operate on data products passed among them [DUNE 1], [DUNE 111], [DUNE 20]. These algorithms then serve as operators to higher-order functions (HOFs) that operate on data-product families.

To guide the discussion of Phlex’s conceptual model, we refer to Fig. 3.1, which shows a small fictitious workflow that creates vertices from simulated energy deposits. Various framework aspects are demonstrated by that figure:

data-flow graph

The data-flow graph is formed by ingesting the configuration file and recording the data-product dependencies required of each algorithm (see Section 3.1).

data-product flow

Data products (see Section 3.2) are passed along graph edges. As mentioned in Section 2.6.1, the data passed between HOFs are expressed as families. Fig. 3.1 thus formally passes families (e.g. \([\textit{GoodHits}_{ijk}]\)) between nodes [1].

framework driver

The driver instructs the framework what to process (see Section 3.6).

The driver in Fig. 3.1 is configured so that all Spills in the specified ROOT input files are processed.

data-product providers

Data-product providers are framework components that provide data products from external entities to downstream user algorithms (see Section 3.7). From a functional programming perspective they are transforms that map a data cell to one of the data products within that data cell.

In the workflow, one provider reads a \(\textit{SimDepos}\) data product from each Spill in the HDF5 input files, and the other reads a single \(\textit{Geometry}\) corresponding to the Job from a GDML file.

HOFs and user-provided algorithms

Arguably the most important aspect of the framework is how user-provided algorithms are bound to HOFs and registered with the framework (see Section 3.3, Section 3.4 and Section 3.5).

All seven HOFs supported by Phlex (see Table 2.2) are used in Fig. 3.1. For the main processing chain of creating vertices:

• An unfold HOF is configured to create a family of \(\textit{Waveforms}\) objects—creating one \(\textit{Waveforms}\) object per APA—from one \(\textit{SimDeps}\) data product in each Spill.

• A configured transform HOF is run on the family of \(\textit{Waveforms}\) objects to create a family of \(\textit{GoodHits}\) objects.

• To make a \(\textit{GoodTracks}\) data product, a window algorithm is applied to pairs of \(\textit{GoodHits}\) objects that come from adjacent APAs.

• Lastly, another transform algorithm operates on the \(\textit{GoodTracks}\) data products to produce vertices.

There are additional parts of the graph that are not directly related to creating vertices:

• A fold algorithm is executed over the \(\textit{GoodHits}\) data products to sum the hit energy (i.e. \(\textit{TotalHitEnergy}\)) across all APAs for a given Spill.

• After a filter has been applied with the predicate \(\textit{high\_energy}\), an observe algorithm is used to fill a histogram with hit-related information from the \(\textit{GoodHits}\) data products.

data-product writers

Data-product writers are plugins that write data products to framework outputs (see Section 3.8)

Each of the five writers in Fig. 3.1 is responsible for writing to one or more output files.

resources

Most workflows require access to some external resource (see Section 3.9).

The histogramming resource in Fig. 3.1 enables the observe algorithm to fill and write histograms to a ROOT analysis file.

Note that in this workflow, the names Spill and APA are not special to the Phlex framework; they are names (hypothetically) chosen by the experiment. Each data product is also indexed, thus associating it with a particular data cell (e.g. \(\textit{GoodHits}_{3,5,9}\) denotes the \(\textit{GoodHits}\) data product belonging to APA \(9\) of Spill \(5\) of Run \(3\)).

Footnotes