4.5. Task Management

Todo

  • Includes scheduling of tasks as well as any relevant task priorities or concurrency constraints

  • Respecting data-marshaling demands

  • Relates to algorithms and data (product) sets

  • Is this where the grammar goes for the predicate expression (Section 3.5.4.1)?

The execution of computational tasks will be managed by oneTBB Flow Graph. With oneTBB, the user does not explicitly manage threads but specifies tasks that should be scheduled by oneTBB for execution using its own thread pool.

Depending on the need, oneTBB interface may be explicitly used in algorithm code to take advantage of intra-algorithm parallelism. Such use will integrate smoothly with Phlex’s scheduled tasks as both the framework and the algorithm will be using oneTBB’s shared thread pool [DUNE 46].

4.5.1. Creating the Data-Flow Graph

The data-flow graph is constructed using the pattern described in the subsections below.

4.5.1.1. Program Configuration

The Phlex program ingests a program configuration that includes a specification of all algorithms that are graph-node candidates.

4.5.1.2. Required Data Products

All configured observers and data sinks are constructed as graph nodes. The framework records the list of data products required by each of these nodes. A data sink can be configured to accept all data products created by the algorithms in the configuration, without specifying the individual data products created by each algorithm.

4.5.1.3. Created Data Products

The transforms, folds, and unfolds specified in the configuration are created as graph nodes. The framework records the list of the data products produced by each of these nodes.

4.5.1.4. Connecting Algorithms

The data product lists recorded in Section 4.5.1.2 and Section 4.5.1.3 are compared. Edges are created between the nodes in Section 4.5.1.3 that create data products and those that match what is required by those in Section 4.5.1.2. This step is iterative so that all graph-node candidates from Section 4.5.1.1.

4.5.2. Data-Marshaling

4.5.3. Translators

A translator is a node whose input is of one concrete data-product type and whose output is of a different concrete data-product type, both of which model the same data-product concept. A translator will be automatically inserted into the graph between two nodes whose data-product types are concept-compatible but not identical. Translator nodes belong to the computational graph. There are no translators that precede a provider. There are no translators that follow a preserver.

From the flow-graph point of view, translators are treated as first-class nodes, while being mostly irrelevant or invisible to the user. Translator nodes for a given data representation conversion should be singular by construction and should not be duplicated. Translators are not configured directly by the framework user; they must be discovered by the framework based on needs implied by the user configuration. The framework should be able to discover translators and select which ones belong in the graph without loading every discovered translator DLL. The translator metadata required for graph building should be generated at translator plugin build time. Users will be able to register their own translators; such translators will be available for discovery by the framework and can be used to create transator nodes as needed for graph building.

4.5.4. Drivers and Runtime Hierarchy Handling

Phlex will implement a driver interface that enables opening an input file.

The hierarchy of data-cell indices will be declared to the framework before processing any data-cell indices.

Different policies for supporting hierarchies will be supported:

  • a fixed hierarchy may be declared for the entire job

  • the global hierarchy may be updated based on hierarchies stored in input files in the middle of the job

The data-layer hierarchy will not be known until the input file is opened. Whenever the global hierarchy is established (e.g. by the first input file) or updated (e.g. by a subsequent input file), the framework validates it against what the computational graph requires.