3.3. Operators and Algorithms

As mentioned in Section 2.5, a user-defined algorithm is bound to the framework through an operator to a higher-order function (HOF). It is the operator registered with the framework that is executed as part of the data-flow graph. As will be illustrated in Section 3.4, a framework-agnostic algorithm can often serve directly as the HOF operator, without any framework-specific wrapper code.

In general, Phlex supports the registration of C++ operators with function signatures like (see Section 3.5 for a list of supported HOFs):

return_type function_name(P1, Pn..., Rm...) [quals];

where the types P1, Pn... denote types of data products and the types Rm... indicate resources. The bracketed [quals] term indicates that Phlex allows for class member functions that have trailing qualifiers (e.g. const). Each registered operator must accept at least one data product.

The signature of a Python operator needs to be available through reflection, either because the function is JITed (e.g. with Numba), bound (e.g. with ctypes), or annotated. The latter is good practice regardless and commonly required by Python coding conventions:

def function_name(p1: P1, pn: Pn..., rm: Rm...) -> return_type:

We will first discuss the data-product and resource types in Section 3.3.1, followed by the return types in Section 3.3.2, and then the function name and optional qualifers in Section 3.3.3.

3.3.1. Input Parameters

A data product of type P may be presented to a C++ operator if the corresponding input parameter (i.e. the relevant P1, ..., PN type) is one of the following:

  • P const& — read-only access to a data product provided through a reference

  • P const* — read-only access to a data product provided through a pointer

  • P — the data product is copied into an object (assumes data product is copyable) [1]

  • phlex::handle<P> — a lightweight object that provides read-only access to a data product as well as any metadata associated with it

For each of these cases, the data product itself remains immutable. A Python operator can receive a phlex::handle or a direct reference to the data product. There is no equivalent language support for read-only access, but it will be enforced where possible.

Data-product ownership

As mentioned in Section 3.2.3, the framework owns all in-memory data products and releases them from memory once they are no longer required by downstream work. An operator (and any code used by that operator) must, therefore, not retain a handle object, a pointer, or a reference to an input data product after the operator returns. An operator that requires a data product in a later invocation must receive it through the framework as an input to that invocation.

Whereas data products may be copied, resources of type R may not. The following types are therefore supported:

  • R const& — read-only access to a resource provided through a reference

  • R const* — read-only access to a resource provided through a pointer

  • R& — read-and-write access to a resource provided through a reference (if supported by resource)

  • R* — read-and-write access to a resource provided through a pointer (if supported by resource)

Resources are described in more detail in Section 3.10.

3.3.2. Return Types

The meaning of an operator’s return type depends on the HOF and is discussed in Section 3.5. However, to simplify the discussion we introduce the concept of the created data-product type. As mentioned in Section 3.2.3, the framework manages all data products returned by operators. This means that the data products created by operators must have types that connote unique ownership. An operator’s returned object must therefore model a created data-product type, which can be:

  • a value of type T, or

  • a std::unique_ptr<T>, where the created object is non-null.

For Python, this means that an operator should not retain any external hard references to a returned object.

The following types (or their equivalents) are forbidden as created data-product types because they do not imply unambiguous ownership:

  • bare pointer types, such as T* or T const*

  • reference types, such as T& or T const&

3.3.3. Function Names and Qualifiers

The function_name in Section 3.3 above may be any function name supported by the C++ language. Code authors should aim to implement operators (or, equivalently, algorithms) as free functions. However, in some cases it may be necessary for class member functions to be used instead. When member functions are required, the qualifier const should be specified to indicate that the class instance remains immutable during the execution of the member function [2].

Footnotes

References