Developing Flow 5.5 Python Components

Folder and File Structure

To make a a python component discoverable and available to add to the canvas it needs to be in the local 'components' folder within a specific Flow Project. This enables the whole project to be easily packaged up and shared by going to File > Export Project in the menu.

The location of the local components folder is dependent on the location that Flow is storing your Flow Projects in. This can be set in Settings > Projects and hitting browse to set the Project Folder.

TIP

The project folder is typically on a shared drive such as Google Drive or Dropbox, in order to aid collaboration.

Within the Flow Project Folder the local components folder can be found at <FlowProjectFolder>/<ProjectName>/components. There will be a sub-folder for each separate component and that sub-folder will contain the following files:

• flow-component.toml - A required metadata file that contains information such as a unique identifier for the component, human readable port names and the file name of the corresponding python file. It is always named flow-component.toml regardless of the component name.
• <component_name>.py - The python code that the component runs. By convention the component name matches the name of the sub-folder that it sits within, but it must match the name specified within the metadata file.
• README.md - An optional markdown file that describes the component. The description is visible in the component browser area of Flow's UI.

If your component and its supporting files are in the correct location and error free, the component will automatically be discovered by the Flow UI and the component will be available to add to the canvas through the component search box or the component browser.

Python File

The python file of a python component contains four main sections:

• Port definitions: The section of the file where we define how many inports and how many outports the component will expect data from, and the types of data that each will accept.
• Get data from ports: The part of the file where data from the ports is accessed and converted into regular python data types to be passed onwards within the component.
• Component content: The engineering work that the component will do. If you are bringing some pre-existing python models into Flow, this section may just be an import or a copy-paste of the code you have already written.
• Send data on outports: The section of the file where results are formatted correctly to be sent along to the outports of the component.

Simple Example

Below is a very simple example python component that adds the number from each inport together:

from flow import Component, Definition, Inport, Outport
from flow_types import base

# Define the ports
value1 = Inport(id="value1", types=[base.Double], multi_connection=False)
value2 = Inport(id="value2", types=[base.Double], multi_connection=False)
result = Outport(id="result", types=[base.Double])

# Create the component definition
definition = Definition(inports=[value1, value2], outports=[result])

# The process that the component performs
def process(component: Component):
    if not component.has_data():
        return

    # Source the data from the inports
    val1 = float(component.get_data("value1"))
    val2 = float(component.get_data("value2"))

    # Do whatever it is that your custom component does
    out = val1 + val2
    
    # Cast your result(s) to a flow message type
    out_msg = base.Double(out)

    # Send the result message to the outports
    component.send_data(out_msg, "result")

• Port definitions are from lines 5 - 10
• Data is retrieved from the ports on lines 18 - 19
• The component content is just the single line on line 22
• And the output is put into the correct format on line 25 and sent to the outport on line 28

Inport and Outport Options

You can have as many inports or outports as you like in your component and they can each have a variety of different settings and types:

1. They can accept multiple wires on a single inport with multi_connection=True or just a single wire at a time with multi_connection=False. Generally it is best practice to avoid multi_connection=True for inports except for in a few specific circumstances. Outports always have multi_connection=True which is the default and therefore does not need to be specified.

2. Ports can be required with required=True in their declaration, or optional with required=False. If they are not required you will need to check whether they have data, and build out logic for what to do if they do not, within the component process. This is described in further detail here. The default is required=True, and therefore this does not need to be specified.

3. Acceptable port data types are defined within a list: types=[..., ...]. A single type e.g. types=[base.Double] or multiple types e.g. types=[base.Double, base.Int, base.Bool] can be specified. The different options for type are:

   1. base.Double for normal floating point numbers
   2. base.Int for numbers that must be integers
   3. base.Bool for True or False boolean values
   4. base.String for text
   5. base.MdDouble for multidimensional arrays or simple lists of floating point numbers. These behave like a NumPy array or a normal python list respectively.
   6. base.MdInt for multidimensional arrays or lists of integers
   7. base.MdBool for multidimensional arrays or lists of boolean values
   8. base.MdString for multidimensional arrays or lists of text (string) values
   9. base.Table for multidimensional data where the data type of each column differs
   10. None in order to specify that the port can take any of the above types

Since in python a bool is a special instance of an integer (where True=1 and False=0) it is common to accept both base.Bool and base.Int when an integer is desired. Similarly, when a float is desired it is common to also accept base.Int and base.Bool too. This will allow the maximum flexibility for the users of your components.

To restrict a multidimensional port so that it only accepts input that matches a specific dimension, an optional dimension= argument can be added:

value1 = Inport(id="value1", types=[base.MdDouble(dimension=1)])

If further specificity is required the shape= argument can be added instead:

value1 = Inport(id="value1", types=[base.MdDouble(shape=[2,2])])

Options for Getting Data From Inports

Data that is passed along wires from component to component in Flow is sent in the form of a FlowType message. An example FlowType message (in this case a floating point double with a value of 20) is below:

{
   "Info":{
      "Id":"WvJdsvhjuedmsgndjpAnrj",
      "Type":{
         "Id":"base.Double"
      },
      "Timestamp":"2021-07-15T20:31:30.818Z"
   },
   "Value":20
}

In order to do any calculations on the data that comes into a custom component, the value or values contained within the FlowType must first be extracted as regular python data types.

If the data type of the inport FlowType is one of the single value types such as base.Double or base.Int, its data can be accessed from within the component using a statement such as:

value1 = float(component.get_data("value1"))

The casting to a float is to ensure we are confident that we are only carrying a single python data type forward into our calculations, even though the inport may have accepted a union of FlowTypes such as [base.Double, base.Int, base.Bool].

If the data type is multidimensional such as base.MdDouble it can be turned into a list with the .to_list() method. If it is more convenient when writing the component process, there is also a .to_ndarray() method which gives a NumPy array. In most cases, even with lists, it is important to carry only a single data type forwards into the calculations. This can be accomplished with a statement similar to the below:

value_list = component.get_data("value_list").to_ndarray().astype(float).tolist()

Component Content

Once you have extracted all of the data from the inports and saved them as variables with regular python types, you are free to use them as normal to do the work you want to do. Typically this work has been developed outside of Flow and is simply pasted or imported into the custom component.

Flow comes preinstalled with a variety of useful libraries for performing common engineering tasks, such as NumPy and SciPy. These can be imported at the very top of your custom component and used as normal.

The list of libraries that come preinstalled with Flow can be found here. If you have a specific need to work with a specialist library that is not in this list, then please contact us through your regular support channel and we will likely be able to help you.

TIP

Sometimes it can be hard to decide what should go into a single custom component and what should be split out across many. Typically we would advise dividing your work across multiple components; However if a block of work will take place within a loop, it makes most sense to place the whole of that loop within a single custom component.

Throwing Errors

If you need to check for certain conditions in your code it can sometimes be useful to throw an error for the component user to take note of. These appear behind the bell icon in the top right of the Flow UI. Typically these errors are raised in the normal python way:

if my_value != 0:
	raise ValueError("Arrgghhh, this should have been 0")

Sending Data Out

Before data can be sent out of a component along a wire it must be cast back into a FlowType. This is achieved by wrapping your result in your chosen FlowType, for example:

out_msg = base.Double(out)

It is only once the data is in this form that it can be sent to a specific outport:

component.send_data(out_msg, "result")

NOTE

Only native python data types can be cast into FlowTypes. Native NumPy arrays for example will not work. If in doubt cast your result to a native python data type before casting to a FlowType e.g. base.MdDouble(list(numpy_array)) or base.Int(int(number)).

Metadata File

The flow-component.toml file typically contains four main sections:

• Component: This contains the file name of the matching python file and a unique identifier number so that the component cannot be mistaken for another. To generate these UUIDs you should go here and click copy.
• Details: This section contains the name that the user will identify the component by in the component browser and search functions. It also contains a namespace to help with organisation, and a description to add more detail.
• Inports: Inports are kept separate from outports and the id on the left of the equals sign must match the port id defined in the python file. The name and optional description field appear in the component browser.
• Outports: These are defined in the same way as inports above.

Example

Below is a dummy example of a custom component metadata file:

[component]
file = "test.py"
id = "8e72f3a0-1ac9-4ecb-bfe6-5b93c5fde3b5"

[component.details]
name = "Test Component"
namespace = "Custom Component"
description = "This component is only a test"

[component.details.inports]
value1 = { name = "Value 1", description = "The first value" }
value2 = { name = "Value 2", description = "The second value" }

[component.details.outports]
result = { name = "Result", description = "The resulting value" }

Next Steps

This documentation covers a fairly simple dummy example of a custom Flow component however very complex blocks have been built up by some of our customers. If you think something should be possible it probably is! Please contact us for help and we can expand on any small details not covered fully in this documentation.