Flow Docs

# Developing Flow 6.0 Python Components

# Folder and File Structure

To make a a python component discoverable and available to add to the canvas it needs to be in the local 'components' folder within a specific Flow Project. This enables the whole project to be easily packaged up and shared by going to File > Export Project in the menu.

The location of the local components folder is dependent on the location that Flow is storing your Flow Projects in. This can be set in Settings > Projects and hitting browse to set the Project Folder.

TIP

The project folder is typically on a shared drive such as Google Drive or Dropbox, in order to aid collaboration.

Within the Flow Project Folder the local components folder can be found at <FlowProjectFolder>/<ProjectName>/components. There will be a sub-folder for each separate component and that sub-folder will contain the following files:

  • flow-component.toml - A required metadata file that contains information such as a unique identifier for the component, human readable port names and the file name of the corresponding python file. It is always named flow-component.toml regardless of the component name.
  • <component_name>.py - The python code that the component runs. By convention the component name matches the name of the sub-folder that it sits within, but it must match the name specified within the metadata file.
  • README.md - An optional markdown file that describes the component. The description is visible in the component browser area of Flow's UI.

If your component and its supporting files are in the correct location and error free, the component will automatically be discovered by the Flow UI and the component will be available to add to the canvas through the component search box or the component browser.

# Settings and Dynamic Components

In Flow 6.0 and above, components (both built into the product and written by you) can now have settings that are accessed by double clicking on the block on the canvas. These settings can be used for a variety of purposes, but some of the most common are:

  • Add in additional ports or remove ports from the component
  • Modify the data type that the ports accept
  • Modify the functionality of the component. It is best to limit the options to a few related behaviours that the user would expect.

These setting changes are processed in a portion of the custom component code called the setup which is also new in Flow 6.0. This code is called once when a component is added to the canvas, and again whenever any settings are modified, but it is not called simply because the data on any of the inports has changed.

Components that can have the quantity of ports changed or the port data type modified are often called dynamic components or components with dynamic ports.

# Examples of When and Why to Use Settings

# Number of Ports

A common use for component settings is to add additional ports to a component based on a users choices within the settings. Consider a hypothetical add block for adding the numbers together that come in on the inports. Without component settings, we would have to hard code the number of inports that the add block accepts. If we hard coded two inports but when using the component we discovered that we had three numbers to add, we would either have to write a new component or use two add blocks back to back. Taken to the extreme either of these approaches could quickly become unwieldy.

However if we defined a integer setting called 'number of ports' we could handle this scenario much more elegantly. Within the component setup we could define a loop from zero to the 'number of ports' and on each iteration add a new port to the add block. During the component process we could loop again through each port and add the data from that port to a running total. This dynamic add block would have two key advantages over its non-dynamic counterpart:

  • Fewer blocks needed on the canvas to achieve the same outcome
  • Fewer components needed in the library because each component is more flexible

# Multiple Data Types

Another scenario where settings and dynamic components make sense is when the functionality of the block can work equally well with a variety of different data types. Typically outports on Flow components only send out a single data type e.g. a floating point number. But if we wanted the flexibility to write a single component that performs its function on either integers or floating point numbers we could give the user a setting to choose the data type for the component. If they select 'integer' then both the inports and outports could be defined to only accept integers, and if they select 'float' then both inports and outports could be defined to only accept floating point numbers.

# Changing Functionality

Sometimes it can make sense to group several related but subtly different behaviours for a component into a single block and allow the user to choose which one they would like to use with the component settings. A good example of this could be a 'round' block. The block could be given a setting that offers the user a dropdown of three options to choose between:

  • Round up
  • Round down
  • Round nearest

If they select 'round up' then the component would use a different branch of logic to generate an answer than if they selected 'round down'.

# Python File

The python file of a python component contains six main sections:

  • Port definitions: The section of the file where we define how many non-dynamic inports and how many non-dynamic outports the component will expect data from, and the types of data that each will accept.
  • Settings: The area where any controls to be added to the component settings are defined.
  • Setup: The code that processes the settings and generates any of the required dynamic ports.
  • Get data from ports: The part of the file where data from the ports is accessed and converted into regular python data types to be passed onwards within the component.
  • Component content: The engineering work that the component will do. If you are bringing some pre-existing python models into Flow, this section may just be an import or a copy-paste of the code you have already written.
  • Send data on outports: The section of the file where results are formatted correctly to be sent along to the outports of the component.

# Example

Below is a example python component that adds the number from each inport together. This is a dynamic component, where the first two inports are always present, and any additional inports that are required can be added in the component settings:

from flow import Ports, Process, Settings, Setup
from flow_types import base

# Define Ports
ports = Ports()

# Add non-dynamic inports
ports.add_inport(id="value1", types=[base.Double, base.Int, base.Bool])
ports.add_inport(id="value2", types=[base.Double, base.Int, base.Bool])

# Add non-dynamic outports
ports.add_outport(id="result", types=[base.Double])

# Define component settings
settings = Settings()
settings.add_int_setting(id="terms", default=2, minimum=2)


def setup(component: Setup):

    # Get the value set in component settings
    terms = component.get_setting("terms")

    # Use the setting to add additional inports to the component if required
    inport_ids = []
    for in_id in range(terms - 2):
        inport_id = f"value{in_id + 3}"
        inport_ids.append(inport_id)
        component.add_inport(name=f"Value {in_id + 3}", id=inport_id, types=[base.Double, base.Int, base.Bool])

    # Set a variable to be retrieved in the component process
    component.set_variable("inport_ids", inport_ids)


def process(component: Process):
    if not component.has_data():
        return

    # Retrieve the variable
    inport_ids = component.get_variable("inport_ids")

    # Source the data from the non-dynamic inports
    value1 = float(component.get_data("value1"))
    value2 = float(component.get_data("value2"))

    # Add the data from the two non-dynamic ports and get the data from the dynamic ports to add in
    result = value1 + value2 + sum(float(component.get_data(inport_id)) for inport_id in inport_ids)

    # Cast your result(s) to a flow message type send to the outports
    component.send_data(base.Double(result), "result")
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
  • Port definitions are from lines 5 - 12
  • Settings are defined on lines 15-16
  • The setup code is processed and the dynamic ports are added on lines 22-29
  • Data is retrieved from the non-dynamic ports on lines 43 - 44
  • The data from the non-dynamic ports is summed with the data accessed port by port from the dynamic ports on line 47
  • And the output is put into the correct format and sent to the outport on line 50

# Inport and Outport Options

You can have as many inports or outports as you like in your component. Each port has both an id and a type, and optionally also a default:

  1. The port id is the label that is used to refer to a specific port in order to gather data from it or send data to it. Typically it is written in lower case using underscore as a delimiter. A more human readable name can be specified for use in the UI as part of the metadata definition later.

  2. Acceptable port data types are defined within a list: types=[..., ...]. A single type e.g. types=[base.Double] or multiple types e.g. types=[base.Double, base.Int, base.Bool] can be specified. The different options for type are:

    1. base.Double for normal floating point numbers
    2. base.Int for numbers that must be integers
    3. base.Bool for True or False boolean values
    4. base.String for text
    5. base.MdDouble for multidimensional arrays or simple lists of floating point numbers. These behave like a NumPy array or a normal python list respectively.
    6. base.MdInt for multidimensional arrays or lists of integers
    7. base.MdBool for multidimensional arrays or lists of boolean values
    8. base.MdString for multidimensional arrays or lists of text (string) values
    9. base.Table for multidimensional data where the data type of each column differs
    10. base.Null for ports that should be optional, but where the default value is irrelevant
    11. ANY_FLOW_TYPE this FlowType is seldom used. You will mostly see it when data needs to pass through a component completely unchanged. If you think you are likely to need to use this, please reach out to us and we will let you know where it can be imported from.

  3. Some ports are required and must always have data for the component to be able to run. Others are optional and can be left disconnected. In order to enable this, any optional ports must be provided with a default value to use if no data is provided along a wired connection. This is done by setting default=base.<FlowType>(<value>) where the FlowType is any of the types in point 2 above, and the value is a value compatible with that FlowType.

TIPS

  • Since in python a bool is a special instance of an integer (where True=1 and False=0) it is common to accept both base.Bool and base.Int when an integer is desired. Similarly, when a float is desired it is common to also accept base.Int and base.Bool too. This will allow the maximum flexibility for the users of your components. This would also apply for the multi-dimension variants of these types.

  • To restrict a multidimensional port so that it only accepts input that matches a specific dimension, an optional dimension= argument can be added:

value1 = Inport(id="value1", types=[base.MdDouble(dimension=1)])
1
  • If further specificity is required the shape= argument can be added instead:
value1 = Inport(id="value1", types=[base.MdDouble(shape=[2,2])])
1
  • It is typically considered best practice to write components so that each outport only has a single FlowType, for example type=[base.Double]

# Settings

There are six kinds of setting that can be added to a component. It is possible to have multiple of these in any order, and they always return a regular python data type when you retrieve the value from the setting later:

  1. text_setting provides a text box for the user to type some text into and returns a str.
  2. number_int_setting gives the user a box to type an integer into as well as some plus and minus buttons. Returns an int.
  3. number_double_setting is similar to the integer setting above but accepts and returns a float.
  4. toggle_setting provides an on-off switch where the on position returns a bool of True and off returns False.
  5. select_setting can be configured to present the user with a dropdown of some preconfigured available options. Each option has both a 'label' which is presented to the user and a corresponding 'value' which is returned as an str. The 'label' field is optional, and if none is provided the user will be presented with the 'value' string in the dropdown.
  6. checkbox_setting works in a similar way to the select setting above with a series of pre-configured labels and values, however multiple options can be selected by way of checkboxes. The setting therefore returns a list of strings of the values that were selected as a List[str].

Once a settings instance is invoked with settings = Settings(), any of the above fields can be added by calling settings.add_<setting_name>(). Every setting field has the option of specifying a default= value. number_int and number_double also have min= and max= constraints that can be applied.

If no default is provided the following values will be used:

  • text_setting uses ""
  • number_int_setting or number_double_setting use 0 if allowed by the min and max limits
  • toggle_setting uses False
  • checkbox_setting gives an empty list [] - the equivalent of no checkboxes ticked

NOTE

The in the following situations a default value must be provided:

  • On a number_int_setting or number_double_setting, if 0 is not a valid value do to the min and max limits, a default must be specified.
  • On a select_setting a default must always be set.

# The Setup Process

Three key jobs are done during the setup process:

  • Any values passed by the user in the settings are read and used to modify the component
  • Any dynamic ports are added or modified
  • Variables are set which pass any important information over from the 'setup' function of the component to the 'process' function of the component

# Reading Data From Settings

Reading data from a setting simply involves calling component.get_setting("id_of_setting") and assigning it to a variable. The data type that comes back from the setting depends on the type of the setting and is described in detail here. This data can then be used for any purpose within the component setup process.

# Adding Dynamic Ports

Dynamic ports are added by calling component.add_inport(name="Example", id=example, types=[base.Double]) or the equivalent for outports. There are two key differences here compared to defining a static port:

  • We use component.add_inport rather than port.add_inport because we are adding them to the component setup rather than the port definition
  • The name= argument which sets the human readable name for the port in the UI must be specified. Because of the dynamic nature of the port, this cannot be read from the component metadata file like it would for a static port.

# Set Variables

The 'setup' of a component is completely separate from the 'process' of a component but sometimes it can be useful to pass data between them. For example if some dynamic ports have been added in the setup, a list of their ids would be needed in the component process in order to get data from each of them. This is achieved by setting a variable. The syntax is simply component.set_variable("variable_name", data_to_set).

# Get Variables

If your component has dynamic ports or separate streams of logic for different functionality you will likely have set a variable during component setup that you will need to access in the component process. The syntax is simply component.get_variable("variable_name"). This can then be set to a regular python variable to be used within your component process.

# Getting Data From Inports

Data that is passed along wires from component to component in Flow is sent in the form of a FlowType message. An example FlowType message (in this case a floating point double with a value of 20) is below:

{
   "Info":{
      "Id":"WvJdsvhjuedmsgndjpAnrj",
      "Type":{
         "Id":"base.Double"
      },
      "Timestamp":"2021-07-15T20:31:30.818Z"
   },
   "Value":20
}
1
2
3
4
5
6
7
8
9
10

In order to do any calculations on the data that comes into a custom component, the value or values contained within the FlowType must first be extracted as regular python data types.

If the data type of the inport FlowType is one of the single value types such as base.Double or base.Int, its data can be accessed from within the component using a statement such as:

value1 = float(component.get_data("value1"))
1

The casting to a float is to ensure we are confident that we are only carrying a single python data type forward into our calculations, even though the inport may have accepted a union of FlowTypes such as [base.Double, base.Int, base.Bool].

If the data type is multidimensional such as base.MdDouble it can be turned into a list with the .to_list() method. If it is more convenient when writing the component process, there is also a .to_ndarray() method which gives a NumPy array. In most cases, even with lists, it is important to carry only a single data type forwards into the calculations. This can be accomplished with a statement similar to the below:

value_list = component.get_data("value_list").to_ndarray().astype(float).tolist()
1

# Component Content

Once you have extracted all of the data from the inports, you are free to use them as normal to do the work you want to do. Sometimes, depending on the component settings, there may be different branches of logic that the component should follow. Typically the engineering work that the component should do has been developed outside of Flow and is simply pasted or imported into the custom component.

Flow comes preinstalled with a variety of useful libraries for performing common engineering tasks, such as NumPy and SciPy. These can be imported at the very top of your custom component and used as normal.

The list of libraries that come preinstalled with Flow can be found here. If you have a specific need to work with a specialist library that is not in this list, then please contact us through your regular support channel and we will likely be able to help you.

TIP

Sometimes it can be hard to decide what should go into a single custom component and what should be split out across many. Typically we would advise dividing your work across multiple components; However if a block of work will take place within a loop, it makes most sense to place the whole of that loop within a single custom component.

# Throwing Errors

If you need to check for certain conditions in your code it can sometimes be useful to throw an error for the component user to take note of. These appear behind the bell icon in the top right of the Flow UI. Typically these errors are raised in the normal python way:

if my_value != 0:
	raise ValueError("Arrgghhh, this should have been 0")
1
2

# Sending Data Out

Before data can be sent out of a component along a wire it must be cast back into a FlowType. This is achieved by wrapping your result in your chosen FlowType, for example:

out_msg = base.Double(out)
1

It is only once the data is in this form that it can be sent to a specific outport:

component.send_data(out_msg, "result")
1

NOTE

Only native python data types can be cast into FlowTypes. Native NumPy arrays for example will not work. If in doubt cast your result to a native python data type before casting to a FlowType e.g. base.Int(int(number)).

# Metadata File

The flow-component.toml file typically contains five main sections:

  • Component: This section contains the name that the user will identify the component by in the component browser and search functions. It also contains a namespace to help with organisation, a description to add more detail, and a unique identifier number so that the component cannot be mistaken for another. To generate these UUIDs you should go here and click copy.
  • Files: This contains the file name of the matching python file as well as an optional reference to a readme file. Remove the readme line completely if no readme file will be created for the component.
  • Settings: The user readable text that will describe a certain setting in the UI is called a label. Each setting id that is used in the python file must be defined here set equal to a corresponding label.
  • Inports: Inports are kept separate from outports and the id on the left of the equals sign must match the port id defined in the python file. The name and optional description field appear in the component browser.
  • Outports: These are defined in the same way as inports above.

# Example

Below is a dummy example of a custom component metadata file:

[component]
id = "8e72f3a0-1ac9-4ecb-bfe6-5b93c5fde3b5"
name = "Test Component"
namespace = "Custom Component"
description = "This component is only a test"

[component.files]
main = "test.py"
readme = "README.md"

[component.settings]
setting1 = { label = "Setting 1" }

[component.inports]
value1 = { name = "Value 1", description = "The first value" }
value2 = { name = "Value 2", description = "The second value" }

[component.outports]
result = { name = "Result", description = "The resulting value" }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# Component Testing

The quickest and easiest way to test a python component is at the bottom of the python file itself rather than on the canvas. However in order to do this, you will need to have setup your development environment to use the version of python packaged up with Flow. Working in this way will give you access to all of the dependencies required to run component tests.

# Setting Up Your Development Environment

Coming soon. Reach out to us for help

# Running Component Tests

Component tests can be run by importing the following module at the top of your component file:

from flow.testing import ComponentTest
1

You can then define the parameters of your test at the very bottom of your component file. Here is some example syntax:

if __name__ == "__main__":
    setting_data = {
        "terms": 2,
    }

    inports_data = {
        "value1": base.Double(1),
        "value2": base.Double(2),
    }

    outport_value = ComponentTest(__file__).run(inports_data, setting_data)
    print(outport_value["result"])
    assert outport_value["result"] == base.Double(3)
1
2
3
4
5
6
7
8
9
10
11
12
13

It is important to note:

  • Tests must be within the if __name__ == "__main__": statement.
  • Settings are passed to the component test as a dictionary containing the setting id and its corresponding value. Settings always take native python types.
  • Inport data is also a dictionary of inport id and value, however inports always take a FlowType such as base.Double().
  • You can either print the value received on the outport by referencing its port name as a dictionary key (e.g. outport_value["result"]), or you can assert that it should match a certain FlowType and value (e.g. assert outport_value["result"] == base.Double(3)).

# Next Steps

This documentation covers some fairly simple examples of custom Flow components however very complex blocks have been built up by some of our customers. If you think something should be possible it probably is! Please contact us for help and we can expand on any small details not covered fully in this documentation.

← Breaking Changes →