1. Introduction¶
The fundamental abstractions in ThingFlow are 1) sensors, which provide a means to sample a changing value representing some quanity in the physical world, 2) event streams, which are push-based sequences of sensor data readings, and 3) things, which are reusable components to generate, transform, or consume the events on these streams. Things can have simple, stateless logic (e.g. filter events based on a predicate) or implement more complex, stateful algorithms, such as Kalman filters or machine learning. Using ThingFlow, you describe the flow of data through these things rather than programming low-level behaviors.
Although ThingFlow presents a simple dataflow model to the user, internally it
uses an event-driven programming model, building on
Python’s asyncio
module. In addition to being a natural programming model for
realtime sensor data, it reduces the potential resource consumption of Ant
Events programs. The details of event scheduling are handled by the framework.
Separate threads may be used on the “edges” of a dataflow, where elements
frequently interact with external components that have blocking APIs.
ThingFlow integrates with standard Python data analytics frameworks, including NumPy, Pandas, and scikit-learn. This allows dataflows involving complex elements to be developed and refined offline and then deployed in an IoT environment using the same code base.
We call the implementation described here “ThingFlow-Python”, as it should be
possible to port the ideas of ThingFlow to other languages. Currently, one such
port exists: “ThingFlow-MicroPython”. This is a port of ThingFlow to MicroPython,
a limited version of Python 3 that runs “bare metal” on embadded devices. The
ThingFlow-MicroPython port is included in the ThingFlow-Python repository
under the subdirectory micropython
. It is documented in
Section 7 of this document.
Example¶
To give the flavor of ThingFlow, below is a short code snippet for the Raspberry Pi that reads a light sensor and then turns on an LED if the running average of the last five readings is greater than some threshold:
lux = SensorAsOutputThing(LuxSensor())
lux.map(lambda e: e.val).running_avg(5).map(lambda v: v > THRESHOLD)\
.GpioPinOut()
scheduler.schedule_periodic(lux, 60.0)
scheduler.run_forever()
The first line instantiates a light sensor object and wraps it in an output thing to handle sampling and progagation of events.
The next two lines
create a pipeline of things to process the data from the sensor. We call things
which have a single input and output filters, as they can be composed to process
a stream of events.
The map
filter extracts the data value from the sensor event, the
running_avg
filter averages the last five values, and the next map
filter converts
the value to a a boolean based on the threshold. The GpioPinOut
thing is
an adapter to the outside world. It turns on the LED based on the value of
its input boolean value.
Finally, the last two lines of the example schedule the sensor to be sampled at a sixty second interval and then start the scheduler’s main loop.
Platforms¶
ThingFlow does not have any required external dependendencies, so, in theory at least, it can be run just about anywhere you can run Python 3. It has been tested on the Raspberry Pi (Rasbian distribution), Desktop Linux, and MacOSX. In a desktop environment, you might find the Anaconda Python distribution helpful, as it comes with many data analytics tools (e.g. Jupyter, NumPy, Pandas, and scikit-learn) pre-installed.
ThingFlow has been ported to Micropython, so that it can run on very small
devices, like the ESP8266. Since these devices have stringent memory
requirements, the code base has been stripped down to a core for the
Micropython port. The port is in this repository under the micropython
directory.
Installing ThingFlow¶
We recommend installing into a virtualenv
rather than directly into the
system’s Python. To do so, first run the activate
script of your chosen
virtual environment. Next, you can either install from the Python Package
Index website (pypi.python.org) or from the ThingFlow source
tree.
Installing from the Python Package Index¶
The package name of thingflow-python on PyPi is thingflow
. You
can use the pip
utility to install, as follows:
pip3 install thingflow
If you have activated your virtual environment, this should pick up the
version of pip3
associated with your environment, and install ThingFlow
into your environment rather than into the system’s Python install.
Installing from the source tree¶
Go to the thingflow-python
directory and then run:
python3 setup.py install
If you have activated your virtual environment, this should pick up the
version of python3
associated with your environment, and install ThingFlow
into your environment rather than into the system’s Python install.
Using ThingFlow without installation¶
You can also run the ThingFlow code in-place from the git repository by adding
the full path to the thingflow-python
directory to your PYTHONPATH
. This
is how the tests and the examples are run.
Directory Layout¶
The layout of the files in the ThingFlow code repository (the thingflow-python
directory) is as follows:
README.RST
- a short introduction and pointer to resourcesMakefile
- builds the source distribution and documentation; can run the testssetup.py
- used to install the core code into a python environmentthingflow/
- the core code. This is all that will get installed in a production systemthingflow/base.py
- the core definitions and base classes of thingflowthingflow/adapters
- reader and writer things that talk to the outside worldthingflow/filters
- elements for filter pipelines, in the style of Microsoft’s Linq framework
docs/
- this documentation, build using Sphinxtests/
- the tests. These can be run in-place.examples/
- examples and other documentation.micropython/
- port of the ThingFlow core to MicroPython
More Examples¶
Additional can be found throughout this document, particularly in Section 6: More Examples. There is also a separate repository with larger ThingFlow applications. It is at https://github.com/mpi-sws-rse/thingflow-examples. This repository includes an automated lighting application and a vehicle traffic analysis.
Next, we will go into more detail on ThingFlow with a tutorial.