Fluksio

A data-science view

While doing my phd in quantum computing I've had to perform various numerical experiments, e.g. while working on the Fourier fingerprints or beyond gates paper. Naturally, experiments are not always as straight-forward as they are being reported in papers, thus one tries various approaches, adjusts parameters and so on.

I've mostly used Kedro in combination with MLflow (available through a plugin) where the former allows to build a node-based pipeline and the latter enables the nodes (python functions) to report metric values, log parameters and track changes across several runs. While in principle this suffices my needs, I wasn't quite happy with that. First, Kedro came with a huge boilerplate code which sometimes forced more complexity than what was needed for a particular project. Second, while MLflow as working well in principle, there were issues with race conditions on SLURM and getting the reported data out of the storage was quite inconvenient. Third, there is a more philosophical problem with the fact that Kedro encourages a clean input-processing-output pipeline but then having MLflow kinda like sitting in the processing part and bypassing this pipeline by reporting values from within the middle.

A home-automation view

At the same time I'm running the automation in my Tiny House using NodeRed and experienced quite some frustration over the years as the system grew and canvases became more and more a mess of various nodes sending around signals to other nodes, adjusting global or local parameters which were impossible to keep track of and sometimes causing weird race conditions. Not even starting here with mobile-friendly support, modern user interfaces or the lack of possibility to version-control or debug the system. NodeRed itself is of course a great piece of software, it's stable, fast and highly scalable so the issues mentioned might just be me-problems.

Naturally, there are alternatives to that; In the very beginning I was using OpenHab which, while easy to set up, turned out to not scale well beyond simple light switches. A more common solution is probably HomeAssistent which has a broad community support, integrates well with various IoT providers and is also easy to set up. However, I never felt comfortable with it, as support for Python, testing and debugging and more advanced control options were missing.

A solution to both

Interestingly, both worlds (data-science and home-automation) can be combined as they share the same core principle: getting data from A to C via B and eventually reusing B in other instances. In the first case, this could be some training data A, piped through a model B which produces some weights C. These weights then are later re-used in B for inference. For the home-automation view, this would translate to sensor A producing some data which is processed by B and based on the data, a light C is turned on or off.

So I built fluksio

It is a node-based automation software providing a web interface to create flows, nodes and dashboards with many common home automation connectors available. On the data-science side, it offers a terminal UI and an sdk to version runs assign runner flavors and integrates with SLURM for job submission. Local setups can easily paired with a remote hub such that data science experiments and home automation can be controlled and tracked remotely.

Fluksio offers custom Python nodes, but also predefined nodes to aggregate data, allows you to inspect the most recent data samples on each input/output of a node and allows you to put your flows to the test before deploying them. Dashboards can be utilized to visualize data or trigger events by providing a wide range of widgets.