Getting started with node-red for home/lab/process automation

First of let’s be clear: Node-red wouldn’t be the solution I’d choose for some serious automation (independent if it’s used for industrial automation, laboratory / experimental automation or home automation) - at least not when used on the long term. It’s really nice to try out some new ideas or do some pretty basic stuff - and of course that’s only a personal opinion because I have some strong ideas on how systems that one depends on should work (i.e. building from minimal to no external dependencies, using a well defined build process, involving down to zero interactivity after initial development, using languages that are stable and do not change too fast, reducing dependencies also implies not using interpreted languages like JavaScript, etc.)

What is node-red

Node-red is basically a flow graph editor developed by IBM and released as open source project later on. It allows to graphically chain different building blocks that pass messages between them to perform automation tasks (might be used for data analysis, industrial/home/laboratoy automation, etc.). The concept might be recognized by someone who has used tools like LabView in the past. Node-red is built on top of node.js so one is capable of using all capabilities node.js provides. Node.js is an event framework built on top of the V8 JavaScript engine (that’s the same engine used in the Chromium and Chrome browser as well as many other JavaScript applications like the Electron framework). It provides everything necessary to interface to native components as well as to run JavaScript handlers on top of a highly concurrent network server, handling UDP messages, etc.

Installing node-red

First off - I’m using a FreeBSD machine as basis (no mather if it’s on the RaspberryPi (note: Amazon affilate link, this pages author profits from qualified purchases) or - like my own deployments - on a virtual machine on one of the VM hosts I’ve got available for such stuff). Of course since node-red is based on node.js and is running entirely on an JavaScript engine although it’s possible to extend node with native modules it’s not exactly the most resource efficient piece of software. So a RaspberryPi might be sufficient for most small scale solutions but as soon as for example video is being processed it might get tight on available RAM. Also the relieability of SD cards in RaspberryPi requires some serious considerations (disabling swap space, disabling syslog or logging - which defeats the purpose of the log - into a RAM-Disk, using a centralized syslog server, preventing nearly all periodic writes, etc.) so this setup focuses on a simple deployment of node-red on a VM with 1 GB RAM, 20 GB virtual harddisk (everything totally over the top for a small deployment).

The installation is pretty easy - after initial system setup and hardening one just has to install node.js and the node package manager (npm). The easiest way is to simply install the www/npm package via pkg

pkg install www/npm

After that one should manually install python2 because some of the packages of node-red dependency chain depend on python during their installation process:

pkg install python2

At the end node.js can be installed using npm. This will fetch all required dependencies.

npm install -g --unsafe-perm node-red

One can now perform the necessary configuration (setting passwords). To be capable of setting the passwords bcryptjs is required:

npm install bcryptjs

After that password hashes that will be set inside ~/.node-red/settings.js can be created using the CLI command

node -e "console.log(require('bcryptjs').hashSync(process.argv[1], 8));"	password

Note that this will write your password inside your shell history! At least the authentication information for admin and user should be set inside the configuration file ~/.node-red/settings.js. It’s good practice to not run node as root user and apply common security practices (like applying firewall rules, etc.).

Now node red can be started (for testing) on an interactive shell (or inside a screen session) which is a good idea on the first launch because one can see debug output:

node-red

This will launch node red on the local host listening on all IPv4 adresses on port 1880 via plain unencrypted http.

Launching Node-RED at boottime on FreeBSD

To launch node-red on FreeBSD at boot time (i.e. automatically) as one would be required for any useful deployment one can use a simple rc.d script. The scrip I’m using is available as a GitHub GIST

Enabling persistent context store

In case one wants context to be persistent - context allows one to store keys and values inside function nodes and many of the nodes store their current state in case they’re not stateless inside flow, node or global context - one has to edit ~/.node-red/settings.js again. Usually the contextStorage is provided as a comment. To enable filesystem backend context store:

contextStorage: {
	default: {
		module:"localfilesystem"
	},
},

This allows context to be persistent over node-red restarts, system crashes and updates.

Solving simple problems

How does one use node red? Basically it’s an event driven flowgraph system. There are nodes that are capable of injecting events into the graph that are then passed down the flow as specified in the flowgraph editor. Messages are normally copied when passed to multiple downstream nodes to allow message modification without affecting different branches.

Since more current versions all messages are passed asynchronously, previously one could see the call stack being traversed and messages being deployed synchronously in order. Simply do not depend on the order of actions taken at branches and you’re on the safe side.

In the following section the most common nodes will be described

Most common nodes

Inject

This is a node that can be used to inject messages either manually triggered which is great for testing and getting started (just place the inject node on your graph, deploy the flow and click on the button on the left side of the node to deploy an arbitrary message).

The injection can also trigger automatically a specific time after flow deployment and if one wants also on a recurring shedule after the initial injection.

One can inject arbitrary messages, timestamp being just the selected default for msg.payload - and one can also select an arbitrary msg.topic

Whenever one enters a name this is the label shown on the flowgraph editor as usual.

Debug

A really useful node. Simply dumps the received messages into the debug window found in the sidebar. This allows to inspect messages and behaviour and is nice when hacking around or performing debugging. After on is done one can simply disable the debug node by using the button on the right side of the node.

One has to select which part of the msg one wants to dump; most commonly this will be either payload or topic. One can of course also dump complex JavaScript objects like arrays or objects and inspect them in the debug view.

Function

This is by far the most flexible node one will ever encounter. It provides the ability to write arbitrary JavaScript functions that exploit the whole functionality of the node.js server. It’s normally used whenever other flowgraph nodes would be either be incapable or too complex to perform a given action.

Note that the function declaration is not written by the user. Node red passes us a simple msg object that describes the message flowing into the function node.

The function can return:

• nothing (simply return) so no message will be sent downstream
• Modify the existing msg and return the object (return msg;)
• Create a new object (for example as return { payload : "Example", topic : "whatever" }; )

Besides returning single messages one can also asynchronously transmit messages using the node.send function:

node.send({
    payload : "Example 1",
    topic : "topic1"
});
node.send({
    payload : "Example 2",
    topic : "topic2"
});

will send two messages asynchronously to all attached downstream nodes.

As with anz other node a function node can have multiple outputs (at the flowgraph that’s selectable at the bottom - just set the desired number of output pins). Now one can return an array containing the objects. If the array contains null no object is sent. If an function node has 3 outputs and one wants to write hello world to the zero’th and second:

return [
   { payload : 'hello world pin 0' },
   null,
   { payload : 'hello world pin 2' }
];

Switch

This is a simple switch/case or routing node. It can also be used to filter messages. It can filter on any property of msg, the flow, global variables, simple JavaScript expressions and environment variables. This node can have an arbitrary number of output ports configured. Note that the default mode is to check all rules and not stop after the first matching one - so the message is passed to every output that matches the given condition. This is especially important when one uses the else branch that matches whenever no other branch matches.

Change

Allows to set, modify, delete or move an property inside the msg or one of the context objects.

Delay

Allows one to simply delay the message deliver. This can be either done for every message, it can be done to rate-limit the flow or it can introduce a random configureable delay.

Interfacing the outside world

Of course an automation system is not much worth when one cannot invoke actions on the outside world. Node red offers a variety of methods to interface to the outside world via network (and also using the serial port via extensions like node-red-node-serialport or directly interfacing a bus system like CANBus via ndoe-red-contrib-canbus, KNX via node-red-contrib-knx or 1Wire via the one wire filesystem and node-red-contrib-owfs, etc. - there is nearly no bus system there is no extension for)

Using MQTT

This is one of the most commonly used message passing methods with node-red and with many home automation solutions. Its also the native protocol of TheThingsNetwork when one wants to do simple actions based on LoRaWAN Nodes. There are two flow nodes as with most message passing frameworks - the mqtt-in and mqtt-out nodes. As usual the in node connects to a message broker and subsribes to a configured topic. One can also specify the QoS level (0 being at-most-once delievery that guarantees that a message is never transmitted more than once but may be lost, level 1 being at-least-once delievery that guarantees that a message is delivered but may be delivered multiple times. The level 2 is the most complex one which guarantees that a message is delivered exactly once and not lost). When using TTN one normally uses QoS level 0. All messages received are transmitted to downstream flows.

The mqtt-out node works similar. It requires a server and topic configuration as well as the specified QoS level for transmission. One can also specify if the retain flag should be set. One can also decide to lease topic, qos and retain unspecified and pass these options - additionally to the payload - via the msg object. Whenever they are specified in the node any specification in msg is ignored.

MQTT is mostly interesting with existing automation solutions as well as with NodeMCU boards that are based on either ESP8266 and ESP32 (note: both links are Amazon affilate links, this pages author profits from qualified purchases) which supports MQTT out of the box with both NONOS and RTOS SDKs (and also with the Arduino modules which makes development really easy).

Using AMQP

This works similar to MQTT but with full blown AMQP support.

Using XMPP

To use XMPP one requires an extension - the one I’m using is node-red-node-xmpp. This extension provides two flowgraph nodes: xmpp-in and xmpp-out. They are used - as expected - to receive and transmit messages. Before deplyoment one has to configure the XMPP JID and Server settings to be used, the nodes establish connections (or try to) directly after deployment. In case of errors they try to reconnect as soon as possible.

The xmpp-in node provides two output pins. The first one delivering messages received on the given JID, the second delivering state information about the XMPP session. All nodes use the msg.payload for message payload and msg.payload for source (in case of receive) or target JID (i.e. sender or receiver JIDs). If one doesn’t want to use msg.topic one can configure a static JID in both nodes. In this case the input node only delivers messages received from the given JID, the output node always transmits to the configured JID.

Using TCP requests

TCP requests work somewhat different. The input node supports two modes - either listening on a local port or connecting to a remote one and returning data received. Data can either be received as buffer, base64 encoded string or raw string. Any time data is received on the TCP port it’s passed as message downstream.

TCP out nodes allow node red to connect to an external service and push data over the connection. It optionally supports encoding binary data with base64 and reconnecting for each event. One has to specify the host and port.

On the other and the TCP output node also supports to respond to incoming connections and write data into an open established connection when received via TCP in.

A TCP in and TCP out node can form a pair to bidirectionally transmit data oder a TCP stream.

The last node supported is tcp request. This node transmits the data passed via a message and pushes the response received from the service downstream.

Using UDP

UDP works similar to TCP nodes. There is an UDP in and UDP out node. The UDP in node creates a message every time an UDP message is received on a local port, UDP out allows one to transmit messages. If one doesn’t want to communicate with a fixed target endpoint one can set the msg.port and msg.ip properties to select the target of the message.

Using HTTP

If one doesn’t want to use a message broker, scrape webpages or provide a REST API to other services HTTP is the choice. It allows to make requests to the outside world using the commonly used HTTP procotol which allows easy interfacing to server applications (REST APIs), scraping webpages, making requests to ESP8266/ESP32 nodes (either using their SDKs or the Arduino frameworks which make running an webserver on these controllers pretty easy), etc.

On the other hand one can declare API endpoints that receive requests and transmit responses (for example as the backend for a JavaScript dashboard or other nodes or services that schould trigger actions, gather reports, etc.)

There are 3 nodes available for that:

• http request transmits an request as specified and optionally transmits a payload supplied via the msg object. The result is returned as message
• To create a local REST endpoint http in is used. This node emits a message whenever a request is made from the outside world.
• To respond to requests received via http in the counterpart node http out is used (this allows to set status code and return payload as specified inside the msg object).

Parsing external data

Node red initialy supplies a whole bunch of nodes that are capable of parsing data - for example parsing CSV files, HTML fetched from external webservice, JSON structures, XML and YAML. These can be supplied via any of the above mentioned methods (or any other mean of data transfer).

Some interesting packages

Sheduler: node-red-contrib-schedx

This is a nice an simply sheduler that can be reprogrammed via inbound messages and provide some basic cron-like sheduling (but also supports keywords like sunset, goldenhour and stuff like offseting and random timing inside the range of the selected time and offset). It’s pretty simple to use. Note that reprogramming via messages does not survive redeployment and system restarts.

Dashboard: node-red-dashboard

This is a pretty simple but effective dashboard offering buttons, dropdown lists, switches, sliders, numeric and text input fields, date pickers, colour pickers, forms, text fields, gauges, charts (automatically collecting data), notifications, filling HTML templates and even audio output inside the browser. It should be sufficient for most simple applications.

Note that one should keep data displayed in graphs down to the necessary minimum since they’re always transferred as a whole bunch of data on every update. This might lead to an unresponsive dashboard in case of usage via unrelieable or slow network connections. In this case one should think about moving away from node red or at least use a full blown application on both sides that interacts with node red using one of the many communication methods (http, tcp, etc.).

node-red-node-openweathermap

Interesting for home and garden automation. Periodically polls the openweathermap service and if the weather forecast has changed returns a message containing the new forecast. Requires an (free) API key for OpenWeatherMap. It is possible to query either current forecast or 5 day forecast.

This node might be interesting for people doing garden or home automation (or other agrar) automation - or people automating HVAC equipment.

node-red-node-xmpp

Easy to use interface to XMPP servers. Requires an external XMPP server and account and allows to send and receive arbitrary XMPP messages. Allows easy notification services, chatbots and also more unconventional XMPP messaging systems. Really easy to use.

node-red-contrib-amqp

If MQTT is not enough or not the message broker of choice inside the used environment this plugin allows interfacing to full blown AMQP infrastructures or simple AMQP servers (like RabbitMQ).

node-red-contrib-machine-learning

In case one wants to do some data analysis the node-red-contrib-machine-learning package provides a nice interface to some basic tasks like decision tree classifiers, neuronal networks implemented on top of tensorflow, random forrest classifiers, SVM, k-means classifiers and much more.

This article is tagged:

• Node Red
• Electronics
• DIY
• Home automation
• Sonoff
• Programming