Rule Engine

Why this matters

Mosquitto hands every topic's messages to its subscribers untouched, so you never get data that has been picked over or reshaped. If you want to strip the noise out of a sensor reading in a smart home, move it to another topic, or send a notification the moment a client drops off, what you need is a broker that can process the data itself.

EMQX's Rule Engine is that capability: SQL-like syntax describes where the data comes from, how to pick it, and what to do with it at the end, which turns the broker from a plain forwarder into the hub of a data pipeline. By the end of this chapter you can write your own first rule and know how to check it with the built-in test tool.

Core concepts

At its core the Rule Engine is three stages, source → transformation → action:

• Data source: where the rule gets its data. It can be MQTT messages (one topic or several), an MQTT event ($events/…), or data handed back by Data Integration.
• Transformation: the rule SQL decides which records to take, which fields to keep, and what conditions and functions to apply. FROM names the source, WHERE adds the condition, and SELECT decides the output.
• Action: what happens once a rule matches. EMQX 5.8.9 offers republish, console (print to the console or the log) and Forwarding to Sinks (send it to an external system).

Which layer do rules live in? EMQX 5.x files them under Data Integration, and you create or edit one from the Dashboard sidebar at Integration → Rules. The Rule Engine handles more than messages: it also handles events such as a client connecting or disconnecting.

Terms at a glance

Hands-on

The fastest way to learn is to work through the t/#a/1 republish rule from the official EMQX tutorial.

1. Open the rules page

   In the EMQX Dashboard sidebar click Integration → Rules, then Create at the top right to open "Create Rule".

2. Fill in the rule details and write the SQL

   Give the rule a name and a note, then type SELECT * FROM "t/#" into the SQL Editor so the rule selects every message whose topic matches t/#.

3. Test the SQL with Try It Out

   Turn on the Try It Out switch on the Create Rule page, choose the matching Data Source, fill in the simulated data (Client ID, Username, Topic, QoS, Payload and so on) and click Run Test. You should see Test Passed, and Output Result on the right shows what the SQL produced.

4. Add an action

   Click Add Action on the right, choose Republish, set the target topic, QoS and Payload (${payload}), then click Add to return to the Create Rule page.

5. Create the rule and trigger it

   Back on the Create Rule page, click Create at the bottom to finish. Subscribe to a/1, publish a message on topic t/1, and the republished message should arrive on a/1.

Rule SQL: SELECT/FROM/WHERE

The basic shape of the rule SQL is:

SELECT <fields and expressions> FROM <data source> [WHERE <condition>]

FROM names the data source, in one of two main shapes:

• topic: for example SELECT * FROM "t/#" handles every message whose topic matches t/#. Separate several topics with commas, as in FROM "t/1", "t/2". A topic must be wrapped in double or single quotes.
• event: for example SELECT clientid FROM "$events/client_connected" WHERE clientid = 'c1' handles the event of a client connecting successfully. Event topics start with $events/ and cannot be read with an ordinary subscription.

WHERE adds a further filter. A field can be metadata (such as clientid or username) or a field inside the payload (dot syntax such as payload.x.y, as long as the payload is JSON/Map); you can also combine conditions with and/or.

SELECT picks the fields to output, and can rename them:

SELECT clientid, payload.clientid as myclientid FROM "t/#"

The output is a JSON object. Once the SQL has run, a later action can reference its fields with ${field}.

Expressions support arithmetic (+/-/*/div/mod), comparison (=, <>, <, >…) and logic (and/or), plus a long list of built-in functions. A separate FOREACH statement can emit several results at once (splitting an array of sensor readings in the payload into one message each, for example); learn it when you get to that point.

Data fields and built-in functions

Which fields a rule can reference depends on the data source. For an MQTT message, the metadata gives you these common fields:

An event source (such as $events/client_connected) adds fields like clientid, username, keepalive, is_bridge and connected_at; $events/client_disconnected also carries reason, which says why the client dropped. The official docs list every event and its fields.

Built-in functions can be used inside SELECT and WHERE. The common groups are:

• Math: abs, floor, ceil, round, power, sqrt…
• Type: is_array, is_bool, is_null…
• String: upper, lower, tokens, concat…
• Other: now_timestamp, now_rfc3339, uuid_v4, date conversion, mapping/array operations, and the advanced jq processor.

SELECT upper(clientid) as cid FROM "t/#"

If the payload is not JSON, you have to cast it or use a function before dot syntax will work; EMQX also has a Schema Registry guide for non-JSON formats.

Troubleshooting

• The rule never fires: check that the topic pattern in FROM matches the topic being published to (wildcards, letter case), that the rule is enabled, and that the event name is not misspelled ($events/client_connected, for example).
• The SQL output is empty, or the fields are wrong: check the official field reference and confirm the field name exists on that data source; metadata and payload are two separate layers of fields.
• The payload is not JSON but you want a field inside it: dot syntax only works on JSON/Map. For anything else, use a cast function or run it through a Schema first.
• The rule matches but the action does not run: use "Test the Rule" in Try It Out to simulate the matching event or publish, or publish a test message directly, then read the Action error in that run's log.

FAQ

Official sources

• EMQX 5.8 Rule Get Started (pinned docs branch)
• EMQX 5.8 Rule SQL Syntax (pinned docs branch)
• EMQX 5.8 Rule Events and Fields (pinned docs branch)
• EMQX 5.8 Rule Engine (pinned docs branch)
• EMQX 5.8 Built-in SQL Functions (pinned docs branch)