Logs, API and diagnosis

Why this matters

The EMQX Dashboard makes the state of the broker visible, but once the question gets specific — why was this client rejected, why did this rule never match — you still have to go to the log and to Trace. EMQX also exposes a REST API, so plenty of this you can automate without opening a browser at all.

This chapter is about knowing where to look when something breaks. Start by reading log levels and Log Trace, then narrow the problem down with the tools in the Diagnose module, and finish with the concepts behind the REST API and its health check endpoint. In practice those four are what you reach for the next time something goes wrong.

Core concepts

EMQX writes its log to two output streams: Console Log (out to the console) and File Log (into a file). The default level is warning. You adjust both on the Dashboard's Management → Logging page.

Log Trace is live debug logging that picks out one specific client ID, IP address, topic (wildcards allowed) or rule ID and nothing else. It is what you want in production, when you do not want to be buried in log lines.

The REST API (/api/v5/...) is the HTTP admin surface of EMQX, authenticated with an API key over HTTP Basic. Think of it and the Dashboard as two ways of driving the same engine.

The Diagnose module gathers tools such as WebSocket Client, Log Trace, Slow Subscriptions and Topic Metrics in one place, so you can test and pin down a problem quickly.

Terms at a glance

Hands-on

1. Look at the current log settings

   In the sidebar, Management → Logging. Switch between the Console Log and File Log tabs to change the level (warning by default). The change takes effect immediately.

2. Send one test message with WebSocket Client

   In the sidebar, Diagnose → WebSocket Client. Open a connection, then use subscribe/publish to check quickly how a topic behaves.

3. Create a Log Trace

   In the sidebar, Diagnose → Log Trace → Create. Choose the client ID, IP address or topic you want to trace, and collection starts.

4. Call the REST API to check

   Save the API key somewhere, then GET /api/v5/nodes with curl to verify that your API answers with JSON.

Log levels and logging settings

EMQX supports 6 levels (6 of the 8 in RFC 5424). The default is warning, and from weakest to strongest they are debug < info < notice < warning < error < critical.

• debug: fine-grained debugging data such as variables and functions.
• info: minor anomalies such as an authorization denial, and the result of a configuration change that succeeded.
• warning: things that may need a look — a dropped connection, a connection timeout, an authentication failure.
• error: cannot reach an external database, a subscription that does not exist, and similar errors.
• critical: a configuration error that stops a component from starting, for example.

Both the Console and File handlers can be adjusted on the Logging page. For the file handler you can set the file name (log/emqx.log by default), the maximum number of rotated files (10 by default), the rotation size (enabled by default) and so on.

There is also Log Throttling: within a time window it records only the first of a repeated event and counts the rest, which keeps the log from exploding. It is on by default, with a window of 1 minute.

Using Log Trace on one client or topic

Log Trace is live debug-level logging aimed at one specific target, which suits production far better than turning the whole log up to debug. You will find it under Diagnose → Log Trace.

To create one:

1. Click Create, and under Type choose Client ID, Topic (wildcards allowed), IP Address or Rule ID.
2. Client ID and IP have to be entered in full; only Topic takes wildcards.
3. Pick the start/end times and click Create to start collecting.

From the list you can view or download that trace's log. The system runs at most 30 traces at a time; each node caps its trace logs at 512MB, and once that is full it stops appending and raises a warning in the main log. The trace logs are also on the server itself, in the /data/trace directory.

Test Rule creates a trace automatically and deletes it when the test ends, so debugging a rule gets you the log from the other direction too.

The Diagnose modules at a glance

The Dashboard's Diagnose module is where the debugging tools land. It holds:

• WebSocket Client: a built-in MQTT test client. Open a connection, subscribe, publish, and check quickly how a client behaves, without standing up a separate tool.
• Topic Metrics (Enterprise): counts and rates of messages received, sent and dropped for a specific topic.
• Slow Subscriptions (Enterprise): finds subscriptions whose delivery time is over a threshold.
• Log Trace: the targeted log tracing described above.
• Alerts (Enterprise): current and historical system alerts.

REST API concepts and health checks

The EMQX REST API follows the OpenAPI 3.0 specification, and every path starts with /api/v5. You can open http://<host>:18083/api-docs/ and try it straight from Swagger UI.

The REST API authenticates with an API Key (create one in the Dashboard under System → API Key). HTTP Basic takes the API Key as the username and the Secret Key as the password. A Dashboard user account cannot call the API directly.

curl -X GET http://localhost:18083/api/v5/nodes \
  -u <your-api-key>:<your-api-secret> \
  -H "Content-Type: application/json"

If you are putting a load balancer in front, EMQX offers a health check endpoint, GET /api/v5/load_rebalance/availability_check: a healthy node answers 200, a node that has been evacuated answers 503.

Remember to write API keys, secrets and tokens as placeholders, every one of them. Never put the real value into your own notes or into a log.

Troubleshooting

• The log you want keeps getting cut short: by default Log Throttling records only the first of a repeated event. To get the full detail, set the log level to debug — throttling is disabled at that level.
• Log Trace does not find its target: check that the client ID / IP you entered is the exact value. Only Topic supports wildcards. Once a trace has ended you can find the file in /data/trace.
• The REST API returns 401: usually you called it with a Dashboard user account. Use an API Key/Secret pair created under System → API Key instead.
• Slow Subscriptions / Topic Metrics / Alerts have disappeared from the sidebar: all three belong to EMQX Enterprise, so they are not visible under Diagnose in the sidebar in Open Source 5.8.9.

FAQ

Official sources

• EMQX 5.8 log levels and settings (pinned docs branch)
• EMQX 5.8 Log Trace (pinned docs branch)
• EMQX 5.8 Diagnose module (pinned docs branch)
• EMQX 5.8 REST API (pinned docs branch)
• Woow EMQX add-on README (pinned commit)