Understanding the Log Pipeline#

The log pipeline connects rsyslog inputs, rulesets, and actions: receive → parse/transform → deliver. This page introduces the concept with diagrams and a commented example.

Status:: Draft
Audience:: intermediate
Tier:: conceptual core

Overview#

rsyslog processes logs through a log pipeline — internally known as the message pipeline. An input receives events, a ruleset filters or transforms them, and actions deliver them.

Queues connect these stages to provide buffering and concurrency.

Pipeline stages#

Input – how rsyslog receives data (e.g., imuxsock, imjournal, imtcp, imudp, imfile).
Ruleset – processing logic (filters, parsing, transformation). Inputs can be assigned to different rulesets.
Actions – destinations (omfile, omfwd, omrelp, omelasticsearch, ompgsql).
Queues – buffers between stages; tune for throughput and reliability.

Example: parse and transform JSON, then write#

This example parses JSON from the raw payload, converts dotted keys into nested objects (unflatten), and writes only the structured subtree.

Input → transform → output (commented)#

# Output only the structured subtree we build during processing
template(name="outfmt" type="subtree" subtree="$!qradar_structured")

# Load parser/transformer modules used in this pipeline
module(load="mmjsonparse")
module(load="mmjsontransform")

# Step 1: Parse JSON from the raw message payload.
# useRawMsg="on" is robust when a syslog header may or may not be present.
action(
  type="mmjsonparse"
  container="$!qradar"
  mode="find-json"
  useRawMsg="on"
)

# Step 2: Only proceed if parsing succeeded to avoid partial outputs
if $parsesuccess == "OK" then {
  # Convert dotted keys (e.g., src.ip) to nested JSON ("src": {"ip": ...})
  action(
    type="mmjsontransform"
    input="$!qradar"
    output="$!qradar_structured"
    mode="unflatten"
  )

  # Step 3: Write just the structured subtree to a file
  action(
    type="omfile"
    file="/var/log/qradar-structured.json"
    template="outfmt"
  )
}

Design patterns (with diagrams)#

1) Fan-out (branching to multiple actions)

Best practice: log locally for audit, ship reliably for centralization, index for search.

2) Staged processing (ruleset chaining)

Use call to hand off between rulesets. Gate transforms with $parsesuccess.

3) Isolate workloads with per-ruleset queues

Tune queue sizes/workers per workload; avoid burst interference.

Key ideas & tips#

Gate transforms using $parsesuccess to avoid writing malformed data.
Prefer subtree templates for clean, stable downstream contracts.
Filter early to drop noise before expensive transforms.
Configure action/ruleset queues for resilience under load.
Compose stages with modules (mmjsonparse, mmjsontransform, mmnormalize) as needed.

Troubleshooting checklist#

Validate: rsyslogd -N1 (syntax & module checks).
Inspect fields: temporarily add action(type="omfile" file="/tmp/debug" template="RSYSLOG_DebugFormat").
Verify JSON presence: mode="find-json" parses only if JSON exists.
Check queues: monitor queue sizes/latency; increase workers if needed.