Telegraf: Your Go-Powered Data Collection Agent for Metrics & Logs
Imagine you're building a software system. You've got servers, databases, microservices, and maybe even some IoT devices. How do you know if everything is running smoothly? How do you detect issues before they become major problems? That's where Telegraf comes in!
Telegraf is a plugin-driven agent that helps you collect, process, aggregate, and write metrics, logs, and other arbitrary data. Think of it as a data Swiss Army knife for your infrastructure.
Here's how it's incredibly helpful for software engineers
Observability
Telegraf is a cornerstone of observability. It allows you to gather vital information about your applications and infrastructure, giving you insights into their health and performance. This is crucial for
Monitoring
Keep an eye on CPU usage, memory consumption, network traffic, application-specific metrics (like request latency or error rates), and much more.
Troubleshooting
When something goes wrong, having detailed metrics and logs helps you quickly pinpoint the root cause of the problem.
Performance Optimization
Identify bottlenecks and areas where your system can be made more efficient.
Capacity Planning
Understand resource usage trends to anticipate future needs.
Data Collection Versatility
With its vast array of plugins, Telegraf can collect data from almost anywhere
System Metrics
CPU, memory, disk I/O, network.
Application Metrics
From databases (MySQL, PostgreSQL, MongoDB, Redis), message queues (Kafka, RabbitMQ), web servers (Nginx, Apache), and custom applications.
Logs
Parse and collect logs from files, syslog, or other sources.
IoT Devices
Collect data from sensors, smart devices, etc.
Cloud Platforms
Integrate with cloud-specific monitoring services.
Data Processing and Transformation
Telegraf isn't just a data vacuum cleaner. It can also
Filter Data
Only collect the metrics you care about.
Aggregate Data
Combine multiple data points into a single, more meaningful value (e.g., calculate averages, sums).
Add Tags
Enrich your data with metadata for better organization and querying.
Apply Transformations
Convert data types, rename fields, etc.
Integration with Data Stores
Telegraf can output data to a wide range of destinations (output plugins), including
InfluxDB
The time-series database from InfluxData, often used in conjunction with Telegraf for powerful time-series analysis and visualization (with Grafana).
Other Databases
Prometheus, Elasticsearch, Graphite, OpenTSDB, Splunk, and more.
Message Queues
Kafka, MQTT (as you specifically mentioned!), RabbitMQ.
Cloud Services
AWS CloudWatch, Google Cloud Monitoring.
Simplicity and Efficiency (Go-powered!)
Telegraf is written in Go, which means it's lightweight, has a small memory footprint, and is very efficient. This makes it ideal for deploying on various systems, even resource-constrained ones.
Getting Telegraf up and running is pretty straightforward. Here's a common way to do it
Download and Install
You can find official installation instructions for various operating systems on the InfluxData website. Here are common approaches
Linux (Debian/Ubuntu)
wget -qO- https://repos.influxdata.com/influxdata-archive_compat.key | sudo gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/influxdata.gpg > /dev/null
echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata.gpg] https://repos.influxdata.com/debian stable main' | sudo tee /etc/apt/sources.list.d/influxdata.list
sudo apt-get update
sudo apt-get install telegraf
Linux (RHEL/CentOS)
sudo tee /etc/yum.repos.d/influxdata.repo <<EOF
[influxdata]
name = InfluxData Repository - RHEL/\$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdata-archive_compat.key
EOF
sudo yum install telegraf
macOS (using Homebrew)
brew install telegraf
Windows
You can download the Windows installer from the InfluxData downloads page and follow the installation wizard.
Configuration File
After installation, Telegraf comes with a default configuration file, usually located at /etc/telegraf/telegraf.conf on Linux/macOS or in the installation directory on Windows. This is where you tell Telegraf what to collect and where to send it.
The configuration file is well-commented, making it easy to understand. It's divided into sections for global_tags, agent settings, input plugins, processor plugins, aggregator plugins, and output plugins.
Let's create a simple telegraf.conf example to demonstrate collecting basic system metrics and sending them to an MQTT broker.
First, you'll need an MQTT broker running somewhere (e.g., Mosquitto).
# Global tags can be applied to all metrics collected by this Telegraf instance.
[global_tags]
host = "$HOSTNAME" # Use the hostname as a tag
# Agent settings define how Telegraf itself behaves.
[agent]
interval = "10s" # Collect data every 10 seconds
round_interval_to_period = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s" # Flush data to outputs every 10 seconds
flush_jitter = "0s"
precision = ""
hostname = "" # If empty, this is set to the host's hostname
omit_hostname = false
###############################################################################
# INPUT PLUGINS #
###############################################################################
# Collect system CPU usage
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
# Collect system memory usage
[[inputs.mem]]
# Collect system disk usage
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "overlay", "squashfs"]
# Collect network interface stats
[[inputs.net]]
###############################################################################
# OUTPUT PLUGINS #
###############################################################################
# Send metrics to an MQTT broker
[[outputs.mqtt]]
servers = ["tcp://localhost:1883"] # Replace with your MQTT broker address and port
topic_prefix = "telegraf/" # All metrics will be published under this prefix
data_format = "json" # Output metrics in JSON format
qos = 0 # Quality of Service for MQTT messages (0, 1, or 2)
#username = "your_username" # Uncomment if your MQTT broker requires authentication
#password = "your_password" # Uncomment if your MQTT broker requires authentication
Explanation of the example
[global_tags]
We're adding a host tag to all metrics, which will be the hostname of the machine running Telegraf. This is super useful for identifying where metrics are coming from.
[agent]
These settings control Telegraf's behavior, such as how often it collects (interval) and flushes (flush_interval) data.
[[inputs.cpu]], [[inputs.mem]], [[inputs.disk]], [[inputs.net]]
These are input plugins. Each [[inputs.PLUGIN_NAME]] block enables a specific input. Here, we're collecting
CPU usage (per CPU core and total).
Memory usage.
Disk usage (ignoring temporary filesystems).
Network interface statistics.
[[outputs.mqtt]]
This is the output plugin.
servers
Specifies the address(es) of your MQTT broker. In this example, it's tcp://localhost:1883, assuming your MQTT broker is running on the same machine on the default port.
topic_prefix
Metrics will be published to topics like telegraf/cpu, telegraf/mem, etc.
data_format = "json"
Crucially, this tells Telegraf to format the metrics as JSON before sending them. This is often preferred for downstream processing.
qos
Quality of Service for MQTT messages. 0 means "at most once."
username and password
Uncomment and configure if your MQTT broker requires authentication.
When Telegraf sends data to MQTT with data_format = "json", the payload will look something like this (simplified for clarity)
For CPU metrics
{
"name": "cpu",
"tags": {
"cpu": "cpu-total",
"host": "my_server_hostname"
},
"fields": {
"usage_guest": 0,
"usage_guest_nice": 0,
"usage_idle": 90.12,
"usage_iowait": 0.5,
"usage_irq": 0,
"usage_nice": 0,
"usage_softirq": 0,
"usage_steal": 0,
"usage_system": 2.3,
"usage_user": 7.08
},
"timestamp": 1678886400
}
For Memory metrics
{
"name": "mem",
"tags": {
"host": "my_server_hostname"
},
"fields": {
"active": 2048567296,
"available": 1024567296,
"available_percent": 50,
"buffered": 123456789,
"cached": 987654321,
"free": 512345678,
"total": 4096000000,
"used": 3071432704,
"used_percent": 75
},
"timestamp": 1678886400
}
Key elements in the JSON output
name
The measurement name (e.g., "cpu", "mem").
tags
Key-value pairs of metadata (e.g., host, cpu).
fields
The actual metric values (e.g., usage_idle, used_percent).
timestamp
The Unix timestamp when the metric was collected.
Once you have your telegraf.conf file set up
Start the Telegraf service (Linux/macOS)
sudo systemctl start telegraf
sudo systemctl enable telegraf # To start on boot
Run Telegraf in the foreground (for testing/development)
telegraf --config /etc/telegraf/telegraf.conf --test # Test configuration
telegraf --config /etc/telegraf/telegraf.conf # Run in foreground
(Replace /etc/telegraf/telegraf.conf with the actual path to your config file if you've put it elsewhere).
The real power of Telegraf comes from its extensive plugin ecosystem. As a software engineer, you'll find these invaluable
Custom Application Metrics
Use the inputs.exec plugin to run a script that outputs metrics in a Telegraf-compatible format (InfluxDB Line Protocol, JSON, etc.). This is perfect for instrumenting your own applications.
Log Parsing
The inputs.tail plugin can follow log files, and processors.parser can parse them into structured data, which you can then send to a log management system (like Elasticsearch).
HTTP/API Data
The inputs.http or inputs.http_listener plugins allow Telegraf to pull data from HTTP endpoints or listen for incoming HTTP requests with metrics.
StatsD/Prometheus Scrape
If your applications are already emitting metrics in popular formats like StatsD or Prometheus, Telegraf has input plugins to consume them.
Processor Plugins
Beyond parser, you can use filter, converter, rename, starlark (for powerful custom scripting), and more to transform your data before it's sent out.
