posted on May 21, 2024 by Ilija M
Collectd is a Unix daemon capable of collecting system and application performance metrics. Metrics are reports that indicate the performance of a system’s aspect at any given time. They can also be regarded as snapshots of specific functions of a system, which can be delivered at regular intervals.
Typically, a daemon is a process that runs in the system’s background and can generate metrics. Each data measurement is delivered over the network to a server, which evaluates the information. While collectd can collect and transmit metrics, it cannot evaluate these metrics on its own.
For this reason, a tool is needed to collect and visualize these metrics. There is an agent that operates on a server, which is configured to measure specific attributes while relaying the information to a preset destination.
Collectd involves an extensive measurement engine, which ensures that users can gather a wide variety of data. At the moment, collectd is most often adopted for getting insight into core infrastructure. To explain more, this comprehensive guide will provide a deep insight into what users must know about Collectd and the tools it can use.
What Does Collectd Measure?
Collectd is capable of measuring a wide variety of metrics. Essentially, collectd enables the CPU, interface, load, and memory plugins. All these are great for gathering essential metrics from servers. Users can add about 100 plugins to collectd. Many of these plugins specialize in networking topics, while others relate to popular devices, from UPS to sensors.
Advantages of Using collectd
Some important benefits of using collectd are briefly introduced in this section.
- It’s Free A major benefit of using collectd is that you are not charged per agent. Besides, collectd can be pushed to as many systems as possible. Any kind of plugin can also be used with collectd.
- It’s lightweight From a memory and disk standpoint, Collectd is designed with a small footprint. With its modular architecture, the agent can have a minimal size to perform the job.
- Collectd makes users less dependent on software vendors Collectd can only gather and relay metrics. This information can be directed to any tools that can consume that data.
- Users can achieve flexibility in what they gather With collectd, users can specifically affirm what metrics must be captured and the frequency you want them captured. For this reason, users can scale back the collection of their metrics to the data that is needed for observability purposes. Collectd can also be used to gather data every few minutes on systems that don’t allow mission-critical SLAs.
However, certain challenges are associated with working with collectd. One of these includes actively managing the manner of distributing agents.
How Do Splunk and collectd Work Together?
Splunk can help in indexing and analyzing both metrics and machine data. Even though it is a required step to carry out analysis and indexing of data effectively, the focus of Splunk is to offer analytics around infrastructure data. On the other hand, collectd places emphasis on gathering and transmitting data to analytics tools free and easy. For this reason, by combining Splunk and collectd, users can analyze a great amount of data securely and with ease.
Nowadays, placing data under analysis goes beyond creating dashboards or alerting on thresholds. As your observability needs to keep growing, it is important to adopt a top software like Splunk to help you identify issues, trends, and relationships that may be difficult to observe with the naked eye.
Essential metrics, including those gathered by collectd, only tell part of
the story. They are needed for monitoring. Even though it might be difficult to get an answer to questions like “Why is this problem happening?”, it can help in answering other questions like “what is going on?” Correlating metrics and logs in a time-series manner can help users add the context that originates uniquely from logs.
Is Collectd cloud-native?
The answer to this is not a straightforward one. Cloud-native systems are characterized by the following:
- They’re Packaged in Containers There are over five million pills across various repos, which makes it likely that collectd can be a containerized package while also being widely used by the container community.
- They are Managed Dynamically Writing a config file dynamically may be the ideal way possible to manage the behavior and state of collectd. You will even find certain open source projects that adopt an environment variable as a technique that can help to dynamically configure the state of collectd, which can help provide runtime configurations. However, it can become more challenging to configure and manage the state of collectd, thanks to the lack of interfaces. The costs of operating and maintaining when operating it in a containerized environment can be enhanced.
Extensibility of Collectd
Collectd offers top performance, flexibility, and reliability. As a result, it can be considered a good, battle-tested system monitoring agent available for users. Besides this, it also boasts an official catalog of over 130 plugins. These plugins perform various functions, including pulling vital system metrics (like CPU, memory, and disk usage) and technology-specific metrics for new technologies such as NetApp, MySQL, and more.
Many of these plugins help extend data forwarding and collection abilities of collectd. This allows it to play the role of a central collection point for multiple collectd agents operating in an environment. Like any other open-source agent, the strength of collectd is its remarkable extensibility. Take, for instance, the entire plugins that are present in the official collectd documentation that are written in C. This ensures that collectd can run with minimal effect on the host system while offering extensibility without the need for extra dependencies. This is not necessarily the case. For instance, the plugin on collectd for Python requires certain versions of Python to be present on the host system.
Yet, many of these plugins do not require a dependent set of libraries. Yet, C comes with its challenges. After all, it is a language that is not widely discussed or used. Luckily, other paths are available for extending the collectd’s core capabilities. With the collectd GO plugin, developers can write powerful extensions of collectd in GO, representing a similar language to C. It can be adopted without the need for any external dependencies.
Analyzing Data from Collectd
Just like any other monitoring solution, the tool is only as good and reliable
as the data. Likewise, the data is only as good as its application. A strategy ensuring users can collect it all and figure it out later can be a good approach. It is always good to start with this strategy to tackle classic huge data problems that involve data that extends to multiple environments, sources, and formats.
The first thing to do is to understand the answers to certain questions. Some of them are introduced below:
Are the hosts performing as expected?
- Is the CPU under or over-utilized?
- Are the hard drives filling up?
- Is there network traffic going to servers as expected?
Collectd boasts both out-of-the-box plugins for these data sources. This is to make it possible for users to deliver relevant data to their environment. From here, it is only right to understand how collectd can be configured.
Configuring Settings for Plugins
This section will take a deep insight into how plugins can be set up. After all, each plugin is designed with its distinct settings. For this reason, when collectd is set up for the first time, it is wise to set the logging to info, which ensures that these settings are right. Through a syntax error, the collectd daemon can fail. Once a user can be sure of their settings, then it is possible to change the log level to something precise.
#####################################################
# Plugin configuration #
#####################################################
<Plugin logfile>
LogLevel info
File “/etc/collectd/collectd.log”
Timestamp true
PrintSeverity true
</Plugin>
<Plugin syslog>
LogLevel info
</Plugin>
What this part of the conf file ensures is that it makes it possible for the users to gather vital information about collectd while figuring out any form of errors in the log files.
Users can either troubleshoot agent deployment issues by making use of these logs in the command line, or they can rely on analysis tools, such as Splunk, which can help in achieving better analytics.
- Logfile For this, the user is setting the log level while adjusting some key settings, including location (File), whether to add the severity or to add a timestamp. The level can be changed to debug for a more verbose log or a WARN or ERROR for critical errors only.
- Syslog This informs collectd about the level to write to the systemlog. However, this may be irrelevant, especially if data is being sent to a collectd.log. It is relevant to complete this particular step if Syslog is used to troubleshoot and investigate a host.
- CPU This plugin will facilitate the collection of measurements by CPU. To gather aggregate CPU measures across every CPU core, users can disable this setting. ReportByState splits the CPU metrics into the actual CPU state (that is, system, user, idle, etc.). If this is not established, the metrics will be for Idle and aggregated Active state only.
<Plugin cpu>
ReportByState true
ValuesPercentage true
</Plugin>
Users may not consider the CPU values. However, this can be very relevant if the aim is to make sure that the logical CPUs are utilized by workloads passed to the system. For most applications that involve monitoring, it can prove enough to get utilization in aggregate simply.
- Interface For this, no certain settings are available for the interface. For this reason, users can gather network data from every interface.
<Plugin interface>
</Plugin>
Network traffic remains one of the first things to verify anytime a server experiences a reduction in overall resource utilization. A common challenge in a data center includes losing connectivity in a local network.
If there is complete idle CPU usage, it is not impossible that the server is not getting traffic from any source. This configuration uses a basic setup to gather network data for all interfaces. With the collectd interface plugin, a user can neglect specific interfaces when required.
It is worth noting that there are various reasons why an interface can be omitted. Since no physical hardware is related to loop back interfaces, subjecting their performance to measurements is not valuable. By explicitly listing interfaces, it is possible to ignore or completely get rid of monitored interfaces dynamically by using regular expressions.
All the interfaces should be aggregated together when analyzing this data. After all, as a user, you might need basic activity across all the interfaces. The collectd interface plugin gathers Net I/O as Octets. According to those who embrace music, an octet is a group of 8 musicians, a small ensemble. Similarly, in computing, octets are roughly equivalent to 8 bits or a byte. It is worth pointing out that various interface metrics come out as counters. This implies that the values stay constant unless a reset to 0 is done.
Now, how can Splunk be used with collectd? The next section will explain more about this.
Using Splunk With Collectd
Getting data from Collectd into Splunk can facilitate a wide variety of write plugins, which could be adopted to get metrics into Splunk. Many of these are targeted toward a product or service, including TSDB, Kafka, and MongoDB, while others allow more generic technologies. Investing collectd metrics in Splunk involves the use of the write_http plugin. This sends metrics in a standardized JSON payload to any HTTP endpoint through a POST request.
On the other end, it is easy to configure the HTTP Event Collector (HEC) endpoint in Splunk to receive and parse these payloads. With this, it is easy to ingest them into a metrics index.
Before the write_http plugin is set up, the HEC must be enabled to gather data. Since these data input details are needed for the collectd configuration, it is important to configure this data input before setting up collectd. Follow these simple steps:
- In Splunk Web, click Settings > Data Inputs.
- Under Local Inputs, select HTTP Event Collector.
- Ensure that the HEC is enabled. To do this, click Global Settings. For all tokens, select Enabled if this button is not already selected. Note the value for HTTP Port Number, which you’ll need to configure collectd. Click Save.
- Configure an HEC token for sending data by clicking New Token.
- On the Select Source page, for Name, enter a token name, for example, “collectd token”.
- Leave the other options blank or unselected.
- Select Next.
- On the Input Settings page, for Source type, click Select.
- Click Select Source Type, then select Metrics > collectd_http.
- Next to Default Index, select your metrics index or click Create a new index to create one.
- If you choose to create an index, in the New Index dialog box, enter an Index Name. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. Index names cannot begin with an underscore or hyphen. Then, for Index Data Type, select Metrics. After this, configure additional index properties as needed.
- Select Save. Click Review, and then click Submit.
- Copy the Token Value that is displayed, which you’ll need to configure collectd.
Cloud and Collectd
In this section, we will look into how Cloud and Collectd can be combined.
Using Collectd in the Cloud
The adoption of the Cloud is on the rise at a dramatic rate year every year. Server refresh projects are encouraging organizations to focus on moving to the cloud to lower infrastructure and human capital costs for adapting well to the rising demand from the business.
When a multi-cloud approach is considered, users must know how to develop a unified view of servers across cloud and on-premises vendors. Users can monitor and troubleshoot risk and performance by embracing a standard cloud-native approach to gathering system statistics. Also, collecting and comparing these metrics with logs present on these servers in a single solution will ensure that the time required for identification is reduced.
AWS
Getting Started: Install and Configure collectd on the EC2 server
- Launch a new Amazon Linux AMI. Note that all other AMIs/OSs can be used. However, the installation instructions may be different).
- SSH to the AMI to install and configure collectd:
ssh -i <ssh-key> ec2-user@<aws-externalip>
- Install collectd: http://docs.splunk.com/Documentation/Splunk/7.0.0/Metrics/GetMetricsInCollectd. When you are on an Amazon Linux AMI, use the command provided below to install AWS CloudWatch’s version.
sudo yum -y install collectd
By running the following command, you can easily confirm if you have successfully installed collectd and that you are running version 5.6 and above, which is needed to send metrics to Splunk: collectd -h
- Install write_http and disk plugins:
yum -y install collectd collectd-write_http.x86_64
yum -y install collectd collectd-disk.x86_64
- Configure collectd: adding system-level plugins to test integration. To achieve this, you can create a new configuration file.
You can get started by using this sample collectd file:
https://s3.amazonaws.com/splunk-collectd-beginnerguide/collectd.beginnerguide.conf. Then, you should configure the <splunk_host>, <hec_port>, and <hec_token> in the configurations provided with your environment values. Then, save these changes. After this, overwrite the local collectd.conf file with the newly configured version.
curl -sL -o collectd.beginnerguide.conf https://s3.amazonaws.com/splunk-collectd-beginnerguide/collectd.beginnerguide.conf
vi collectd.beginnerguide.conf
By making use of vi, you can update the content of the file. Then, save your updates.
- Overwrite the local collectd.conf with the new updates:
sudo cp ./collectd.beginnerguide.conf /etc/collectd.conf
- Restart collectd: sudo /etc/init.d/collectd restart
Analyze Your Metrics
Follow the steps mentioned below:
- Open Search and Reporting in the Splunk Web UI.
- Query for the list of metrics by name being reported:
Then, list metrics names collected: | mcatalog values(metric_name)
- Use mstats to create aggregations as required.
- Ensure that you average CPU idle over the specified time window: | mstats avg (_value) WHERE metric_name=cpu.percent.idle.value.
- Average CPU idle over specified time window with a specified span: | mstats avg(_value) WHERE metric_name=cpu.percent.idle.value span=5mins.
- Then, average CPU idle over specified time window with a specified span split by host: | mstats avg(_value) WHERE metric_name=cpu.percent.idle.value span=5mins by host.
Troubleshooting:
Is your Collectd proving impossible to start? Then, have a look at the error logs. Most likely, you need to install a missing plugin.
Google Cloud Platform (GCP)
To get started: install and configure collectd on the GCE VM Instance. Follow the instructions introduced below:
- Create a new VM instance, using default Debian GNU/Linux 9 (stretch) as the Boot disk.
- SSH to the AMI to install and configure collectd, gcloud compute ssh
[INSTANCE_NAME].
- Run the following commands:
- sudo apt-get update
- apt-get install –force-yes –assume-yes collectd
It is worth noting that these instructions will vary, based on the OS you have selected.
Configure collectd: adding system-level plugins to test integration.
- Create a new configuration file: vi collectd.conf.
- The content of this file should be added into collectd, collectd.beginnerguide.conf
- Configure the <splunk_host>, <hec_port> and <hec_token> in the configurations provided with your own unique values.
- Save your changes.
- Overwrite local collectd.conf file, cp ./collectd.conf /etc/collectd/collectd.conf.
- Start collectd: sudo /etc/init.d/collectd restart.
Conclusion
With the use of Collectd, system administrators can maintain a form of overview, helping them gather necessary information on available resources to identify upcoming or existing bottlenecks. Collectd can be used with other tools, including Splunk. Furthermore, the configuration of collectd is quite easy to achieve.
