DEV Community

Cover image for Setting Up Datadog Agent for Nginx Log Collection on AWS EC2
Manoj Swami
Manoj Swami

Posted on

Setting Up Datadog Agent for Nginx Log Collection on AWS EC2

#datadog #ec2 #nginx #logging

Introduction

In the world of modern web applications, monitoring and log management are crucial for maintaining system health and troubleshooting issues. This guide will walk you through the process of setting up the Datadog agent to collect Nginx logs from an AWS EC2 instance. We'll cover everything from initial installation to troubleshooting common issues.

Objectives

By the end of this guide, you will be able to:

  1. Install and configure the Datadog agent on an AWS EC2 instance
  2. Set up Nginx log collection
  3. Troubleshoot common issues in the setup process
  4. Verify successful log transmission to Datadog

Prerequisites

  • An AWS EC2 instance running Ubuntu
  • Nginx installed and running on the EC2 instance
  • A Datadog account (sign up at https://www.datadoghq.com/)

Detailed Steps

1. Installing the Datadog Agent

First, we need to install the Datadog agent on our EC2 instance. We'll use the official installation script:

DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.eu" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"
Enter fullscreen mode Exit fullscreen mode

Replace <YOUR_API_KEY> with your actual Datadog API key. You can find this in your Datadog account settings.

Note: We're using datadoghq.eu here. If your Datadog organization is based in the US, use datadoghq.com instead.

2. Configuring the Datadog Agent

After installation, we need to configure the agent. The main configuration file is located at /etc/datadog-agent/datadog.yaml. Let's edit it:

sudo nano /etc/datadog-agent/datadog.yaml
Enter fullscreen mode Exit fullscreen mode

In this file, we need to set up a few key parameters:

api_key: <YOUR_ACTUAL_API_KEY> site: datadoghq.eu cmd_port: 5002 expvar_port: 5002 logs_enabled: true
Enter fullscreen mode Exit fullscreen mode

The cmd_port and expvar_port settings are important. We initially tried ports 5000 and 2001, but they were in use by other applications. Always check for available ports using:

sudo netstat -tulpn | grep LISTEN
Enter fullscreen mode Exit fullscreen mode

3. Configuring Nginx Log Collection

Now, we need to tell Datadog which Nginx logs to collect. Create a new configuration file:

sudo nano /etc/datadog-agent/conf.d/nginx.d/conf.yaml
Enter fullscreen mode Exit fullscreen mode

Add the following configuration:

logs: - type: file path: /var/log/nginx/access.log service: nginx source: nginx - type: file path: /var/log/nginx/error.log service: nginx source: nginx
Enter fullscreen mode Exit fullscreen mode

This configuration tells Datadog to collect both access and error logs from Nginx.

datadog.yaml file code

sudo nano /etc/datadog-agent/datadog.yaml
Enter fullscreen mode Exit fullscreen mode 
api_key: YOUR_KEY DD_API_KEY: 'YOUR_KEY' DD_REMOTE_CONFIGURATION_ENABLED: false remote_configuration_enabled: false expvar_port: 5004 cmd_port: 5003 log_level: debug log_to_console: true forwarder_storage_max_size_in_bytes: 104857600 logs_enabled: true logs_config: container_collect_all: true site: datadoghq.eu # .eu if your datadog is using Europe server else use .com
Enter fullscreen mode Exit fullscreen mode

4. Setting Correct Permissions

One common issue is that the Datadog agent doesn't have permission to read the Nginx log files. Let's fix that:

sudo usermod -a -G adm dd-agent sudo chmod 644 /var/log/nginx/*.log sudo chmod 755 /var/log/nginx
Enter fullscreen mode Exit fullscreen mode

These commands add the Datadog agent user to the adm group (which typically has access to log files) and ensure the log files and directory have the correct permissions.

5. Restarting and Verifying

After making these changes, restart the Datadog agent:

sudo systemctl restart datadog-agent
Enter fullscreen mode Exit fullscreen mode

Now, let's verify our configuration:

sudo datadog-agent status sudo datadog-agent configcheck
Enter fullscreen mode Exit fullscreen mode

These commands will show the status of the agent and check for any configuration errors.

6. Monitoring Logs

To see what's happening in real-time, you can monitor the Datadog agent's logs:

sudo tail -f /var/log/datadog/agent.log
Enter fullscreen mode Exit fullscreen mode

Look for any error messages or warnings here.

Troubleshooting Common Issues

API Key Issues

If you see a message like this:

WARN | api_key '***************************380ff' for domain https://api.datadoghq.com is invalid
Enter fullscreen mode Exit fullscreen mode

Double-check your API key in the Datadog web interface and make sure it's correctly copied in your datadog.yaml file.

Permission Denied Errors

If you see:

WARN | open /var/log/nginx/error.log: permission denied
Enter fullscreen mode Exit fullscreen mode

Revisit step 4 and ensure you've set the correct permissions.

Port Already in Use

If you encounter:

ERROR | Error while starting api server, exiting: unable to start CMD API server: unable to listen to the given address: listen tcp 127.0.0.1:5002: bind: address already in use
Enter fullscreen mode Exit fullscreen mode

Choose a different port in your datadog.yaml file and restart the agent.

Verifying Success

Generate some traffic to your Nginx server, then check the Datadog web interface. Go to Logs > Search, and you should see your Nginx logs appearing.

Conclusion

Setting up the Datadog agent to collect Nginx logs from an AWS EC2 instance involves several steps and potential pitfalls. By following this guide, you should be able to successfully configure log collection and troubleshoot common issues. Remember, proper monitoring and log management are key to maintaining a healthy, performant web application.

Top comments (0)