DEV Community

Cover image for Building a Custom NGINX Module with Ansible: A Dev-Friendly Guide
Athreya aka Maneshwar
Athreya aka Maneshwar

Posted on

Building a Custom NGINX Module with Ansible: A Dev-Friendly Guide

#webdev #programming #beginners #devops

Hi there! I'm Maneshwar. Right now, I’m building LiveAPI, a first-of-its-kind tool that helps you automatically index API endpoints across all your repositories. LiveAPI makes it easier to discover, understand, and interact with APIs in large infrastructures.

NGINX is a powerful and versatile web server, and its extensibility through modules is one of its greatest strengths.

Sometimes, however, you need functionality that isn't available out-of-the-box or as a pre-compiled dynamic module.

This is where custom NGINX modules come in.

In this blog post, we'll walk through a real-world scenario: building NGINX with the ngx_http_consul_backend_module and the ngx_devel_kit (NDK) module.

We'll leverage Ansible to automate the entire process, making it repeatable, reliable, and dev-friendly.

Why Ansible for NGINX Builds?

Manually compiling NGINX with custom modules can be a tedious and error-prone process.

Dependencies, compilation flags, and directory structures all need to be precisely managed.

Ansible simplifies this significantly by:

  • Idempotency: Tasks can be run multiple times without causing unintended side effects.
  • Automation: Automate repetitive tasks, saving time and reducing human error.
  • Version Control: Store your build process in version control, enabling easy collaboration and rollbacks.
  • Consistency: Ensure consistent builds across different environments.

Our Goal: NGINX with Consul Backend Module

We'll be building NGINX 1.23.2 and integrating two specific modules:

  1. ngx_devel_kit (NDK): A collection of utilities and APIs that simplify the development of NGINX modules. Many custom modules rely on NDK.
  2. ngx_http_consul_backend_module: A module that allows NGINX to discover backend services from HashiCorp Consul. This is particularly useful in dynamic, microservices-based environments.

Project Structure

Our Ansible project is organized as a role named nginx-with-consul-module.

This structure promotes reusability and maintainability.

ansible ├─ README.md ├─ ansible.cfg ├─ hosts.ini ├─ install_ansible.sh ├─ nginx-build-playbook.yml └─ roles └─ nginx-with-consul-module ├─ tasks │ ├─ build_consul_backend_module.yml │ ├─ build_nginx.yml │ ├─ configure_build.yml │ ├─ download_sources.yml │ ├─ install_dependencies.yml │ ├─ main.yml │ ├─ purge_deps.yml │ └─ systemd.yml └─ templates └─ nginx.service.j2
Enter fullscreen mode Exit fullscreen mode

The main.yml in the tasks directory orchestrates the entire build process by importing other task files:

--- - import_tasks: purge_deps.yml - import_tasks: install_dependencies.yml - import_tasks: download_sources.yml - import_tasks: build_consul_backend_module.yml - import_tasks: configure_build.yml - import_tasks: build_nginx.yml - import_tasks: systemd.yml
Enter fullscreen mode Exit fullscreen mode

Let's break down each step.

Step 1: Install Dependencies (install_dependencies.yml)

Building NGINX and its modules requires several development tools and libraries.

This task ensures all necessary packages are present on the target system.

--- - name: Install required packages apt: name: - build-essential # For gcc, g++ and make - libpcre3 - libpcre3-dev # For PCRE regular expressions support - zlib1g - zlib1g-dev # For gzip compression support - libssl-dev # For OpenSSL (HTTPS) support - git # To clone the Consul backend module - wget # To download source archives - curl # General utility state: present update_cache: true
Enter fullscreen mode Exit fullscreen mode

Step 2: Download Sources (download_sources.yml)

Before we can compile anything, we need the source code for NGINX, NDK, and the Consul backend module.

We'll download these to a temporary directory.

--- - name: Create build directory file: path: /tmp state: directory - name: Download nginx 1.23.2 source get_url: url: https://nginx.org/download/nginx-1.23.2.tar.gz dest: /tmp/nginx.tgz - name: Extract nginx source unarchive: src: /tmp/nginx.tgz dest: /tmp/ remote_src: yes - name: Download NDK module get_url: url: https://github.com/simpl/ngx_devel_kit/archive/v0.3.0.tar.gz dest: /tmp/ngx_devel_kit-0.3.0.tgz - name: Extract NDK module unarchive: src: /tmp/ngx_devel_kit-0.3.0.tgz dest: /tmp/ remote_src: yes - name: Clone Consul backend module git: repo: https://github.com/hashicorp/ngx_http_consul_backend_module.git dest: /go/src/github.com/hashicorp/ngx_http_consul_backend_module
Enter fullscreen mode Exit fullscreen mode

A crucial step here for the Consul backend module is a small patch.

The original code's strlen call with backend might cause a warning or error with newer compilers due to type mismatch.

We're explicitly casting backend to (const char*) to resolve this.

- name: Replace backend string length calculation replace: path: /go/src/github.com/hashicorp/ngx_http_consul_backend_module/src/ngx_http_consul_backend_module.c regexp: 'ngx_str_t ngx_backend = { strlen\(backend\), backend };' replace: "ngx_str_t ngx_backend = { strlen((const char*)backend), backend };"
Enter fullscreen mode Exit fullscreen mode

Step 3: Build Consul Backend Module (build_consul_backend_module.yml)

The ngx_http_consul_backend_module is written in Go and needs to be compiled as a C shared library. This involves several steps:

--- - name: Ensure nginx ext directory exists file: path: /usr/local/nginx/ext/ state: directory owner: "{{ ansible_user | default('root') }}" group: "{{ ansible_user | default('root') }}" mode: "0755" - name: Change ownership of Consul backend module directory file: path: /go/src/github.com/hashicorp/ngx_http_consul_backend_module state: directory owner: "{{ ansible_user | default('root') }}" group: "{{ ansible_user | default('root') }}" recurse: yes - name: Check Go version shell: export PATH=/usr/local/go/bin:$PATH && /usr/local/go/bin/go version register: go_version_result changed_when: false failed_when: false - name: Initialize Go modules command: /usr/local/go/bin/go mod init github.com/hashicorp/ngx_http_consul_backend_module args: chdir: /go/src/github.com/hashicorp/ngx_http_consul_backend_module register: go_mod_init_result changed_when: go_mod_init_result.rc == 0 failed_when: go_mod_init_result.rc != 0 and "go.mod already exists" not in go_mod_init_result.stderr ignore_errors: true # Ignore if go.mod already exists - name: Tidy Go modules command: /usr/local/go/bin/go mod tidy args: chdir: /go/src/github.com/hashicorp/ngx_http_consul_backend_module - name: Print Go version debug: msg: "{{ go_version_result.stdout }}" - name: Build Go shared library for Consul backend module shell: | export PATH=/usr/local/go/bin:$PATH CGO_CFLAGS="-I /tmp/ngx_devel_kit-0.3.0/src" \ /usr/local/go/bin/go build -buildmode=c-shared -o /usr/local/nginx/ext/ngx_http_consul_backend_module.so ./src/ngx_http_consul_backend_module.go args: chdir: /go/src/github.com/hashicorp/ngx_http_consul_backend_module
Enter fullscreen mode Exit fullscreen mode

Key points:

  • We create a dedicated directory /usr/local/nginx/ext/ to store our compiled dynamic module.
  • We ensure correct ownership for the cloned Consul module directory.
  • We initialize and tidy Go modules to manage dependencies.
  • The go build -buildmode=c-shared command is crucial. It compiles the Go module into a C shared library (.so file) that NGINX can load dynamically.
  • CGO_CFLAGS="-I /tmp/ngx_devel_kit-0.3.0/src" is important because the Consul module depends on headers from the NDK, so we need to tell the Go compiler where to find them.

Step 4: Configure NGINX Build (configure_build.yml)

This step involves running the ./configure script for NGINX. This script generates the Makefile based on the desired modules and features.

# Simplified for brevity, but this is where you'd run ./configure # with all your desired flags and --add-module directives. # The original configure command from the user's prompt (commented out) would be used here. # Example of how it would look if it were a separate task: - name: Configure NGINX build command: > ./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module --with-mail_ssl_module --with-stream_ssl_module --with-debug --add-module=/tmp/ngx_devel_kit-0.3.0 --add-module=/go/src/github.com/hashicorp/ngx_http_consul_backend_module args: chdir: /tmp/nginx-1.23.2
Enter fullscreen mode Exit fullscreen mode

Important configure flags:

  • --prefix=/etc/nginx: Sets the installation prefix for NGINX.
  • --add-module=/path/to/module: This is critical for including our custom modules. We point to the extracted NDK and the cloned Consul backend module source directories.
  • Other --with-* flags enable various built-in NGINX modules like SSL, HTTP/2, etc.

Step 5: Build and Install NGINX (build_nginx.yml)

Once configure has generated the Makefile, we can proceed with compilation and installation.

--- - name: Compile nginx command: make args: chdir: /tmp/nginx-1.23.2 - name: Install nginx command: make install args: chdir: /tmp/nginx-1.23.2 - name: Install apache2-utils # Useful for htpasswd, etc. apt: name: apache2-utils state: present
Enter fullscreen mode Exit fullscreen mode
  • make: Compiles the NGINX source code along with the added modules.
  • make install: Installs NGINX and its components to the paths specified during the configure step.

Step 6: Systemd Integration (systemd.yml)

For a production-ready setup, we need NGINX to run as a system service.

This task typically involves creating a systemd service file.

# Example content for systemd.yml, assuming you have nginx.service.j2 --- - name: Copy nginx systemd service file template: src: nginx.service.j2 dest: /etc/systemd/system/nginx.service owner: root group: root mode: '0644' notify: Reload systemd - name: Enable nginx service systemd: name: nginx enabled: true daemon_reload: true # Ensures systemd picks up the new service file state: started # Handler for 'Reload systemd' # handlers/main.yml # --- # - name: Reload systemd # systemd: # daemon_reload: true
Enter fullscreen mode Exit fullscreen mode

The nginx.service.j2 template would define how NGINX starts, stops, and reloads as a systemd service.

Step 7: Cleanup (Optional but Recommended - purge_deps.yml)

After a successful build, you might want to remove temporary build files and even some build dependencies to free up space, especially in a containerized environment.

# Example content for purge_deps.yml --- - name: Clean up temporary build files file: path: "{{ item }}" state: absent loop: - /tmp/nginx.tgz - /tmp/nginx-1.23.2 - /tmp/ngx_devel_kit-0.3.0.tgz - /tmp/ngx_devel_kit-0.3.0 - /go/src/github.com/hashicorp/ngx_http_consul_backend_module
Enter fullscreen mode Exit fullscreen mode

Running the Ansible Playbook

To execute this entire process, you would have a playbook like nginx-build-playbook.yml:

--- - name: Build NGINX with Consul module hosts: your_nginx_servers become: yes # Run tasks with sudo/root privileges roles: - nginx-with-consul-module
Enter fullscreen mode Exit fullscreen mode

And run it with:

ansible-playbook nginx-build-playbook.yml -i hosts.ini
Enter fullscreen mode Exit fullscreen mode

Conclusion

Building NGINX with custom modules can seem daunting, but by breaking down the process into logical Ansible tasks, we can create a robust, automated, and easily repeatable solution.

This approach is invaluable for consistent deployments across development, testing, and production environments.

You now have a solid foundation for integrating any custom NGINX module into your infrastructure using the power of Ansible!

LiveAPI helps you get all your backend APIs documented in a few minutes.

With LiveAPI, you can generate interactive API docs that allow users to search and execute endpoints directly from the browser.

LiveAPI Demo

If you're tired of updating Swagger manually or syncing Postman collections, give it a shot.

Top comments (4)

Collapse
 
dotallio profile image
Dotallio •

Love how you broke the process into clear Ansible roles, really makes custom NGINX builds less of a headache. How do you usually handle upgrades for these modules later on - have you automated that part as well?

Collapse
 
lovestaco profile image
Athreya aka Maneshwar •

Thanks

Collapse
 
parag_nandy_roy profile image
Parag Nandy Roy •

This is gold for DevOps beginners and pros alike ...

Collapse
 
lovestaco profile image
Athreya aka Maneshwar •

Haha so true