How to build NGINX from the source with modules

Athreya aka Maneshwar - Sep 13 - - Dev Community

I recently encountered an issue with service discovery while using Consul and NGINX.

Specifically, I was trying to route traffic to services deployed via Nomad using Consul DNS (service-name.service.consul), but my setup wasn't working as expected.

What was happening? When I tried to DIG into the SRV records, the IP and port were coming.

ubuntu@master:~/nomad$ dig @127.0.0.1 -p 8600 echo-server.com.service.consul SRV

; <<>> DiG 9.18.28-0ubuntu0.20.04.1-Ubuntu <<>> @127.0.0.1 -p 8600 echo-server.com.service.consul SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3407
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 7

;; ANSWER SECTION:
echo-server.com.service.consul. 0 IN SRV    1 1 25268 ac1f1647.addr.dc1.consul.
echo-server.com.service.consul. 0 IN SRV    1 1 24717 ac1f1647.addr.dc1.consul.

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1) (UDP)
;; WHEN: Mon Sep 09 18:53:03 UTC 2024
;; MSG SIZE  rcvd: 352
Enter fullscreen mode Exit fullscreen mode

When I tried redirecting the traffic from nginx to DNS it was not reaching the server. I just got a welcome nginx page.

But when i tried to curl the service that aswell failed. Only when the port was specified it worked.

ubuntu@master:~/nomad$ curl echo-server-com.service.consul:25268
Accessing port 25268, version 65
Enter fullscreen mode Exit fullscreen mode

After some investigation, I realized that service discovery in NGINX Open Source doesn't natively support DNS-based service discovery without additional modules—this functionality is part of NGINX Plus, a paid version.

To enable the same functionality without upgrading to NGINX Plus, I found the nginx-upstream-dynamic-servers module, which allows dynamic upstream management using DNS with Consul. This module, however, is a static module, meaning I needed to rebuild NGINX from source to install it.

This module allows services deployed with Nomad (using dynamic port allocation) to be accessed via Consul DNS (e.g., echo-server.com.consul). Since this module is a static one, it required me to rebuild NGINX from scratch.

Here's how I added the static module to NGINX to achieve DNS-based service discovery, overcoming the limitation of not having NGINX Plus.

Here’s a guide on how I added a static module to NGINX.

NGINX Module Types

  • Dynamic Modules: Can be easily added to an existing NGINX installation.
  • Static Modules: Require rebuilding NGINX from source.

Since the upstream module is static, I had to rebuild NGINX. I’m currently using NGINX 1.19 because this version is tested with the upstream module.

Step-by-Step Guide

1. Check Existing NGINX Build Files

Before rebuilding, check if you already have NGINX and module files in place:

cd /usr/local/src/nginx-1.19.10
ls
Enter fullscreen mode Exit fullscreen mode

If the necessary files are already present, the process becomes easier. If not, proceed to clone the required source files.

2. Clone the NGINX and Module Repositories

If NGINX or the upstream module isn’t present, download them:

cd /usr/local/src

sudo wget http://nginx.org/download/nginx-1.19.10.tar.gz

sudo tar -zxvf nginx-1.19.10.tar.gz

cd nginx-1.19.10

sudo git clone https://github.com/GUI/nginx-upstream-dynamic-servers.git
Enter fullscreen mode Exit fullscreen mode

Now clone your new module in the same folder so that all the modules are at one place in case the next person wants to add a new module.

After adding adding your module we need to configure the module.

Note: Don't forget to add the existing installed modules else it will break the running infra.

3. Configure NGINX with the Static Module

To add a module during the NGINX build process, use the following configuration command:

sudo ./configure \
    --prefix=/etc/nginx \
    --sbin-path=/usr/sbin/nginx \
    --conf-path=/etc/nginx/nginx.conf \
    --pid-path=/var/run/nginx.pid \
    --lock-path=/var/run/nginx.lock \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-http_realip_module \
    --with-http_auth_request_module \
    --with-http_v2_module \
    --with-http_dav_module \
    --with-http_slice_module \
    --with-http_addition_module \
    --with-http_gunzip_module \
    --with-http_gzip_static_module \
    --with-http_sub_module \
    --with-mail_ssl_module \
    --with-stream_ssl_module \
    --add-module=/usr/local/src/nginx-1.19.10/nginx-upstream-dynamic-servers
Enter fullscreen mode Exit fullscreen mode

Common Configuration Options:

  • --prefix=/etc/nginx: Installation directory for Nginx binaries and configuration files.
  • --sbin-path=/usr/sbin/nginx: Path to the Nginx binary executable.
  • --conf-path=/etc/nginx/nginx.conf: Path to the main Nginx configuration file.
  • --pid-path=/var/run/nginx.pid: Path to the Nginx process ID file.
  • --lock-path=/var/run/nginx.lock: Path to the Nginx lock file.
  • --error-log-path=/var/log/nginx/error.log: Path to the Nginx error log file.
  • --http-log-path=/var/log/nginx/access.log: Path to the Nginx access log file.
  • --with-http_ssl_module: Enable SSL support.
  • --with-http_stub_status_module: Enable the stub_status module for server status monitoring.
  • (Add other desired modules with --with-modulename_module)
  • Make sure to include the --add-module option for each static module you want to build with NGINX.

Once the configuration is complete, you can proceed to remove the existing NGINX installation and rebuild NGINX with the new module.

4. Backup Your Current NGINX Configuration

Before purging the current NGINX installation, you must back up the existing configuration:

sudo cp -r /etc/nginx ~/nginx-backup-1
Enter fullscreen mode Exit fullscreen mode

5. Remove Existing NGINX Installation

If NGINX is already installed, remove it completely:

# Stop nginx service if running
sudo systemctl stop nginx

# Remove NGINX installed from packages (if any)
sudo apt-get remove --purge nginx nginx-common nginx-full

# Remove NGINX binaries from /usr/local if built from source
# Remove any remaining nginx configuration or directories
sudo rm -rf /usr/local/nginx /etc/nginx /var/log/nginx /var/run/nginx.pid
Enter fullscreen mode Exit fullscreen mode

6. Install Required Dependencies

To compile NGINX from source, install the necessary dependencies:

sudo apt-get update
sudo apt-get install build-essential libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev
Enter fullscreen mode Exit fullscreen mode

7. Build and Install NGINX

Once dependencies are installed, build and install NGINX:

sudo make
sudo make install
Enter fullscreen mode Exit fullscreen mode

8. Verify the Installed Modules

Check if the modules have been installed correctly:

/usr/sbin/nginx -V
Enter fullscreen mode Exit fullscreen mode

This should display the version of NGINX along with all configured modules.

ubuntu@master:~$ s nginx -V
nginx version: nginx/1.19.10
built by gcc 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
built with OpenSSL 1.1.1f 31 Mar 2020
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module --with-mail_ssl_module --with-stream_ssl_module --add-module=/usr/local/src/nginx-1.19.10/nginx-upstream-dynamic-servers
Enter fullscreen mode Exit fullscreen mode

9. Restore NGINX Configuration

Copy your backed-up configuration to the new installation:

# sudo cp -r ~/nginx-backup/* /etc/nginx/

# Create new directory for sites-enabled
sudo mkdir /etc/nginx/sites-enabled

# Below command will create a symbolic link for each site in the sites-available directory to the sites-enabled directory.
for site in /etc/nginx/sites-available/*; do
    sudo ln -s "$site" /etc/nginx/sites-enabled/
done
Enter fullscreen mode Exit fullscreen mode

Nginx conf will be available at:

sudo cat /etc/nginx/nginx.conf
Enter fullscreen mode Exit fullscreen mode

Logs will be available at:

# Error log
sudo tail -f /var/log/nginx/error.log

# Access log
sudo tail -f /var/log/nginx/access.log
Enter fullscreen mode Exit fullscreen mode

10. Test and Reload NGINX

After restoring the configuration, test and reload NGINX:

# Test configuration
sudo nginx -t

# Reload NGINX
sudo systemctl reload nginx
Enter fullscreen mode Exit fullscreen mode

11. Set NGINX as a System Service (Optional)

If NGINX is not running as a service, create a systemd service file:

sudo vim /lib/systemd/system/nginx.service
Enter fullscreen mode Exit fullscreen mode

Add the following content:

[Unit]
Description=The NGINX HTTP and reverse proxy server
After=network.target

[Service]
Type=forking
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/usr/sbin/nginx -s stop
PIDFile=/var/run/nginx.pid
PrivateTmp=true

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Enable and start the NGINX service:

sudo systemctl enable nginx
sudo systemctl start nginx
Enter fullscreen mode Exit fullscreen mode

Conclusion

That’s how you add a static module to NGINX!

It’s important to back up your configurations, purge the old installation, and build from source when using static modules like the upstream dynamic servers module.

Once NGINX is built, you can verify the installation and restore your configuration seamlessly.

Happy NGINXing!

Image
First Post

osd600
Image
TinTinLand Web3 + DePIN Learning Month Phala Network

webdev, beginners, tutorial
Image
Best React Frameworks: Which One Should You Choose and When?

webdev, nextjs, javascript, gatsby
Image
Setting Up a Monitoring Stack with Nginx Logging Using Grafana, cAdvisor, Promtail, Prometheus, and Loki
Image
LeetCode Review

dsa, leetcode, competativeprogramming
Image
Deploying AI models on edge devices
Image
App::cpx

perl, cpan, npx
Image
How to Create Card Slider in HTML CSS & JavaScript | Step-by-Step Guide

html, css, javascript, webdev
Image
MySQL Mayhem: How to Reset Your Root Password for MySQL in No Time!
Image
Sick of WordPress?
Image
Scale Nodejs Application with APM Tools

nestjs, node, microservices, javascript
Image
5 Open-source alternatives to Jira, Linear and Asana

javascript, productivity, opensource, ai
Image
Writing your first web component (Learn Modulo.js - Part 1 of 10)

beginners, javascript, tutorial, webdev
Image
Improving MongoDB Operations in a Go Microservice: Best Practices for Optimal Performance

mongodb, go, performance, database
Image
💡Solving LeetCode 34: Find First and Last Position of Element in a Sorted Array

algorithms, java, programming, learning
Image
Webkul pattern question advance pattern for interview with python
Image
2-qism Flexbox: flex-direction, flex-wrap, flex-flow va justify-content
Image
Chapter 5 - Cloning a Remote Repository

github, git, opensource
Image
Unleash Your Inner Hacker: Dive into Linux for Ethical Hacking and Penetration Testing 🐧⚔️

getvm, videocourses, programming, tutorial
. . . . . . . . . . . . .
Terabox Video Player