Building a Scalable Minio Distributed Setup: A Step-by-Step Guide

farhanaliali - Sep 4 - - Dev Community

Minio distributed setup

Author: Farhan ali

Introduction

In this guide, I'll walk you through setting up a distributed Minio cluster across five servers. Four of these servers will be equipped with additional storage, providing a robust and scalable solution for your object storage needs. By the end of this tutorial, you'll have a fully configured Minio setup, ready to handle distributed storage tasks efficiently.

You'll need to set up your 5 base servers. Four of them will require an additional 4 disks each
I'm assuming all servers are listed in DNS as follows:

minio1.domain.com
minio2.domain.com
minio3.domain.com
minio4.domain.com
minio.domain.com (This is the load balance)
Enter fullscreen mode Exit fullscreen mode

All 4 servers should have /dev/sdb,sdc,sdd,sde as the 4 empty disks.
Obviously, you can change as you see fit but will have to adapt this guide.

Step 1: Initial Set Up

On each Minio server set up the mount locations on each server:

sudo mkdir /mnt/minio1
sudo mkdir /mnt/minio2
sudo mkdir /mnt/minio3
sudo mkdir /mnt/minio4
Enter fullscreen mode Exit fullscreen mode

Create a partition on each disk:

sudo fdisk /dev/sdb
sudo fdisk /dev/sdc
sudo fdisk /dev/sdd
sudo fdisk /dev/sde
Enter fullscreen mode Exit fullscreen mode

It will prompt for the following sequence. You can accept the defaults mostly. It's n for the new partition, enter for the default partition number, enter for the first sector, enter for the last sector, and w to write the partition to disk.

Repeat that for all disks.

Add a file system. Specifically xfs!

mkfs.xfs /dev/sdb1
mkfs.xfs /dev/sdc1
mkfs.xfs /dev/sdd1
mkfs.xfs /dev/sde1
Enter fullscreen mode Exit fullscreen mode

Repeat that for all disks.

Now mount the disks on all servers accordingly 

sudo vi /etc/fstab

/dev/sdb    /mnt/minio1 xfs defaults 0 1 
/dev/sdc    /mnt/minio2 xfs defaults 0 1
/dev/sdd    /mnt/minio3 xfs defaults 0 1
/dev/sde    /mnt/minio4 xfs defaults 0 1
Enter fullscreen mode Exit fullscreen mode

save and mount the disks

sudo mount -a
Enter fullscreen mode Exit fullscreen mode

Install minio

Use the following commands to download the latest stable MinIO DEB and install it:

wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20230904195737.0.0_amd64.deb -O minio.deb
sudo dpkg -i minio.deb
Enter fullscreen mode Exit fullscreen mode

We need to create a default config file:

sudo vi /etc/default/minio
Enter fullscreen mode Exit fullscreen mode

This is the file contents for me:

#Set the hosts and volumes MinIO uses at startup
# The command uses MinIO expansion notation {x...y} to denote a
# sequential series.
#
# The following example covers four MinIO hosts
# with 4 drives each at the specified hostname and drive locations.
# The command includes the port that each MinIO server listens on
# (default 9000)

MINIO_VOLUMES="https://minio{1...4}.domain.com:9000/mnt/minio{1...4}"

# Set all MinIO server options
#
# The following explicitly sets the MinIO Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.

MINIO_OPTS="--console-address :9001"

# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.

MINIO_ROOT_USER=minioadmin

# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.

MINIO_ROOT_PASSWORD=SuperSecretPassword

# Set to the URL of the load balancer for the MinIO deployment
# This value *must* match across all MinIO servers. If you do
# not have a load balancer, set this value to to any *one* of the
# MinIO hosts in the deployment as a temporary measure.
MINIO_SERVER_URL="https://minio.domain.com:9000"
Enter fullscreen mode Exit fullscreen mode

The key things to ensure are the volume names are correct, your admin username and password are as desired and your load balancer URL is set.

We need a mini user too and to give permissions to that user:

sudo groupadd -r minio-user
sudo useradd -M -r -g minio-user minio-user
sudo chown minio-user:minio-user /mnt/minio1/ /mnt/minio2/ /mnt/minio3/ /mnt/minio4/
Enter fullscreen mode Exit fullscreen mode

SSL setup

We want to use SSL for all of the communication so let's use LetsEncrypt for free certs! Again edit the domain to reflect the node you are doing this on.

Now on mini1

sudo apt install certbot
sudo certbot certonly --standalone -d minio1.domain.com
sudo mkdir -p /home/minio-user/.minio/certs
cp /etc/letsencrypt/live/minio1.domain.com/fullchain.pem /home/minio-user/.minio/certs/public.crt
cp /etc/letsencrypt/live/minio1.domain.com/privkey.pem /home/minio-user/.minio/certs/private.key
chown -R minio-user:minio-user /home/minio-user/
Enter fullscreen mode Exit fullscreen mode

We need Minio to bind to ports below 1024:

setcap 'cap_net_bind_service=+ep' /usr/local/bin/minio
Enter fullscreen mode Exit fullscreen mode

And we are finally ready to start Minio!

sudo systemctl restart minio
Enter fullscreen mode Exit fullscreen mode

Make sure it’s all healthy.

Now repeart the same configuration remining servers

On loadbalancer server

Install nginx and certbot.

sudo apt install nginx certbot
Enter fullscreen mode Exit fullscreen mode

Create a config for Minio:

sudo vi /etc/nginx/sites-available/minio
Enter fullscreen mode Exit fullscreen mode

This is the file contents for me:

upstream minio_servers {
        server minio1.domain.com:9000;
        server minio2.domain.com:9000;
        server minio3.domain.com:9000;
        server minio4.domain.com:9000;
        }
upstream minio_console {
        ip_hash;
        server minio1.domain.com:9001;
        server minio2.domain.com:9001;
        server minio3.domain.com:9001;
        server minio4.domain.com:9001;
        }
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
        }
server {
        listen 443 ssl http2 default_server;
        listen [::]:443 ssl http2 default_server;
        server_name minio.domain.com;
        client_max_body_size 0;
        ssl_certificate "/etc/letsencrypt/live/minio.domain.com/fullchain.pem";
        ssl_certificate_key "/etc/letsencrypt/live/minio.domain.com/privkey.pem";

     location / {
          proxy_http_version 1.1;
          proxy_set_header Host $http_host;
          proxy_pass https://minio_console;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $connection_upgrade;
           }
        }

server {
          listen       9000 ssl;
          listen  [::]:9000;
          server_name minio.domain.com;
          ignore_invalid_headers off;
          client_max_body_size 0;
          proxy_buffering off;
          proxy_request_buffering off;
          ssl_certificate "/etc/letsencrypt/live/minio.domain.com/fullchain.pem";
          ssl_certificate_key "/etc/letsencrypt/live/minio.domain.com/privkey.pem";

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;
        proxy_pass https://minio_servers;
    }
}
Enter fullscreen mode Exit fullscreen mode

Create the SSL cert from LetsEncrypt again:

sudo systemctl stop nginx
sudo certbot certonly --standalone --preferred-challenges http -d minio.domain.com
Enter fullscreen mode Exit fullscreen mode

Verify the nginx config is all correct and no mistakes or missed characters!

Link the config to the sites-enabled:

sudo ln -s /etc/nginx/sites-available/minio /etc/nginx/sites-enabled/minio
Enter fullscreen mode Exit fullscreen mode

test the config

sudo nginx -t
Enter fullscreen mode Exit fullscreen mode

Lastly, start nginx.

sudo service nginx start
Enter fullscreen mode Exit fullscreen mode

Assuming all of that went fine you now have a working Minio cluster. 4 nodes, 16 disks in total and all load balanced.

access the webui 

https://minio.domain.com
Enter fullscreen mode Exit fullscreen mode
Image
🛠️ Guide: How to Develop Web3 dApps on Sei Network Using NFTScan NFT API
Image
Never apply for a job again - help us help others.

beginners, ai, devops, javascript
Image
Freelance Digital Marketing Strategist From Calicut
Image
Exodus Effect: Is This Book Truly Compelling or A Trick Value, Advantages, Protests, and Survey!

exoduseffect
Image
Playwright - Remember me functionality

playwright, typescript, automation
Image
Error Handling

javascript
Image
Stay On-Trend The Latest Hoodie Styles You Need to Know

webdev, tutorial, ai, productivity
Image
Unlocking the Power of Character AI: Step-by-Step Guide for Creators
Image
A quick overview of Bun's basic features and how it compares to Node.js

node, bunjs, express, ditsmod
Image
Embarking on My Android Development Journey with Kotlin: Seeking Your Expert Tips and Tricks!

android, mobile, programming, kotlin
Image
Shadowing in JavaScript

javascript, programming, webdev, javascriptlibraries
Image
Boletín AWS Open Source, AI Edition

opensource, aws, genai, ai
Image
Hydrogen Market Report: Key Trends, Growth Drivers, and Strategic Opportunities
Image
Why Vulnerability Assessments are Critical for Enterprise Security?

devops, technology, software, trending
Image
Automating EC2 Instances with AWS Lambda and CloudWatch

aws, automation, devops, serverless
Image
The Ferulic Acid Market Poised to Witness High Growth Driven by Increasing Application in Various Industries
Image
NFTScan Insights: Exploring Various NFT Standards and Their Use Cases

nft, web3
Image
How to setup Stripe plugin in Medusa.js on Nomodo: Step‑by‑Step Guide

medusajs, tutorial
Image
Taskwarrior: The Command-Line Task Manager for Power Users

taskwarrior, productivity, taskmanagement, opensource
. . .
Terabox Video Player