Ubuntu-Server
Ultimate Server Admin Guide

Ultimate Server Administration Series

Series Outline:

  1. Foundations of Server Administration: Initial Setup and Best Practices

    • Introduction to server administration
    • Installing and configuring Ubuntu and RedHat
    • Essential tools and configurations for stability and performance
    • Securing the initial setup: user management, SSH hardening, firewalls
    • Backup and recovery planning

  2. Advanced Security Techniques for Server Hardening

    • In-depth firewall configurations (UFW, iptables)
    • SSL/TLS setup for web services (Let's Encrypt, custom certificates)
    • Monitoring system access and logs (Syslog, auditd)
    • Security updates and patch management
    • Best practices for vulnerability scanning

  3. Network Configuration and Optimization

    • Understanding Linux networking: interfaces, routes, and DNS
    • Setting up and optimizing web servers (Nginx, Apache)
    • Configuring Load Balancers and Reverse Proxies
    • Traffic monitoring and network security tools
    • Advanced DNS configuration (BIND, Cloudflare)

  4. Managing Storage and File Systems

    • Disk partitioning, LVM, and RAID setup
    • Optimizing file systems for performance
    • Implementing quotas and backup strategies
    • Network storage (NFS, SMB) setup and management
    • Troubleshooting file system and storage issues

  5. Service Management and Process Monitoring

    • Using systemd to manage services
    • Monitoring processes and resource usage (top, htop, iotop)
    • Automating service restarts and notifications on failures
    • Understanding and managing kernel parameters
    • Analyzing and optimizing system performance

  6. Automation with Bash, Ansible, and Python

    • Introduction to automation in server administration
    • Writing effective shell scripts for automation
    • Using Ansible for configuration management
    • Advanced Ansible playbooks and roles
    • Integrating Python scripts for custom tasks

  7. Building Resilient Web Servers: High Availability and Load Balancing

    • Introduction to high availability concepts
    • Setting up load balancers (HAProxy, Nginx)
    • Configuring failover and redundancy
    • Ensuring data integrity across servers (database replication)
    • Testing resilience and recovery strategies

  8. Continuous Integration and Continuous Deployment (CI/CD)

    • Overview of CI/CD pipelines in server administration
    • Setting up Jenkins, GitLab CI, and GitHub Actions
    • Automating infrastructure provisioning with Terraform
    • Docker containerization and orchestration
    • Deployment strategies for zero downtime

  9. Integrating Machine Learning and OpenAI with Azure

    • Introduction to AI and ML in server automation
    • Setting up Azure OpenAI services for intelligent automation
    • Creating custom AI-driven server management solutions
    • Integrating AI with CI/CD pipelines
    • Exploring future trends in AI and server administration

  10. Troubleshooting, Maintenance, and Scaling Globally

    • Advanced troubleshooting techniques for networking and services
    • Setting up monitoring solutions (Prometheus, Grafana)
    • Alerting and incident response best practices
    • Scaling infrastructure across multiple regions
    • Future-proofing server architectures

Part 1: Foundations of Server Administration

Initial Setup and Best Practices

Introduction to Server Administration

Server administration is the backbone of any IT infrastructure. Whether you're managing a personal project or deploying enterprise-grade applications, understanding how to efficiently and securely manage servers is essential. In this part, we will explore the core concepts of server administration, focusing on setting up a fresh server with Ubuntu or RedHat, securing it, and ensuring optimal performance.

1.1 Choosing the Right Server Distribution: Ubuntu vs. RedHat

Choosing between Ubuntu (Debian-based) and RedHat (RHEL-based) depends on your project requirements and support preferences. Here’s a comparison to help you decide:

  • Ubuntu:
    • Free and widely used.
    • Suitable for beginners and experts.
    • Large community support.
    • Ideal for web servers, containers, and cloud deployments.
  • RedHat:
    • Enterprise-focused with paid support.
    • Strong in enterprise data centers, financial systems, and critical applications.
    • Provides stability and long-term support (CentOS Stream or AlmaLinux are free alternatives).

1.2 Installing Ubuntu 24.04 LTS

Step 1: Download and Boot from ISO

Download the latest Ubuntu Server ISO from Ubuntu's official site (opens in a new tab) and boot your server or virtual machine from the ISO.

Step 2: Installation Process

  1. Select your language and keyboard layout.
  2. Choose your network configuration (DHCP or static IP).
  3. Partition disks (automatic setup or manual for custom configurations).
  4. Install base packages.
  5. Configure system users (create an admin user and set passwords).
  6. Install OpenSSH if you'd like remote access right after setup.

Step 3: First Boot Once installation is complete, log in with the user credentials created during setup.

1.3 Initial Setup After Installation

1.3.1 Updating the System

It is crucial to update your system packages to ensure security patches and updates are applied.

For Ubuntu:

sudo apt update && sudo apt upgrade -y

For RedHat:

sudo dnf update -y

1.3.2 Creating a Non-Root User

Using the root account for regular administration is considered risky. It's best to operate with a non-root user with sudo privileges.

Create a user:

sudo adduser yourusername

Add the user to the sudoers group:

sudo usermod -aG sudo yourusername

For RedHat:

sudo usermod -aG wheel yourusername

1.3.3 Configuring SSH Access

SSH is the primary method for remote server management. A strong setup prevents unauthorized access.

  1. Install OpenSSH (if not installed during setup):

    sudo apt install openssh-server -y   # Ubuntu
sudo dnf install openssh-server -y   # RedHat

  2. Harden SSH:

    • Disable root login via SSH by editing /etc/ssh/sshd_config: 
      sudo nano /etc/ssh/sshd_config
      Change the following: 
      PermitRootLogin no
    • Set SSH to use key-based authentication for stronger security: Generate an SSH key pair on your local machine: 
      ssh-keygen -t rsa -b 4096
      Copy your public key to the server: 
      ssh-copy-id yourusername@server_ip

  3. Restart SSH to apply changes:

    sudo systemctl restart ssh

1.3.4 Setting Up a Firewall (UFW for Ubuntu, firewalld for RedHat)

A firewall adds an additional layer of protection by restricting open ports.

For Ubuntu (UFW):

  1. Install UFW: 
    sudo apt install ufw -y
  2. Allow SSH and other necessary services: 
    sudo ufw allow OpenSSH
sudo ufw allow 80/tcp   # HTTP
sudo ufw allow 443/tcp  # HTTPS
  3. Enable UFW: 
    sudo ufw enable

For RedHat (firewalld):

  1. Install firewalld: 
    sudo dnf install firewalld -y
  2. Start and enable firewalld: 
    sudo systemctl start firewalld
sudo systemctl enable firewalld
  3. Allow services through the firewall: 
    sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

1.4 Essential Tools and Configurations for Stability

1.4.1 Installing Essential Monitoring Tools

To maintain a healthy system, install monitoring tools for system performance:

  • htop (interactive process viewer): 
    sudo apt install htop -y   # Ubuntu
sudo dnf install htop -y   # RedHat
  • vnstat (network monitoring): 
    sudo apt install vnstat -y   # Ubuntu
sudo dnf install vnstat -y   # RedHat

1.4.2 Disk and File System Monitoring

Keep track of your server's disk usage:

df -h        # Show disk usage
du -sh *     # Show folder sizes

Set up automatic updates to ensure security patches are applied without manual intervention:

  • For Ubuntu:

    sudo apt install unattended-upgrades -y
sudo dpkg-reconfigure -plow unattended-upgrades

  • For RedHat, schedule updates using dnf-automatic:

    sudo dnf install dnf-automatic -y
sudo systemctl enable --now dnf-automatic.timer

1.5 Backup and Recovery Planning

Backup and recovery are crucial for maintaining data integrity and minimizing downtime in case of system failure.

  1. Setting up regular backups using rsync:

    sudo apt install rsync -y   # Ubuntu
sudo dnf install rsync -y   # RedHat

    Example backup command:

    rsync -av /important/data /backup/location

  2. Scheduling backups using cron:

    sudo crontab -e

    Add a daily backup job:

    0 2 * * * rsync -av /important/data /backup/location

  3. Testing backups regularly to ensure recovery readiness:

    rsync -av /backup/location /restore/test/location

Part 2: Advanced Security Techniques for Server Hardening

Strengthening Your Servers Against Threats

Securing your server is one of the most crucial tasks for any system administrator. In this section, we will go deep into advanced security configurations, focusing on protecting your server from both external and internal threats. You will learn how to manage firewalls, secure web traffic with SSL/TLS, monitor suspicious activity, and keep your system up to date against vulnerabilities.

2.1 Enhancing Firewall Configurations

Firewalls are the first line of defense in any server environment. Configuring your firewall correctly ensures that only necessary traffic reaches your services.

2.1.1 Configuring UFW (Ubuntu)

The Uncomplicated Firewall (UFW) is a simple, user-friendly tool for managing firewall rules on Ubuntu systems.

  1. Checking the status of UFW:

    sudo ufw status

  2. Denying all incoming connections by default:

    sudo ufw default deny incoming
sudo ufw default allow outgoing

  3. Allowing specific services:

    • SSH: 
      sudo ufw allow OpenSSH
    • HTTP/HTTPS: 
      sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
    • Custom ports (e.g., a database server): 
      sudo ufw allow 5432/tcp  # PostgreSQL

  4. Limiting SSH attempts to prevent brute force attacks:

    sudo ufw limit OpenSSH

  5. Enabling UFW:

    sudo ufw enable

2.1.2 Configuring firewalld (RedHat)

firewalld is the default firewall management tool on RedHat-based systems.

  1. Check the firewall status:

    sudo firewall-cmd --state

  2. Enable and start firewalld:

    sudo systemctl enable firewalld
sudo systemctl start firewalld

  3. Default deny incoming:

    sudo firewall-cmd --set-default-zone=block

  4. Open ports for services:

    • Open SSH and HTTP/HTTPS: 
      sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

  5. Limiting connections for brute force protection: You can create custom rules using rich rules to limit SSH attempts:

    sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='0.0.0.0/0' service name='ssh' limit value='5/m' accept"
sudo firewall-cmd --reload

  6. Verifying the rules:

    sudo firewall-cmd --list-all

2.2 Securing Web Traffic with SSL/TLS

Securing your web traffic with SSL/TLS is critical for protecting data in transit. We will use Let's Encrypt to provide free and easy SSL certificates for your web services.

2.2.1 Installing Certbot (Let's Encrypt)

Ubuntu:

  1. Add the Certbot repository and install Certbot: 
    sudo apt update
sudo apt install certbot python3-certbot-nginx -y

RedHat:

  1. Install Certbot and the Nginx plugin: 
    sudo dnf install certbot python3-certbot-nginx -y

2.2.2 Generating SSL Certificates

With Certbot, you can easily obtain SSL certificates for your domain. Here’s how:

  1. Run Certbot to automatically obtain and configure your certificate for Nginx:

    sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com

  2. Certbot will modify your Nginx configuration to include the SSL certificate. Verify it by checking the configuration:

    sudo nginx -t

  3. Reload Nginx to apply the changes:

    sudo systemctl reload nginx

2.2.3 Automating Certificate Renewal

Certbot sets up a cron job for automatic renewal by default. However, it's important to test that it works correctly:

  1. Test renewal: 
    sudo certbot renew --dry-run

2.3 Monitoring System Access and Logs

Keeping an eye on logs and access attempts is essential for identifying suspicious activity before it becomes a threat. Tools like Syslog and auditd help in this regard.

2.3.1 Configuring System Logging with rsyslog

rsyslog is the default logging system on most Linux distributions.

  1. Check log files in /var/log/:

    ls /var/log/

  2. Important logs to monitor:

    • /var/log/auth.log (Ubuntu) or /var/log/secure (RedHat) for authentication attempts.
    • /var/log/syslog for system-level logs.

  3. Filter logs by IP or keywords using grep:

    grep "Failed password" /var/log/auth.log  # For failed login attempts

2.3.2 Auditing System Events with auditd

auditd allows you to track critical system changes, which can be helpful for detecting unauthorized activity.

  1. Install auditd:

    • Ubuntu: 
      sudo apt install auditd -y
    • RedHat: 
      sudo dnf install audit -y

  2. Start and enable auditd:

    sudo systemctl start auditd
sudo systemctl enable auditd

  3. Monitoring file access: Track who is accessing or modifying critical system files by adding a rule to /etc/audit/rules.d/audit.rules:

    -w /etc/passwd -p wa -k passwd_changes

  4. View audit logs:

    sudo ausearch -k passwd_changes

2.4 Automating Security Updates

Keeping your system up-to-date with security patches is one of the simplest ways to prevent vulnerabilities.

2.4.1 Configuring Unattended Security Updates (Ubuntu)

Ubuntu offers a simple way to configure automatic security updates using unattended-upgrades.

  1. Install the package:

    sudo apt install unattended-upgrades -y

  2. Configure the file /etc/apt/apt.conf.d/50unattended-upgrades to only allow security updates by ensuring the following line is present:

    Unattended-Upgrade::Allowed-Origins {
    "Ubuntu focal-security";
};

  3. Enable automatic updates:

    sudo dpkg-reconfigure --priority=low unattended-upgrades

2.4.2 Configuring Automatic Updates (RedHat)

For RedHat, dnf-automatic can be used to apply updates regularly.

  1. Install the package:

    sudo dnf install dnf-automatic -y

  2. Edit the config file /etc/dnf/automatic.conf and set the apply_updates parameter to yes:

    [commands]
apply_updates = yes

  3. Enable the service to run regularly:

    sudo systemctl enable --now dnf-automatic.timer

2.5 Running Vulnerability Scans

To further harden your servers, use vulnerability scanners like Lynis to perform in-depth audits and detect weak configurations.

2.5.1 Installing and Running Lynis

Lynis is an open-source security auditing tool for Linux.

  1. Install Lynis on Ubuntu:

    sudo apt install lynis -y

    Install on RedHat:

    sudo dnf install lynis -y

  2. Run a security scan:

    sudo lynis audit system

  3. Review the security report and address any vulnerabilities: The report provides insights into weak configurations, missing patches, and other security recommendations.

Part 3: Network Configuration and Optimization

Building Efficient and Scalable Networks

Network performance and stability are critical to running efficient, responsive web servers and applications. In this part, we will explore configuring networking on Ubuntu and RedHat, optimizing performance, and setting up load balancing for scaling and fault tolerance. We’ll also delve into network monitoring and troubleshooting, ensuring your server’s network stays resilient under high traffic conditions.

3.1 Understanding Linux Network Interfaces

Networking in Linux revolves around network interfaces, which represent the connection between the operating system and the network. Let's begin by reviewing how to list and configure network interfaces.

3.1.1 Listing Network Interfaces

  1. Use ip command (preferred):

    ip a

    This lists all available network interfaces, their states (up/down), assigned IP addresses, and more.

  2. Using ifconfig (older method):

    ifconfig

    While deprecated in favor of ip, ifconfig is still available on many systems.

3.1.2 Configuring Network Interfaces (Static IP)

To manually configure a static IP, edit the network configuration files. We will cover this for both Ubuntu (with netplan) and RedHat (with nmcli).

3.1.2.1 Configuring a Static IP with Netplan (Ubuntu)

Ubuntu 18.04 and later use Netplan for network configuration.

  1. Edit the relevant configuration file in /etc/netplan/. Example:

    sudo nano /etc/netplan/00-installer-config.yaml

  2. Set a static IP for the interface:

    network:
  version: 2
  renderer: networkd
  ethernets:
    ens33:
      dhcp4: no
      addresses:
        - 192.168.1.100/24
      gateway4: 192.168.1.1
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4

  3. Apply the configuration:

    sudo netplan apply
3.1.2.2 Configuring a Static IP with nmcli (RedHat)

RedHat-based distributions use NetworkManager by default. The nmcli command-line utility helps manage connections.

  1. List connections:

    nmcli con show

  2. Modify the connection to use a static IP:

    nmcli con mod "System eth0" ipv4.addresses 192.168.1.100/24
nmcli con mod "System eth0" ipv4.gateway 192.168.1.1
nmcli con mod "System eth0" ipv4.dns "8.8.8.8 8.8.4.4"
nmcli con mod "System eth0" ipv4.method manual
nmcli con up "System eth0"

3.2 Optimizing Network Performance

Optimizing network performance is essential for delivering fast and reliable services, especially when dealing with high traffic. We’ll focus on TCP tuning, adjusting MTU, and tweaking kernel parameters for network performance.

3.2.1 TCP Tuning for Better Performance

TCP (Transmission Control Protocol) governs much of network communication, and tuning its settings can improve network throughput, especially for high-latency connections.

  1. View current TCP settings:

    sysctl -a | grep net.ipv4.tcp

  2. Increase the maximum buffer sizes: Adding these lines to /etc/sysctl.conf optimizes TCP buffer sizes, improving performance for larger data transfers.

    net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_congestion_control = cubic

  3. Apply the changes:

    sudo sysctl -p

3.2.2 Adjusting MTU (Maximum Transmission Unit)

The MTU defines the maximum size of a packet that can be transmitted over the network. Adjusting it can prevent fragmentation and optimize performance for specific environments.

  1. Check the current MTU:

    ip link show eth0

  2. Change the MTU size (e.g., to 1400 bytes):

    sudo ip link set dev eth0 mtu 1400

  3. Make the MTU setting persistent by editing the network configuration:

    • Ubuntu (Netplan): 
      ethernets:
  ens33:
    mtu: 1400
    • RedHat (nmcli): 
      nmcli con mod "System eth0" 802-3-ethernet.mtu 1400
nmcli con up "System eth0"

3.3 Configuring Load Balancing for Scalability and Redundancy

Load balancing distributes traffic across multiple servers, improving both availability and performance. We will configure Nginx as a load balancer for your web servers.

3.3.1 Nginx as a Load Balancer

  1. Install Nginx (if not already installed):

    • Ubuntu: 
      sudo apt install nginx -y
    • RedHat: 
      sudo dnf install nginx -y

  2. Configure Nginx for load balancing by editing the config file /etc/nginx/sites-available/load-balancer:

    upstream backend {
    server 192.168.1.101;
    server 192.168.1.102;
}
 
server {
    listen 80;
    server_name yourdomain.com;
 
    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

  3. Enable the new configuration:

    sudo ln -s /etc/nginx/sites-available/load-balancer /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

3.3.2 DNS-Based Load Balancing with Round Robin

While Nginx provides a straightforward way to load balance, DNS-based round-robin load balancing allows traffic to be distributed at the DNS level.

  1. Edit your DNS records to include multiple A records for your domain, pointing to different server IPs:

    yourdomain.com    A    192.168.1.101
yourdomain.com    A    192.168.1.102

  2. DNS-based load balancing distributes traffic across these IPs in a round-robin manner.

3.4 Monitoring and Troubleshooting Network Performance

Network monitoring and troubleshooting are crucial for identifying bottlenecks and ensuring smooth performance.

3.4.1 Monitoring Network Traffic with iftop

iftop is a real-time command-line tool that shows bandwidth usage on a network interface.

  1. Install iftop:

    • Ubuntu: 
      sudo apt install iftop -y
    • RedHat: 
      sudo dnf install iftop -y

  2. Monitor network traffic on a specific interface:

    sudo iftop -i eth0

3.4.2 Troubleshooting Network Latency with MTR

MTR combines the functionality of traceroute and ping, making it an excellent tool for diagnosing network issues.

  1. Install MTR:

    • Ubuntu: 
      sudo apt install mtr -y
    • RedHat: 
      sudo dnf install mtr -y

  2. Run MTR to trace the path to a destination and check for latency:

    mtr google.com

Part 4: Efficient Storage and Filesystem Management

Mastering Disk Partitions, LVM, RAID, and Performance Tuning

In this part of the series, we will cover the fundamentals of Linux storage management, including configuring disk partitions, managing logical volumes using LVM (Logical Volume Manager), setting up RAID for redundancy and performance, and optimizing file system performance. This part will equip you with the skills to manage storage on Ubuntu and RedHat efficiently.

Official documentation links are included for each tool and concept for further reading.

4.1 Disk Partitioning and Management

Managing partitions is one of the most fundamental tasks when setting up storage for a Linux server. You’ll learn how to view, create, and modify partitions using fdisk and parted tools.

4.1.1 Viewing Disk Information

  1. List disk partitions using fdisk:

    sudo fdisk -l

    This lists all available disks and their partitions.

  2. Using lsblk to list block devices:

    lsblk

    This provides a tree-like view of all block devices, including disks, partitions, and mount points.

4.1.2 Partitioning a Disk with fdisk

  1. Open the disk for partitioning (e.g., /dev/sdb):

    sudo fdisk /dev/sdb

  2. Create a new partition:

    • Press n to create a new partition.
    • Follow the prompts to specify the partition type and size.
    • Press w to write changes to the disk.

  3. Format the partition (e.g., as ext4):

    sudo mkfs.ext4 /dev/sdb1

  4. Mount the partition:

    sudo mount /dev/sdb1 /mnt/data

4.2 Logical Volume Management (LVM)

LVM allows for more flexible disk management than traditional partitioning. With LVM, you can resize volumes dynamically, create snapshots, and aggregate multiple physical disks into a single volume group.

4.2.1 Setting Up LVM

  1. Install LVM tools:

    • Ubuntu: 
      sudo apt install lvm2 -y
    • RedHat: 
      sudo dnf install lvm2 -y

  2. Create Physical Volumes (PVs):

    sudo pvcreate /dev/sdb
sudo pvcreate /dev/sdc

  3. Create a Volume Group (VG):

    sudo vgcreate vg_data /dev/sdb /dev/sdc

  4. Create a Logical Volume (LV):

    sudo lvcreate -L 50G -n lv_data vg_data

  5. Format and mount the logical volume:

    sudo mkfs.ext4 /dev/vg_data/lv_data
sudo mount /dev/vg_data/lv_data /mnt/data

  6. Resize the logical volume (if needed):

    sudo lvresize -L +10G /dev/vg_data/lv_data
sudo resize2fs /dev/vg_data/lv_data

4.3 RAID Configuration for Redundancy and Performance

RAID (Redundant Array of Independent Disks) can improve disk performance, provide redundancy, or both. We’ll use the mdadm tool to set up software RAID on Linux.

4.3.1 Installing mdadm

  1. Install mdadm:

4.3.2 Creating a RAID Array

  1. Create a RAID 1 array (mirroring) with two devices (/dev/sdb and /dev/sdc):

    sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc

  2. Verify the RAID array:

    cat /proc/mdstat

  3. Create a filesystem and mount the RAID array:

    sudo mkfs.ext4 /dev/md0
sudo mount /dev/md0 /mnt/raid

  4. Configure RAID to start at boot:

    sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u

4.4 File System Performance Optimization

Efficient file system management is crucial for maintaining performance under heavy workloads. Here we’ll discuss how to tune file systems for optimal performance and introduce XFS, a high-performance journaling file system used in many enterprise systems.

4.4.1 Choosing the Right File System

While ext4 is the default file system in many Linux distributions, XFS offers better performance for large files and data sets. You can choose a file system based on the workload.

  • ext4: General-purpose, widely supported.
  • XFS: High-performance, scalable for large data sets.

4.4.2 Mount Options for Performance

Mounting a file system with the right options can significantly improve performance. For example, enabling write-back caching or noatime (disabling access time updates) can reduce disk I/O.

  1. Edit /etc/fstab to mount a file system with performance options:

    /dev/sdb1 /mnt/data ext4 defaults,noatime 0 2

    Common mount options:

    • noatime: Disables the update of file access times, reducing disk writes.

    • data=writeback: Defers metadata updates to improve performance (at the cost of consistency).

    • Official fstab documentation:
      fstab man page (opens in a new tab)

4.5 Monitoring and Managing Storage

Regular monitoring and proactive management of storage ensure that your system operates smoothly and doesn’t run out of space.

4.5.1 Checking Disk Usage

  1. View disk usage with df:

    df -h

  2. Check inode usage (number of files and directories):

    df -i

4.5.2 Using du to Find Disk Space Hogs

du is a great tool for finding which directories are using the most space.

  1. Check disk usage for a directory:

    du -sh /path/to/directory
    1. Find the top largest directories:
    du -ah / | sort -rh | head -n 10

4.5.3 Monitoring I/O with iostat

To check how busy your disks are, use iostat (provided by the sysstat package).

  1. Install sysstat:

    • Ubuntu: 
      sudo apt install sysstat -y
    • RedHat: 
      sudo dnf install sysstat -y

  2. Monitor disk I/O:

    iostat -x 5

Part 5: Networking Essentials for Linux Servers

Configuring, Troubleshooting, and Securing Network Interfaces, Firewalls, and DNS

Networking is the backbone of any server infrastructure. In this part, we will cover foundational network configuration, securing network interfaces using firewalls, and managing DNS. We’ll focus on best practices for configuring network interfaces, working with tools like ip, netplan (on Ubuntu), NetworkManager (RedHat), and setting up and troubleshooting firewalls using ufw and firewalld.

Official documentation links are provided for further reading.

5.1 Network Interface Configuration

5.1.1 Checking Network Interfaces

  1. View network interface details using ip:

    ip a

  2. Using ifconfig (deprecated):

    ifconfig

    While still available, ip is now the preferred method.

5.1.2 Configuring Static IP Addresses

For static IP configurations, we will look at both Ubuntu (using netplan) and RedHat (using NetworkManager).

5.1.2.1 Ubuntu with netplan

  1. Edit the netplan configuration file:

    sudo nano /etc/netplan/00-installer-config.yaml

  2. Example static IP configuration:

    network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s3:
      dhcp4: no
      addresses:
        - 192.168.1.100/24
      gateway4: 192.168.1.1
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4

  3. Apply the changes:

    sudo netplan apply
5.1.2.2 RedHat with NetworkManager

  1. Configure a static IP using nmcli:

    sudo nmcli con mod ens33 ipv4.addresses 192.168.1.100/24
sudo nmcli con mod ens33 ipv4.gateway 192.168.1.1
sudo nmcli con mod ens33 ipv4.dns "8.8.8.8 8.8.4.4"
sudo nmcli con mod ens33 ipv4.method manual

  2. Restart the network connection:

    sudo nmcli con up ens33

5.2 DNS Configuration and Management

Domain Name System (DNS) is essential for name resolution in network communication. Configuring and troubleshooting DNS ensures your server can communicate effectively with the outside world.

5.2.1 Setting DNS Servers

  1. On Ubuntu (via netplan): In the netplan configuration file, set DNS under nameservers:

    nameservers:
  addresses:
    - 8.8.8.8
    - 8.8.4.4

  2. On RedHat (using nmcli):

    sudo nmcli con mod ens33 ipv4.dns "8.8.8.8 8.8.4.4"
sudo nmcli con up ens33

5.2.2 Testing DNS Resolution

  1. Check DNS resolution using dig:

    dig example.com

  2. Check DNS resolution using nslookup:

    nslookup example.com

5.3 Network Troubleshooting

5.3.1 Using ping for Basic Connectivity Testing

  1. Ping a host to test network connectivity:

    ping google.com
    • This sends ICMP echo requests to test if the remote host is reachable.

  2. Ping by IP to test DNS issues:

    ping 8.8.8.8

    If the ping by IP works but the DNS name doesn’t, the issue is likely with DNS.

5.3.2 Using traceroute to Diagnose Network Paths

  1. Install traceroute:

    • Ubuntu: 
      sudo apt install traceroute
    • RedHat: 
      sudo dnf install traceroute

  2. Run traceroute to a destination:

    traceroute google.com

    This shows each hop in the path between your server and the destination.

5.3.3 Using ss to Monitor Network Connections

  1. List all active network connections:

    ss -tuln

    This command lists TCP and UDP connections, including listening ports.

5.4 Firewall Configuration and Security

Firewalls protect your server from unauthorized access by controlling incoming and outgoing network traffic.

5.4.1 Using ufw on Ubuntu

ufw (Uncomplicated Firewall) is the default firewall tool on Ubuntu, providing an easy-to-use interface.

  1. Enable ufw:

    sudo ufw enable

  2. Allow SSH access:

    sudo ufw allow ssh

  3. Allow specific port access (e.g., HTTP):

    sudo ufw allow 80/tcp

  4. Check firewall status:

    sudo ufw status

5.4.2 Using firewalld on RedHat

firewalld is a dynamic firewall management tool with support for network zones.

  1. Start and enable firewalld:

    sudo systemctl start firewalld
sudo systemctl enable firewalld

  2. Allow SSH access:

    sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --reload

  3. Allow HTTP traffic:

    sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --reload

  4. Check firewall rules:

    sudo firewall-cmd --list-all

Ultimate Server Administration Series

Part 6: Proactive System Monitoring and Logging

Monitoring and logging are crucial for maintaining the health, security, and performance of your servers. In this part, we’ll cover how to monitor server resources, configure logging for troubleshooting, and automate alerts for critical issues. We’ll explore both built-in tools like top, htop, and journalctl, as well as powerful open-source solutions like Prometheus and Grafana.

6.1 System Resource Monitoring with Built-In Tools

6.1.1 Monitoring CPU and Memory Usage

  1. Using top:
    The top command provides a dynamic, real-time view of the system's CPU, memory usage, and running processes.

    top

    • Key metrics:

      • %CPU: CPU usage by each process.
      • %MEM: Memory usage by each process.
      • Load averages: Displays the system load over 1, 5, and 15-minute intervals.

    • Official top documentation:
      top man page (opens in a new tab)

  2. Using htop:
    htop is an improved, interactive version of top with a more user-friendly interface.

    • Install htop:

      • Ubuntu: 
        sudo apt install htop
      • RedHat: 
        sudo dnf install htop

    • Run htop:

      htop

    • Use F6 to sort processes by different criteria (CPU, Memory, etc.).

    • Official htop documentation:
      htop GitHub page (opens in a new tab)

6.1.2 Monitoring Disk Usage

  1. Using df to check filesystem disk space:

    df -h

    • This displays the disk space usage in human-readable format, showing mounted filesystems, their usage, and available space.

    • Official df documentation:
      df man page (opens in a new tab)

  2. Using du to analyze specific directory sizes:

    du -sh /var/log

6.2 Log Management for Linux Servers

6.2.1 Understanding System Logs with journalctl

  1. Viewing the systemd journal:

    sudo journalctl
    • This command shows logs for services managed by systemd. You can scroll through logs or limit the output with filters.

  2. View logs for a specific service (e.g., nginx):

    sudo journalctl -u nginx
    • The -u flag specifies the unit (service) you want to monitor.

  3. View the most recent logs:

    sudo journalctl -r

6.2.2 Using Traditional Log Files

  1. System logs in /var/log:
    Linux stores most log files in /var/log. Key logs include:

    • /var/log/syslog: General system logs (Ubuntu).
    • /var/log/messages: General system logs (RedHat).
    • /var/log/auth.log: Authentication-related logs.
    • /var/log/nginx/access.log: NGINX access logs.
    • /var/log/nginx/error.log: NGINX error logs.

  2. Viewing logs with tail:

    • To view the last 10 lines of a log file:

      sudo tail /var/log/syslog

    • To view logs in real-time:

      sudo tail -f /var/log/syslog

    • Official tail documentation:
      tail man page (opens in a new tab)

6.2.3 Log Rotation with logrotate

Logs can grow large over time, consuming disk space. Log rotation ensures that old logs are archived and new logs are created.

  1. Check the default logrotate configuration:

    cat /etc/logrotate.conf

  2. Custom logrotate configuration:

    • Add specific configuration for a service (e.g., NGINX):

      sudo nano /etc/logrotate.d/nginx

    • Example content:

      /var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

    • Official logrotate documentation:
      logrotate man page (opens in a new tab)

6.3 Advanced Monitoring with Prometheus and Grafana

Prometheus and Grafana are widely used open-source tools for monitoring and visualizing system metrics. In this section, we’ll set up Prometheus to collect server metrics and Grafana to visualize them.

6.3.1 Installing Prometheus

  1. Download and install Prometheus:

    wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz
tar -xvf prometheus-*.tar.gz
sudo mv prometheus-2.46.0.linux-amd64 /usr/local/prometheus

  2. Create a Prometheus configuration file:

    sudo nano /usr/local/prometheus/prometheus.yml

  3. Basic configuration example:

    global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'linux'
    static_configs:
      - targets: ['localhost:9090']

  4. Start Prometheus:

    cd /usr/local/prometheus
./prometheus --config.file=prometheus.yml

6.3.2 Installing Grafana

  1. Install Grafana:

    • Ubuntu: 
      sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt-get update
sudo apt-get install grafana
    • RedHat: 
      sudo dnf install grafana

  2. Start Grafana:

    sudo systemctl start grafana-server
sudo systemctl enable grafana-server

  3. Access Grafana:
    Open your browser and navigate to http://<your-server-ip>:3000.

  4. Add Prometheus as a data source:

    • In Grafana, go to Configuration > Data Sources, and add Prometheus.

    • Set the URL to http://localhost:9090.

    • Official Grafana documentation:
      Grafana.com (opens in a new tab)

6.4 Alerting with Prometheus Alertmanager

Alertmanager allows you to configure alert rules in Prometheus and notify teams via email, Slack, or other integrations.

6.4.1 Setting up Prometheus Alerting Rules

  1. Define alert rules in the prometheus.yml configuration:

    rule_files:
  - "alerts.yml"

  2. Create the alerts.yml file:

    sudo nano /usr/local/prometheus/alerts.yml

  3. Example alert rule:

    groups:
  - name: example
    rules:
      - alert: HighCPUUsage
        expr: rate(node_cpu_seconds_total{mode!="idle"}[5m]) > 0.85
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage is above 85% for more than 1 minute"

6.4.2 Installing and Configuring Alertmanager

  1. Download and install Alertmanager:
   # Download Alertmanager
   wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz
   
   # Extract the downloaded tar file
   tar -xvf alertmanager-*.tar.gz
   
   # Move to the extracted directory
   sudo mv alertmanager-0.25.0.linux-amd64 /usr/local/alertmanager

  1. Create Alertmanager configuration file:

    sudo nano /usr/local/alertmanager/alertmanager.yml

  2. Basic configuration example:

    global:
  resolve_timeout: 5m
 
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'email'
 
receivers:
  - name: 'email'
    email_configs:
      - to: 'alert@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'your_password'

  3. Start Alertmanager:

    cd /usr/local/alertmanager
./alertmanager --config.file=alertmanager.yml

  4. Configure Prometheus to use Alertmanager: Update your prometheus.yml to include the Alertmanager configuration:

    alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

6.5 Conclusion

Monitoring and logging are integral parts of server administration that help ensure performance, availability, and security. By leveraging built-in tools and powerful frameworks like Prometheus and Grafana, you can create a robust monitoring system for your infrastructure.

In this part, we have covered:

  • Basic system monitoring using built-in commands.
  • Log management strategies and tools.
  • Advanced monitoring and alerting setups with Prometheus and Grafana.

Official Documentation Resources

Turn Ubuntu into Router FirewallUser Privileges and Rbac