Chapter 09: Monitoring and Ongoing Maintenance

After setting up your secure and optimized web server, it's crucial to ensure it continues to run efficiently and securely. This chapter outlines essential tasks for maintaining server performance, including software updates, PHP upgrades, and key troubleshooting tips.

Server Monitoring

Effective server monitoring helps identify potential issues before they become critical. This section covers monitoring tools, alert policies, and how to investigate alerts.

1. Server Monitoring Tools

If you're using DigitalOcean, performance monitoring is straightforward with built-in tools. For other hosting environments, consider tools like Netdata or SolarWinds Observability.

DigitalOcean Monitoring:

Automatic Setup: If monitoring was enabled during Droplet creation, you’ll have access to performance metrics directly from the Droplet dashboard.
Manual Installation: If monitoring was not enabled initially, install the DigitalOcean metrics agent manually. Follow the DigitalOcean documentation (opens in a new tab) for installation instructions.

Alternative Tools:

Netdata: Offers real-time monitoring with comprehensive metrics.
SolarWinds Observability: Provides extensive monitoring and alerting capabilities.

2. Configuring Alert Policies

Alert policies help automate the monitoring process by notifying you when metrics exceed predefined thresholds.

Access Alert Policies:
- Navigate to the Monitoring tab on your server’s dashboard.
- Click on "Create alert policy."
Set Up Alerts:
- Create alert policies for key metrics such as CPU, memory, and disk utilization.
- Example Thresholds: Set CPU, memory, and disk utilization alerts at 80%.

Alert Policies Setup:

3. Investigating Alerts

High resource usage alerts indicate potential issues. Follow these steps to investigate:

SSH into Your Server:
```
ssh abe@pluto.turnipjuice.media
```
Monitor System Resources:
- Use htop for real-time resource usage:
```
htop
```
- Sorting in htop:
  - Press F6 to change the sort column. For memory usage alerts, sort by memory; for CPU usage, sort by CPU.
Check Specific Services:
- Nginx or PHP: Examine access and error logs.
- MySQL: Enable the slow query log or use application monitoring tools like New Relic.
  - Enabling Slow Query Log:
```
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL slow_query_log_file = '/var/log/mysql/slow.log';
```
Use Additional Monitoring Tools:
- New Relic: Provides detailed application performance insights.
- Grafana and Nagios: Offer advanced monitoring and alerting capabilities.

Example Investigation Scenario:

Recent high CPU alerts revealed slow database queries due to a missing index. Adding the index led to reduced CPU usage:

4. Site Monitoring

Monitor your website’s availability and performance in addition to server health.

Tools for Site Monitoring:

SpinupWP: Includes built-in site monitoring for specific plans.
Oh Dear! or Pingdom: Alternatives for site uptime and performance monitoring.

5. Updating Plugins and Themes

Regular updates are crucial to maintaining security and functionality.

Update WordPress Core, Themes, and Plugins Regularly:
- Navigate to the Updates section in your WordPress dashboard.
- Install updates for WordPress core, themes, and plugins.
Delete Obsolete Themes and Plugins:
- Remove inactive themes and plugins to reduce potential security risks.

6. Verifying Backups

Regularly check that backups are running as expected.

Verify Backup Operation:
- Ensure backups are being created and stored correctly.
- Example: Check if backups are being uploaded to S3.
Test Backups:
- Import SQL backups into a test database to ensure usability.

Common Issue Example:

A Python dependency issue caused backup failures. Regularly test backup functionality to avoid such issues.

7. Reviewing Log Files

Log files are essential for identifying issues before they escalate.

Monitor Key Log Files:
- Nginx Error Logs: /var/log/nginx/error.log
- PHP Error Logs: /var/log/php/error.log
- WordPress Debug Log: /var/www/wordpress/wp-content/debug.log
Use Log Management Tools:
- Papertrail: Aggregates logs and integrates with Slack for alerts.
- Setup Example for Papertrail:
  - Configure Papertrail to monitor logs and send alerts for fatal errors.

8. Setting Up Persistent Debug Log

For tracking obscure errors, move debug.log to a secure location.

Update wp-config.php:

define( 'WP_DEBUG', true );
define( 'WP_DEBUG_DISPLAY', false );
define( 'WP_DEBUG_LOG', '/sites/globex.turnipjuice.media/logs/debug.log' );

Configure Nginx to Restrict Access:
- Add the following block to your Nginx configuration:
```
location ~ \.(ini|log|conf)$ {
    deny all;
}
```

9. Enable Log Rotation

Prevent logs from consuming excessive disk space.

Create Logrotate Configuration:

Add a new file to /etc/logrotate.d/ named after your site:
```
sudo nano /etc/logrotate.d/globex.turnipjuice.media
```

Example Configuration:

/sites/globex.turnipjuice.media/logs/*.log {
    daily
    rotate 14
    size 1M
    compress
    missingok
    notifempty
    dateext
    create 0664 abe abe
    sharedscripts
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
    endscript
}

Configuration Options Explained:
- daily: Rotate logs every day.
- rotate 14: Retain logs for 14 days.
- size 1M: Rotate logs larger than 1MB.
- compress: Compress rotated logs.
- missingok: Ignore errors if log files are missing.
- notifempty: Skip rotating empty log files.
- dateext: Append date to rotated log files.
- create: Set permissions and ownership for new log files.
- postrotate: Run commands after log rotation.

10. Updating Server Packages

Keep server packages up-to-date for security and functionality improvements.

Check for Updates:

sudo apt update
sudo apt list --upgradable

Upgrade Packages:
```
sudo apt dist-upgrade
```
Note: Use dist-upgrade to handle dependencies intelligently.
Backup Before Upgrading:
- Create a system backup or snapshot before performing upgrades.

11. Upgrading PHP

Upgrade to a new major PHP version if required.

Install New PHP Version:

sudo apt update
sudo apt install php8.3-fpm php8.3-common php8.3-mysql \
php8.3-xml php8.3-intl php8.3-curl php8.3-gd \
php8.3-imagick php8.3-cli php8.3-dev php8.3-imap \
php8.3-mbstring php8.3-opcache php8.3-redis \
php8.3-soap php8.3-zip -y

Update Nginx Configuration:

Modify the fastcgi_pass directive in your Nginx configuration:

location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass unix:/run/php/php8.3-fpm.sock;
    fastcgi_index index.php;
    include fastcgi_params;
}

Test and Reload Nginx:
```
sudo nginx -t
sudo service nginx reload
```
Verify PHP Version in WordPress:
- Check "Site Health" under Tools in your WordPress dashboard.
Remove Old PHP Versions:
- After confirming functionality, remove outdated PHP versions.

12. Upgrading Ubuntu

Upgrading the Ubuntu OS can be risky; consider a fresh server setup instead.

Consider New LTS Releases:
- Ubuntu releases new LTS versions every two years. Upgrading involves risks, so a fresh server setup is often preferred.
Provision a New Server:
- Set up a new server with the latest Ubuntu version.
- Migrate sites and configurations to the new server, ensuring compatibility.
Switch DNS:
- Once the new server is tested and functional, switch DNS to minimize downtime.

Here’s a more detailed version of the Action Plan section with explanations for each task:

Action Plan: Server Maintenance Tasks

Regular server maintenance is key to ensuring optimal performance, security, and stability. Below is a recommended schedule for essential tasks, along with detailed instructions on how to carry them out.

Monthly Tasks

To maintain your WordPress site and server, perform the following tasks at least once a month:

1. Perform WordPress Updates (Core, Themes, and Plugins)

Keeping WordPress core, themes, and plugins up-to-date is essential for security and functionality.

Steps:

Update WordPress Core:
- Navigate to your WordPress dashboard.
- Go to Dashboard > Updates and update WordPress if a new version is available.
Update Themes and Plugins:
- Scroll down to see available updates for themes and plugins.
- Install updates individually or use the "Select All" option to bulk update.

Why It’s Important:

Updates often include patches for known vulnerabilities, making your site more secure.
Themes and plugins may introduce new features or compatibility fixes.

Troubleshooting:

If the site breaks after an update, check the debug.log in /wp-content/ for errors. Consider rolling back to a previous plugin version if necessary.

2. Ensure Backups Are Running and Usable

Backups are your safety net in case something goes wrong. Ensure they are both running as scheduled and restorable.

Steps:

Verify Backup Schedule:
- Check your backup script or tool (e.g., UpdraftPlus, BackupBuddy, custom script) to confirm backups are being created as expected.
Test Backup Usability:
- Select a recent backup and attempt to restore it on a staging environment or local server to ensure the backup is complete and functional.

Why It’s Important:

Backups are critical for recovering from data loss, server failure, or malicious attacks.
A backup that isn’t restorable is as good as no backup at all.

Troubleshooting:

If a backup fails or cannot be restored, review logs in the backup directory or the relevant plugin's dashboard. Common issues include permission problems or missing files.

3. Check Server Metrics for Unusual Spikes

Monitoring your server's resource usage helps you detect and prevent potential issues before they impact performance.

Steps:

Access Server Metrics:
- If using a platform like DigitalOcean, review your Droplet’s metrics dashboard.
- For self-hosted environments, use tools like Netdata, Grafana, or htop to monitor resource usage.
Look for Spikes:
- Pay attention to any unusual spikes in CPU, memory, or disk usage.
- Investigate the time frame when spikes occurred to correlate it with website traffic or recent updates.

Why It’s Important:

Spikes in resource usage may indicate underlying issues such as inefficient code, malicious activity, or sudden traffic increases.

Troubleshooting:

Use logs to determine the root cause of spikes. For example, a slow MySQL query could result in high CPU usage. Enable slow query logging to identify problematic queries:
```
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL slow_query_log_file = '/var/log/mysql/slow.log';
```

4. Scan Server’s Error Logs for Problems

Regularly reviewing error logs helps you detect and resolve issues before they become critical.

Steps:

Locate Key Log Files:
- Nginx logs: /var/log/nginx/error.log
- PHP logs: /var/log/php8.3-fpm.log
- WordPress Debug Log: /wp-content/debug.log (if enabled)
Scan for Errors:
- Open these logs and search for recurring or serious errors.
- Look for HTTP status codes like 500, 502, or 503, as these indicate server errors.

Why It’s Important:

Persistent errors could indicate problems with server configuration, plugin incompatibilities, or vulnerabilities that need to be addressed.

Troubleshooting:

Use the error message or stack trace from the logs to trace the issue. For PHP errors, focus on identifying the script causing the problem and correct it accordingly.

5. Update Server Packages

Keeping your server’s software up to date ensures you benefit from the latest security patches and performance improvements.

Steps:

Check for Updates:

sudo apt update
sudo apt list --upgradable

Upgrade Packages:
```
sudo apt upgrade
sudo apt dist-upgrade
```
- dist-upgrade intelligently handles package dependencies during the upgrade process.
Reboot If Necessary:
- Some updates, especially kernel updates, may require a server reboot to take effect:
```
sudo reboot
```

Why It’s Important:

Regular updates protect your server from security vulnerabilities and ensure compatibility with the latest software.

Troubleshooting:

If an update fails, check the /var/log/apt/history.log or /var/log/dpkg.log for detailed error messages.

6. Plan for Ubuntu Version Upgrades

Ubuntu Long-Term Support (LTS) versions receive updates for five years, but after their end-of-life (EOL), no further security updates or patches are released. It’s important to stay aware of these dates and plan for timely upgrades.

Steps:

Check Current Version and EOL Date:
```
lsb_release -a
```
- Visit Ubuntu’s release page (opens in a new tab) to verify the end-of-life date for your version.
Provision a New Server:
- When approaching the EOL date, provision a new server with the latest LTS version.
- Migrate your websites and applications to the new server to avoid downtime or security risks.

Why It’s Important:

Running an EOL version of Ubuntu leaves your server vulnerable to exploits since no more updates, including security patches, will be released.

Best Practice:

Plan to move your sites to a fresh server roughly every two years, aligning this with the release of a new Ubuntu LTS version.

8 Migrating WordPress to a Diffrent Server Config