Log Monitoring

Here’s the comprehensive log monitoring manual for Ubuntu with the updated instructions for configuring Prometheus logging:

Comprehensive Log Monitoring Manual for Ubuntu

Logs are essential for system management, performance monitoring, and troubleshooting. Here's how to access and manage various logs on Ubuntu, including how to create log files for services like Prometheus and handle networking logs.

1. System Logs

a. Syslog

Purpose: General system messages, including application logs and system events.
Location: /var/log/syslog
Access:
```
sudo less /var/log/syslog
```
Follow:
```
sudo tail -f /var/log/syslog
```

b. dmesg

Purpose: Kernel ring buffer messages, often related to hardware and driver issues.
Location: Output is not stored in a file but can be accessed using the dmesg command.
Access:
```
dmesg | less
```
Follow:
```
dmesg --follow
```

2. Authentication Logs

a. Auth.log

Purpose: Authentication and authorization-related messages, including login attempts and sudo actions.
Location: /var/log/auth.log
Access:
```
sudo less /var/log/auth.log
```
Follow:
```
sudo tail -f /var/log/auth.log
```

3. Application Logs

a. Apache Logs

Purpose: Web server access and error logs.
Access:
- Access Log: /var/log/apache2/access.log
- Error Log: /var/log/apache2/error.log

Follow:

sudo tail -f /var/log/apache2/access.log
sudo tail -f /var/log/apache2/error.log

b. Nginx Logs

Purpose: Web server access and error logs.
Access:
- Access Log: /var/log/nginx/access.log
- Error Log: /var/log/nginx/error.log

Follow:

sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log

4. Package Management Logs

a. APT Logs

Purpose: Logs related to package installation and upgrades.
Location: /var/log/apt/
Access:
- History Log: /var/log/apt/history.log
- Termlog: /var/log/apt/term.log
Follow:
```
sudo tail -f /var/log/apt/history.log
```

5. Systemd Logs

a. Journalctl

Purpose: Systemd logs that include service status and errors.
Location: Managed by journalctl, not stored in a single file.
Access:
```
sudo journalctl
```
Follow:
```
sudo journalctl -f
```

6. Kernel Logs

a. Kernel Logs

Purpose: Specific to kernel messages, typically related to hardware and system functions.
Location: /var/log/kern.log
Access:
```
sudo less /var/log/kern.log
```
Follow:
```
sudo tail -f /var/log/kern.log
```

7. Mail Logs

a. Mail Logs

Purpose: Logs related to mail server operations, if a mail server is installed.
Location: /var/log/mail.log and /var/log/mail.err

Access:

sudo less /var/log/mail.log
sudo less /var/log/mail.err

Follow:

sudo tail -f /var/log/mail.log
sudo tail -f /var/log/mail.err

8. Custom Application Logs

a. Custom Logs

Purpose: Logs generated by custom applications or services.
Location: Varies depending on the application. Common locations are /var/log/ or application-specific directories.
Access:
```
sudo less /path/to/custom/log.log
```
Follow:
```
sudo tail -f /path/to/custom/log.log
```

9. Creating Log Files for Services

We will use Prometheus as an example a. Prometheus Logs
Purpose: To log Prometheus service activity and errors.

Steps:

Adjust Prometheus Service File:

Edit the Prometheus systemd service file, typically located at /etc/systemd/system/prometheus.service or /lib/systemd/system/prometheus.service.

sudo nano /etc/systemd/system/prometheus.service

Modify the ExecStart line to redirect standard output and error to a log file. For example:

[Service]
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  >> /var/log/prometheus/prometheus.log 2>&1

Create the Log Directory and Set Permissions:

Create the directory and set permissions:

sudo mkdir -p /var/log/prometheus
sudo chown prometheus:prometheus /var/log/prometheus

Reload systemd and Restart Prometheus:
- Reload systemd to apply changes:
```
sudo systemctl daemon-reload
```
- Restart the Prometheus service:
```
sudo systemctl restart prometheus
```
- Check the status to ensure it’s running:
```
sudo systemctl status prometheus
```

Access Prometheus Logs:

View the log file:

sudo less /var/log/prometheus/prometheus.log

Follow the log file:

sudo tail -f /var/log/prometheus/prometheus.log

10. Network Logs

1. Netstat Logs

a. Netstat Command

Purpose: Provides information about network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
Usage:
```
sudo netstat -tulnp
```
- -t: Show TCP sockets
- -u: Show UDP sockets
- -l: Show only listening sockets
- -n: Show numerical addresses instead of resolving hostnames
- -p: Show the process using the socket

b. Log Output

Netstat doesn’t create log files; it provides a snapshot of current network connections. Use this command to monitor active connections and troubleshoot issues.

2. Network Manager Logs

a. Network Manager Logs

Purpose: Logs related to network connections and management handled by NetworkManager.
Location: NetworkManager logs are typically found in the system’s general log files.

Access:

From Syslog:

sudo grep NetworkManager /var/log/syslog

Journalctl:
```
sudo journalctl -u NetworkManager
```
Follow Logs:
```
sudo journalctl -u NetworkManager -f
```

b. Log Output

NetworkManager logs provide information about network events, connection changes, and errors. They help troubleshoot issues with network interfaces and connections.

3. Firewall Logs

a. UFW (Uncomplicated Firewall) Logs

Purpose: Logs related to firewall activity, including blocked or allowed connections.
Location: /var/log/ufw.log
Access:
```
sudo less /var/log/ufw.log
```
Follow Logs:
```
sudo tail -f /var/log/ufw.log
```

b. Log Output

UFW logs record detailed information about firewall actions, such as allowed or denied packets. This is useful for security monitoring and troubleshooting firewall rules.

4. IPTables Logs

a. IPTables Logs

Purpose: Logs related to packet filtering rules configured with iptables.
Location: Can be redirected to /var/log/syslog or another log file.

Access:

Using Syslog:
```
sudo grep 'iptables' /var/log/syslog
```
If redirected to a specific log file:
```
sudo less /path/to/iptables.log
```
Follow Logs:
```
sudo tail -f /path/to/iptables.log
```

b. Configuration for Logging IPTables

Add logging rules to your iptables configuration:

sudo iptables -A INPUT -j LOG --log-prefix "IPTABLES INPUT: " --log-level 4
sudo iptables -A OUTPUT -j LOG --log-prefix "IPTABLES OUTPUT: " --log-level 4

5. TCPDump and Wireshark

a. TCPDump

Purpose: Captures and analyzes network traffic in real-time.
Usage:
```
sudo tcpdump -i eth0
```
- Replace eth0 with the relevant network interface.

b. Wireshark

Purpose: Graphical network protocol analyzer for in-depth traffic analysis.
Usage: Install Wireshark and use its GUI to capture and analyze packets.

c. Log Output

Both tcpdump and Wireshark capture network traffic, which can be saved to a file for later analysis.
- TCPDump:
```
sudo tcpdump -i eth0 -w /path/to/output.pcap
```
- Open the .pcap file with Wireshark or another packet analysis tool.

6. Network Interface Statistics

a. Interface Statistics

Purpose: Provides detailed statistics about network interfaces, including packet counts and errors.

Usage:

sudo ifstat
sudo ip -s link
sudo netstat -i

b. Log Output

Useful for monitoring network interface performance and diagnosing issues such as packet loss or errors.

Additional Notes

Log Rotation: Ensure that network log files are rotated to manage their size and prevent them from consuming too much disk space. Check /etc/logrotate.conf and /etc/logrotate.d/ for configurations.
Permissions: Accessing and managing some network logs may require root or sudo privileges.

Find Command Mobaxterm on Ubuntu