Here’s the comprehensive log monitoring manual for Ubuntu with the updated instructions for configuring Prometheus logging:

Comprehensive Log Monitoring Manual for Ubuntu

Logs are essential for system management, performance monitoring, and troubleshooting. Here's how to access and manage various logs on Ubuntu, including how to create log files for services like Prometheus and handle networking logs.

1. System Logs

a. Syslog

• Purpose: General system messages, including application logs and system events.
• Location: /var/log/syslog
• Access:

  sudo less /var/log/syslog
• Follow:

  sudo tail -f /var/log/syslog

b. dmesg

• Purpose: Kernel ring buffer messages, often related to hardware and driver issues.
• Location: Output is not stored in a file but can be accessed using the dmesg command.
• Access:

  dmesg | less
• Follow:

  dmesg --follow

2. Authentication Logs

a. Auth.log

• Purpose: Authentication and authorization-related messages, including login attempts and sudo actions.
• Location: /var/log/auth.log
• Access:

  sudo less /var/log/auth.log
• Follow:

  sudo tail -f /var/log/auth.log

3. Application Logs

a. Apache Logs

• Purpose: Web server access and error logs.
• Access:
  • Access Log: /var/log/apache2/access.log
  • Error Log: /var/log/apache2/error.log
• Follow:

  sudo tail -f /var/log/apache2/access.log
  sudo tail -f /var/log/apache2/error.log

b. Nginx Logs

• Purpose: Web server access and error logs.
• Access:
  • Access Log: /var/log/nginx/access.log
  • Error Log: /var/log/nginx/error.log
• Follow:

  sudo tail -f /var/log/nginx/access.log
  sudo tail -f /var/log/nginx/error.log

4. Package Management Logs

a. APT Logs

• Purpose: Logs related to package installation and upgrades.
• Location: /var/log/apt/
• Access:
  • History Log: /var/log/apt/history.log
  • Termlog: /var/log/apt/term.log
• Follow:

  sudo tail -f /var/log/apt/history.log

5. Systemd Logs

a. Journalctl

• Purpose: Systemd logs that include service status and errors.
• Location: Managed by journalctl, not stored in a single file.
• Access:

  sudo journalctl
• Follow:

  sudo journalctl -f

6. Kernel Logs

a. Kernel Logs

• Purpose: Specific to kernel messages, typically related to hardware and system functions.
• Location: /var/log/kern.log
• Access:

  sudo less /var/log/kern.log
• Follow:

  sudo tail -f /var/log/kern.log

7. Mail Logs

a. Mail Logs

• Purpose: Logs related to mail server operations, if a mail server is installed.
• Location: /var/log/mail.log and /var/log/mail.err
• Access:

  sudo less /var/log/mail.log
  sudo less /var/log/mail.err
• Follow:

  sudo tail -f /var/log/mail.log
  sudo tail -f /var/log/mail.err

8. Custom Application Logs

a. Custom Logs

• Purpose: Logs generated by custom applications or services.
• Location: Varies depending on the application. Common locations are /var/log/ or application-specific directories.
• Access:

  sudo less /path/to/custom/log.log
• Follow:

  sudo tail -f /path/to/custom/log.log

9. Creating Log Files for Services

• We will use Prometheus as an example a. Prometheus Logs
• Purpose: To log Prometheus service activity and errors.
• Steps:

  1. Adjust Prometheus Service File:

     • Edit the Prometheus systemd service file, typically located at /etc/systemd/system/prometheus.service or /lib/systemd/system/prometheus.service.

     sudo nano /etc/systemd/system/prometheus.service
     • Modify the ExecStart line to redirect standard output and error to a log file. For example:

     [Service]
     ExecStart=/usr/local/bin/prometheus \
       --config.file=/etc/prometheus/prometheus.yml \
       --storage.tsdb.path=/var/lib/prometheus/ \
       --web.console.templates=/etc/prometheus/consoles \
       --web.console.libraries=/etc/prometheus/console_libraries \
       >> /var/log/prometheus/prometheus.log 2>&1

  2. Create the Log Directory and Set Permissions:

     • Create the directory and set permissions:

     sudo mkdir -p /var/log/prometheus
     sudo chown prometheus:prometheus /var/log/prometheus

  3. Reload systemd and Restart Prometheus:

     • Reload systemd to apply changes:

     sudo systemctl daemon-reload
     • Restart the Prometheus service:

     sudo systemctl restart prometheus
     • Check the status to ensure it’s running:

     sudo systemctl status prometheus

  4. Access Prometheus Logs:

     • View the log file:

     sudo less /var/log/prometheus/prometheus.log
     • Follow the log file:

     sudo tail -f /var/log/prometheus/prometheus.log

10. Network Logs

1. Netstat Logs

a. Netstat Command

• Purpose: Provides information about network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
• Usage:

  sudo netstat -tulnp
  • -t: Show TCP sockets
  • -u: Show UDP sockets
  • -l: Show only listening sockets
  • -n: Show numerical addresses instead of resolving hostnames
  • -p: Show the process using the socket

b. Log Output

• Netstat doesn’t create log files; it provides a snapshot of current network connections. Use this command to monitor active connections and troubleshoot issues.

2. Network Manager Logs

a. Network Manager Logs

• Purpose: Logs related to network connections and management handled by NetworkManager.
• Location: NetworkManager logs are typically found in the system’s general log files.
• Access:
  • From Syslog:

    sudo grep NetworkManager /var/log/syslog
  • Journalctl:

    sudo journalctl -u NetworkManager
  • Follow Logs:

    sudo journalctl -u NetworkManager -f

b. Log Output

• NetworkManager logs provide information about network events, connection changes, and errors. They help troubleshoot issues with network interfaces and connections.

3. Firewall Logs

a. UFW (Uncomplicated Firewall) Logs

• Purpose: Logs related to firewall activity, including blocked or allowed connections.
• Location: /var/log/ufw.log
• Access:

  sudo less /var/log/ufw.log
• Follow Logs:

  sudo tail -f /var/log/ufw.log

b. Log Output

• UFW logs record detailed information about firewall actions, such as allowed or denied packets. This is useful for security monitoring and troubleshooting firewall rules.

4. IPTables Logs

a. IPTables Logs

• Purpose: Logs related to packet filtering rules configured with iptables.
• Location: Can be redirected to /var/log/syslog or another log file.
• Access:
  • Using Syslog:

    sudo grep 'iptables' /var/log/syslog
  • If redirected to a specific log file:

    sudo less /path/to/iptables.log
  • Follow Logs:

    sudo tail -f /path/to/iptables.log

b. Configuration for Logging IPTables

• Add logging rules to your iptables configuration:

  sudo iptables -A INPUT -j LOG --log-prefix "IPTABLES INPUT: " --log-level 4
  sudo iptables -A OUTPUT -j LOG --log-prefix "IPTABLES OUTPUT: " --log-level 4

5. TCPDump and Wireshark

a. TCPDump

• Purpose: Captures and analyzes network traffic in real-time.
• Usage:

  sudo tcpdump -i eth0
  • Replace eth0 with the relevant network interface.

b. Wireshark

• Purpose: Graphical network protocol analyzer for in-depth traffic analysis.
• Usage: Install Wireshark and use its GUI to capture and analyze packets.

c. Log Output

• Both tcpdump and Wireshark capture network traffic, which can be saved to a file for later analysis.
  • TCPDump:

    sudo tcpdump -i eth0 -w /path/to/output.pcap
  • Open the .pcap file with Wireshark or another packet analysis tool.

6. Network Interface Statistics

a. Interface Statistics

• Purpose: Provides detailed statistics about network interfaces, including packet counts and errors.
• Usage:

  sudo ifstat
  sudo ip -s link
  sudo netstat -i

b. Log Output

• Useful for monitoring network interface performance and diagnosing issues such as packet loss or errors.

Additional Notes

• Log Rotation: Ensure that network log files are rotated to manage their size and prevent them from consuming too much disk space. Check /etc/logrotate.conf and /etc/logrotate.d/ for configurations.
• Permissions: Accessing and managing some network logs may require root or sudo privileges.