Microsoft Azure Blob Storage with rclone: Procedure Guide

This guide outlines how to configure and utilize Azure Blob Storage with rclone. The steps include creating a remote, managing containers, and performing file operations.

1. Pre-Requisites

• Ensure you have an Azure Storage Account.
• Install rclone on your system. Download it from rclone.org/downloads (opens in a new tab).
• Obtain your Azure Storage Account Name and Key from the Azure Portal.

2. Configure rclone for Azure Blob Storage

Step 1: Start Configuration

Run the following command in your terminal:

rclone config
or
sudo rclone config # For configuration with root privileges

Step 2: Create a New Remote

Follow the interactive prompts:

1. Prompt: No remotes found, make a new one?
   Enter n for a new remote.

2. Prompt: name>
   Provide a name for your remote (e.g., azureblob_remote).

3. Prompt: Type of storage to configure.
   Select Microsoft Azure Blob Storage from the list by typing its corresponding number or entering "azureblob".

4. Prompt: Storage Account Name
   Enter your Azure Storage Account name.

5. Prompt: Storage Account Key
   Paste your Azure Storage Account key.

6. Prompt: Endpoint for the service
   Leave this blank unless using a custom endpoint.

7. Confirmation: Keep this remote?
   Enter y to confirm.

3. Verify Configuration

Step 1: List All Containers

To confirm connectivity, list all containers in your Azure Blob Storage account:

rclone lsd azureblob_remote:

4. Perform Basic Operations

Create a New Container

rclone mkdir azureblob_remote:mycontainer

List Contents of a Container

rclone ls azureblob_remote:mycontainer

Sync Local Files to Azure Blob Storage

Sync a local directory to the remote container, deleting any extra files in the container:

rclone sync --interactive /path/to/local/directory azureblob_remote:mycontainer

Optional: Enable Fast Listing

For improved performance during listing operations, add the --fast-list flag:

rclone ls azureblob_remote:mycontainer --fast-list

5. Advanced Options

Use Server-Side Modification Times

If you prefer using Azure’s built-in LastModified timestamp for files, enable the following flag:

rclone sync /path/to/local/directory azureblob_remote:mycontainer --use-server-modtime

Note: This option is recommended when syncing files to avoid unnecessary updates.

Optimize Large File Uploads

Increase concurrency to improve performance for large files:

rclone sync /path/to/large/file azureblob_remote:mycontainer --azureblob-upload-concurrency 64

6. Filename Restrictions

Azure Blob Storage restricts certain characters and formats in filenames. rclone automatically replaces these:

7. Authentication Methods

Option 1: Use Account Name and Key

This is the most straightforward method. During rclone config, provide the account name and key.

Option 2: Use SAS URL

For container-specific operations:

1. Generate a SAS URL in Azure Portal for the desired container.
2. During rclone config, leave account and key blank and fill in the sas_url field.

8. Example Commands

Sync a Folder (Non-Destructive)

rclone copy /path/to/folder azureblob_remote:mycontainer

Download from Azure Blob Storage

rclone copy azureblob_remote:mycontainer /path/to/local/folder

List with Details

rclone lsl azureblob_remote:mycontainer

Step 8: Automating File Uploads

Automation with a Shell Script

Create a shell script to automate the synchronization process.

1. Create the Script:

   nano automate_sync.sh

2. Add the Following Script:

   #!/bin/bash
   # Set variables
   LOCAL_DIR="/path/to/local/directory"
   REMOTE="azureblobremote:containername"
   LOG_FILE="/path/to/logfile.log"
    
   # Run rclone sync with exclusions
   rclone sync "$LOCAL_DIR" "$REMOTE" --exclude "temp/**" --exclude "*.log" --log-file="$LOG_FILE" --fast-list
    
   echo "Sync completed: $(date)" >> "$LOG_FILE"

3. Make the Script Executable:

   chmod +x automate_sync.sh

4. Run the Script:

   ./automate_sync.sh

Step 9: Schedule Automation with Cron

To run the script automatically at specified intervals, use a Cron job.

1. Open the Cron editor:

   crontab -e

2. Add the following Cron entry to run the script daily at midnight:

   0 0 * * * /path/to/automate_sync.sh

3. Save and exit.

Step 10: Validate and Monitor

• Verify Exclusions: Check the Azure Blob container to ensure excluded files and folders are not uploaded.
• Monitor Logs: Inspect the log file (/path/to/logfile.log) for detailed synchronization status.

Advanced Options

• Modify Upload Concurrency: To speed up uploads for large files, increase the concurrency:

  rclone sync /local/dir azureblobremote:containername --azureblob-upload-concurrency=64
• Use --dry-run: Test the synchronization without actual uploads:

  rclone sync /local/dir azureblobremote:containername --dry-run

References

• Rclone Documentation for Azure Blob (opens in a new tab)
• Exclusion Options in Rclone (opens in a new tab)

This guide ensures secure, efficient, and automated synchronization with granular control over included and excluded files or folders.

Use Rclone with Exclusions

Create a file called exclude.txt and add the directories to exclude

/proc/**
/sys/**
/tmp/**
/dev/**
/run/**
/mnt/**
/media/**
/swapfile/**
/var/log/**
 

Basic Command: Use the --exclude-from flag to provide your exclude file

sudo rclone sync / myremote:backup/ --exclude-from=/path/to/exclude.txt

Explanation of Flags:

• sudo: Ensures you have access to all system files.
• sync: Ensures that only updated and new files are uploaded.
• /: Specifies the root directory to copy.
• myremote:backup/: The destination directory in the remote.
• --exclude-from=/path/to/exclude.txt: Skips the files/directories listed in exclude.txt.

Additional Commands for Exclusion

Dry Run: Test your command without making changes

sudo rclone sync / myremote:backup/ --exclude-from=/path/to/exclude.txt --dry-run

Logging: Add logs for debugging

sudo rclone sync / myremote:backup/ --exclude-from=/path/to/exclude.txt --log-file=/path/to/rclone.log --log-level INFO

Bandwidth Limiting: Useful for limiting network usage

sudo rclone sync / myremote:backup/ --exclude-from=/path/to/exclude.txt --bwlimit=10M

Preserving Timestamps and Symbolic Links

Use --links to preserve symbolic links if necessary:

sudo rclone sync / myremote:backup/ --exclude-from=/path/to/exclude.txt --links

11. Troubleshooting

• Permission Errors: Ensure the Storage Account key or SAS URL has the required permissions.
• File Listing Issues: Use the --fast-list flag to optimize operations for large containers.