Synchronizing files between directories is a common task for managing backups, ensuring consistency across multiple storage locations, or simply keeping data organized.

While there are many tools available to do this, creating a Python script to handle directory synchronization offers flexibility and control.

This guide will walk you through a Python script designed to synchronize files between two directories.

Introduction to the Script

The script begins by importing several essential Python libraries.

These include os for interacting with the operating system, shutilfor high-level file operations, filecmpfor comparing files, argparsefor parsing command-line arguments, and tqdmfor displaying progress bars during lengthy operations.

These libraries work together to create a robust solution for directory synchronization.

import os
import shutil
import filecmp
import argparse
from tqdm import tqdm

The scripts uses mainly Python built-in modules, but for the progress bar is uses the tqdmlibrary, which needs to the installed with:

pip install tqdm

Checking and Preparing Directories

Before starting the synchronization, the script needs to check if the source directory exists.

If the destination directory doesn't exist, the script will create it.

This step is important to make sure the synchronization process can run smoothly without any issues caused by missing directories.

# Function to check if the source and destination directories exist
def check_directories(src_dir, dst_dir):
    # Check if the source directory exists
    if not os.path.exists(src_dir):
        print(f"\nSource directory '{src_dir}' does not exist.")
        return False
    # Create the destination directory if it does not exist
    if not os.path.exists(dst_dir):
        os.makedirs(dst_dir)
        print(f"\nDestination directory '{dst_dir}' created.")
    return True

The check_directories function makes sure that both the source and destination directories are ready for synchronization. Here's how it works:

The function uses os.path.exists() to check if the directories exist.
If the source directory is missing, the script tells the user and stops running.
If the destination directory is missing, the script creates it automatically using os.makedirs(). This ensures that the necessary directory structure is in place.

Synchronizing Files Between Directories

The main job of the script is to synchronize files between the source and destination directories.

The sync_directories function handles this task by first going through the source directory to gather a list of all files and subdirectories.

The os.walk function helps by generating file names in the directory tree, allowing the script to capture every file and folder within the source directory.

# Function to synchronize files between two directories
def sync_directories(src_dir, dst_dir, delete=False):
    # Get a list of all files and directories in the source directory
    files_to_sync = []
    for root, dirs, files in os.walk(src_dir):
        for directory in dirs:
            files_to_sync.append(os.path.join(root, directory))
        for file in files:
            files_to_sync.append(os.path.join(root, file))

    # Iterate over each file in the source directory with a progress bar
    with tqdm(total=len(files_to_sync), desc="Syncing files", unit="file") as pbar:
        # Iterate over each file in the source directory
        for source_path in files_to_sync:
            # Get the corresponding path in the replica directory
            replica_path = os.path.join(dst_dir, os.path.relpath(source_path, src_dir))

            # Check if path is a directory and create it in the replica directory if it does not exist
            if os.path.isdir(source_path):
                if not os.path.exists(replica_path):
                    os.makedirs(replica_path)
            # Copy all files from the source directory to the replica directory
            else:
                # Check if the file exists in the replica directory and if it is different from the source file
                if not os.path.exists(replica_path) or not filecmp.cmp(source_path, replica_path, shallow=False):
                    # Set the description of the progress bar and print the file being copied
                    pbar.set_description(f"Processing '{source_path}'")
                    print(f"\nCopying {source_path} to {replica_path}")

                    # Copy the file from the source directory to the replica directory
                    shutil.copy2(source_path, replica_path)

            # Update the progress bar
            pbar.update(1)

Once the list of files and directories is compiled, the script uses a progress bar provided by tqdm to give the user feedback on the synchronization process.

For each file and directory in the source, the script calculates the corresponding path in the destination.

If the path is a directory, the script ensures it exists in the destination.

If the path is a file, the script checks whether the file already exists in the destination and whether it is identical to the source file.

If the file is missing or different, the script copies it to the destination.

This way, the script keeps the destination directory up-to-date with the source directory.

Cleaning Up Extra Files

The script also has an optional feature to delete files in the destination directory that are not in the source directory.

This is controlled by a --delete flag that the user can set.

If this flag is used, the script goes through the destination directory and compares each file and folder to the source.

If it finds anything in the destination that isn't in the source, the script deletes it.

This ensures that the destination directory is an exact copy of the source directory.

# Clean up files in the destination directory that are not in the source directory, if delete flag is set
    if delete:
        # Get a list of all files in the destination directory
        files_to_delete = []
        for root, dirs, files in os.walk(dst_dir):
            for directory in dirs:
                files_to_delete.append(os.path.join(root, directory))
            for file in files:
                files_to_delete.append(os.path.join(root, file))

        # Iterate over each file in the destination directory with a progress bar
        with tqdm(total=len(files_to_delete), desc="Deleting files", unit="file") as pbar:
            # Iterate over each file in the destination directory
            for replica_path in files_to_delete:
                # Check if the file exists in the source directory
                source_path = os.path.join(src_dir, os.path.relpath(replica_path, dst_dir))
                if not os.path.exists(source_path):
                    # Set the description of the progress bar
                    pbar.set_description(f"Processing '{replica_path}'")
                    print(f"\nDeleting {replica_path}")

                    # Check if the path is a directory and remove it
                    if os.path.isdir(replica_path):
                        shutil.rmtree(replica_path)
                    else:
                        # Remove the file from the destination directory
                        os.remove(replica_path)

                # Update the progress bar
                pbar.update(1)

This part of the script uses similar techniques as the synchronization process.

It uses os.walk() to gather files and directories and tqdm to show progress.
The shutil.rmtree() function is used to remove directories, while os.remove() handles individual files.

Running the Script

The script is designed to be run from the command line, with arguments specifying the source and destination directories.

The argparse module makes it easy to handle these arguments, allowing users to simply provide the necessary paths and options when running the script.

# Main function to parse command line arguments and synchronize directories
if __name__ == "__main__":
    # Parse command line arguments
    parser = argparse.ArgumentParser(description="Synchronize files between two directories.")
    parser.add_argument("source_directory", help="The source directory to synchronize from.")
    parser.add_argument("destination_directory", help="The destination directory to synchronize to.")
    parser.add_argument("-d", "--delete", action="store_true",
                        help="Delete files in destination that are not in source.")
    args = parser.parse_args()

    # If the delete flag is set, print a warning message
    if args.delete:
        print("\nExtraneous files in the destination will be deleted.")

    # Check the source and destination directories
    if not check_directories(args.source_directory, args.destination_directory):
        exit(1)

    # Synchronize the directories
    sync_directories(args.source_directory, args.destination_directory, args.delete)
    print("\nSynchronization complete.")

The main function brings everything together.

It processes the command-line arguments, checks the directories, and then performs the synchronization.

If the --delete flag is set, it also handles the cleanup of extra files.

Examples

Let's see some examples of how to run the script with the different options.

Source to Destination

python file_sync.py d:\sync d:\sync_copy

Destination directory 'd:\sync2' created.
Processing 'd:\sync\video.mp4':   0%|                                                                                                   | 0/5 [00:00<?, ?file/s]
Copying d:\sync\video.mp4 to d:\sync2\video.mp4
Processing 'd:\sync\video_final.mp4':  20%|██████████████████▌                                                                          | 1/5 [00:00<?, ?file/s] 
Copying d:\sync\video_final.mp4 to d:\sync2\video_final.mp4
Processing 'd:\sync\video_single - Copy (2).mp4':  40%|████████████████████████████████▍                                                | 2/5 [00:00<?, ?file/s] 
Copying d:\sync\video_single - Copy (2).mp4 to d:\sync2\video_single - Copy (2).mp4
Processing 'd:\sync\video_single - Copy.mp4':  60%|█████████████████████████████████████████████▌                              | 3/5 [00:00<00:00, 205.83file/s]
Copying d:\sync\video_single - Copy.mp4 to d:\sync2\video_single - Copy.mp4
Processing 'd:\sync\video_single.mp4':  80%|██████████████████████████████████████████████████████████████████▍                | 4/5 [00:00<00:00, 274.44file/s] 
Copying d:\sync\video_single.mp4 to d:\sync2\video_single.mp4
Processing 'd:\sync\video_single.mp4': 100%|███████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 343.05file/s] 

Synchronization complete.

Source to Destination with Cleanup of Extra Files

python file_sync.py d:\sync d:\sync_copy -d

Extraneous files in the destination will be deleted.
Syncing files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 63.29file/s]
Deleting files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<?, ?file/s] 

Synchronization complete.

Conclusion

This Python script offers a powerful and flexible way to synchronize files between two directories.

It uses key libraries like os, shutil, and filecmp, and enhances the user experience with tqdm for tracking progress.

This ensures that your data is consistently and efficiently synchronized.
Whether you're maintaining backups or ensuring consistency across storage locations, this script can be a valuable tool in your toolkit.

Synchronizing Files Between Two Directories Using Python