scandir-rs

scandir_rs is a directory iteration module like os.walk(), but with more features and higher speed. Depending on the function call it yields a list of paths, tuple of lists grouped by their entry type or DirEntry objects that include file type and stat information along with the name. Using scandir_rs is about 2-17 times faster than os.walk() (depending on the platform, file system and file tree structure) by parallelizing the iteration in background.

If you are just interested in directory statistics you can use the Count.

scandir_rs contains following classes:

• Count for determining statistics of a directory.
• Walk for getting names of directory entries.
• Scandir for getting detailed stats of directory entries.

For the API see:

• Class Count doc/count.md
• Class Walk doc/walk.md
• Class Scandir doc/scandir.md

Installation

For building this wheel from source you need the tool maturin.

Install maturin:

cargo install maturin

IMPORTANT: In order to build this project at least Rust version 1.61 is needed!

Build wheel (not on Windows):

maturin build --release --strip

Build wheel on Windows:

maturin build --release --strip --no-sdist

maturin will build the wheels for all Python versions installed on your system.

Building and running tests for different Python versions

To make it easier to build wheels for several different Python versions the script build_wheels.sh has been added. It creates wheels for Python versions 3.7, 3.8, 3.9, 3.10 and 3.11. In addition it runs pytest after successfull creation of each wheel.

Instruction how to install pyenv can be found here.

Examples

Get statistics of a directory:

from scandir_rs import Count, ReturnType

print(Count("/usr", return_type=ReturnType.Ext).collect())

The collect method releases the GIL. So other Python threads can run in parallel.

The same, but asynchronously in background using a class instance:

from scandir_rs import Count, ReturnType

instance = Count("/usr", return_type=ReturnType.Ext))
instance.start())  # Start scanning the directory
...
values = instance.results()  # Returns the current statistics. Can be read at any time
...
if instance.busy():  # Check if the task is still running.
...
instance.stop()  # If you want to cancel the task
...
instance.join()  # Wait for the instance to finish.

and with a context manager:

import time

from scandir_rs import Count, ReturnType

with Count("/usr", return_type=ReturnType.Ext) as instance:
    while instance.busy():
        statistics = instance.results()
        # Do something
        time.sleep(0.01)
    print(instance.results())

os.walk() example:

from scandir_rs import Walk

for root, dirs, files in Walk("/usr"):
    # Do something

with extended data:

from scandir_rs import Walk, ReturnType

for root, dirs, files, symlinks, other, errors in Walk("/usr", return_type=ReturnType.Ext):
    # Do something

os.scandir() example:

from scandir_rs import Scandir, ReturnType

for path, entry in Scandir("~/workspace", return_type=ReturnType.Ext):
    # entry is a custom DirEntry object

Benchmarks

See examples/benchmark.py

In the below table the line Walk.iter returns comparable results to os.walk.

Linux with Ryzen 5 2400G and SSD

Directory /usr with

• 110171 directories
• 862634 files
• 47804 symlinks
• 12275 hardlinks
• 12 devices
• 0 pipes
• 32.7GB size and 34.8GB usage on disk

Around ~3 times faster on Linux (os.walk compared to Walk.iter).

Windows 10 with Laptop Core i7-4810MQ @ 2.8GHz Laptop, MTF SSD

Directory C:\Windows with

• 132604 directories
• 349911 files
• 44.4GB size and 45.2GB usage on disk

Around ~5.3 times faster on Windows 10 (os.walk compared to Walk.iter).

Directory linux-5.9 with

• 4711 directories
• 69973 files
• 1.08GB size and 1.23GB usage on disk

Around ~2.5 times faster on Linux (os.walk compared to Walk.iter).

Around ~6.4 times faster on Windows 10 (os.walk compared to Walk.iter).