Apple Open Source Downloader

This repository defines a Rust crate and CLI program to automate the downloading of Apple's open source code from https://opensource.apple.com/.

The primary goal of this project is to enable more intuitive usage and inspection of Apple's open source code. Using this tool you can:

• Convert the history of an Apple component (like the xnu core OS primitives) to a Git repository and easily view differences between releases.
• Convert the history of a set of open source components (such as everything comprising macOS) to a Git repository and easily view differences between releases.
• Query for all available open source components and releases.

The canonical home for this project is https://github.com/indygreg/apple-opensource-downloader. Please report issues or submit enhancements there.

Installing

# From crates.io
$ cargo install apple-opensource-downloader

# From Git
$ cargo install --git https://github.com/indygreg/apple-opensource-downloader.git --branch main

Using

The apple-opensource-downloader CLI is provided. It defines sub-commands to perform various actions. Run apple-opensource-downloader help to see the help.

Download a Single Component to a Git Repository

The component-to-git sub-command will download all versions of a named software component (see the component list at https://opensource.apple.com/tarballs/) and write their contents as Git commits to a Git repository. This allows you to see differences between the versions of a component.

$ apple-opensource-downloader component-to-git --no-bare xnu aos/xnu
<wait for this to finish>

$ cd aos/xnu
$ git log
commit c2011455c3d75195791bd20d189abae4917c8c81 (HEAD -> main, tag: 7195.141.2)
Author: Apple Open Source <opensource@apple.com>
Date:   Fri Jan 1 00:00:00 2021 +0000

    xnu 7195.141.2

    Downloaded from https://opensource.apple.com/tarballs/xnu/xnu-7195.141.2.tar.gz

commit e76ea20b5519ae2eaf2b74698bd2331141e028fa (tag: 7195.121.3)
Author: Apple Open Source <opensource@apple.com>
Date:   Fri Jan 1 00:00:00 2021 +0000

    xnu 7195.121.3

    Downloaded from https://opensource.apple.com/tarballs/xnu/xnu-7195.121.3.tar.gz

...

The Git trees and commit objects should be deterministic provided that the version of this software is identical and the Apple-hosted source archives don't change. i.e. different machines should produce Git commits with the same commit IDs.

Download all Components to Git Repositories

The components-to-gits sub-command will download each available component and write each to separate Git repositories. It is equivalent to running component-to-git for every named component.

Download An Apple Software Release to a Git Repository

The release-to-git command can be used to download all components in a logical Apple software release (such as macOS or iOS) to a Git repository. This enables you to view the history and changes of open source components between releases.

$ apple-opensource-downloader release-to-git --no-bare macos aos/macOS
$ cd aos/macOS
$ git log

Known Issues

The HTML parsing isn't the most robust and may not scrape all available software.

If Apple changes the HTML on opensource.apple.com, it will break this tool.

If Apple imposes throttling on their servers, it will likely break this tool.

Created Git repositories don't use packfiles and their performance may be sub-optimal. Run git gc after Git repo creation to optimize the Git repositories.

We don't support incrementally updating Git repositories. Git repositories have their history recreated from scratch on every invocation. This is obviously inefficient.

Various advertised URLs on opensource.apple.com result in an HTTP 404. These are sometimes ignored by this tool.

When importing software releases (such as macOS), the components from one release to the next may vary. e.g. SQLite could be there in release A, gone in release B, and reappear in release C. This may make git diff output non-representative.

Git commits have a hard-coded date that has no basis in reality.

The naming and layout of Apple's components can at times be confusing and inconsistent. We don't yet make an effort to reconcile this.

Legal Compliance

The content downloaded by this tool may be governed by license and usage restrictions defined outside this tool. Check for usage restrictions posted at https://opensource.apple.com/ and within the downloaded content.

i.e. if you redistribute the downloaded content, Apple may take an issue with that.