Linky

Extract links from Markdown files and check links for brokenness.

Motivation

Maintaining Markdown documentation you often end up with lots of links to tend to. Linky extracts all links and checks them, pointing out the broken ones so you can fix them. Specifically, linky was created to ease maintenance of Markdown documentation on Github.

Installation

Install stable Rust and Cargo:

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Compile and install linky using Cargo:

$ cargo install linky

Usage

Extracting and checking links

The simplest thing you can do with linky is to extract links from a Markdown file:

$ linky example_site/path/to/example.md
example_site/path/to/example.md:3:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md
example_site/path/to/example.md:4:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#existing
example_site/path/to/example.md:5:  other.md
example_site/path/to/example.md:6:  non-existing.md
example_site/path/to/example.md:7:  other.md#existing
example_site/path/to/example.md:8:  other.md#non-existing
example_site/path/to/example.md:9:  #heading
example_site/path/to/example.md:10:  #non-existing
example_site/path/to/example.md:11:  #heading-with-code
example_site/path/to/example.md:12:  #HEADING

The output lists all the extracted links along with their respective input files and line numbers.

Enable the --check option to resolve those links:

$ linky --check example_site/path/to/example.md
example_site/path/to/example.md:3: OK https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md
example_site/path/to/example.md:4: NO_FRAG https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#existing
example_site/path/to/example.md:5: OK other.md
example_site/path/to/example.md:6: NO_DOC non-existing.md
example_site/path/to/example.md:7: OK other.md#existing
example_site/path/to/example.md:8: NO_FRAG other.md#non-existing
example_site/path/to/example.md:9: OK #heading
example_site/path/to/example.md:10: NO_FRAG #non-existing
example_site/path/to/example.md:11: OK #heading-with-code
example_site/path/to/example.md:12: CASE_FRAG #HEADING

A status token is now added to each line indicating the outcome of the resolution. An OK token indicates that the resolution succeeded without remarks. For details on how links are resolved see the link resolution section.

Recursive directory traversal

Linky doesn't do directory traversal on its own. Instead it integrates well with find and xargs:

$ find example_site -type f -print0 | xargs -0 linky
example_site/path/to/absolute.md:2:  /path/to/other.md
example_site/path/to/absolute.md:3:  /path/to/non-existing.md
example_site/path/to/absolute.md:4:  /path/to/other.md#existing
example_site/path/to/absolute.md:5:  /path/to/other.md#non-existing
example_site/path/to/example.md:3:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md
example_site/path/to/example.md:4:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#existing
example_site/path/to/example.md:5:  other.md
example_site/path/to/example.md:6:  non-existing.md
example_site/path/to/example.md:7:  other.md#existing
example_site/path/to/example.md:8:  other.md#non-existing
example_site/path/to/example.md:9:  #heading
example_site/path/to/example.md:10:  #non-existing
example_site/path/to/example.md:11:  #heading-with-code
example_site/path/to/example.md:12:  #HEADING
example_site/path/to/filename with spaces.md:1:  #
example_site/path/to/follow.md:2:  http://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md
example_site/path/to/follow.md:3:  http://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#non-existing
example_site/path/to/fragment.md:2:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#existing
example_site/path/to/fragment.md:3:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#non-existing
example_site/path/to/other.md:2:  example.md
example_site/path/to/transform.md:2:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/non-existing.md
example_site/path/to/transform.md:3:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/only-on-example-branch.md

Note: In case your paths contain spaces you may need the find -print0 and xargs -0 options.

Absolute local links

By default linky doesn't resolve absolute local links. This way you're able to know which links do redirect. Also, linky doesn't know the location of the document root.

$ linky --check example_site/path/to/absolute.md
example_site/path/to/absolute.md:2: ABSOLUTE /path/to/other.md
example_site/path/to/absolute.md:3: ABSOLUTE /path/to/non-existing.md
example_site/path/to/absolute.md:4: ABSOLUTE /path/to/other.md#existing
example_site/path/to/absolute.md:5: ABSOLUTE /path/to/other.md#non-existing

If you specify the document root using the --root option linky proceeds with the resolution relative to that directory:

$ linky --check --root=example_site example_site/path/to/absolute.md
example_site/path/to/absolute.md:2: OK /path/to/other.md
example_site/path/to/absolute.md:3: NO_DOC /path/to/non-existing.md
example_site/path/to/absolute.md:4: OK /path/to/other.md#existing
example_site/path/to/absolute.md:5: NO_FRAG /path/to/other.md#non-existing

HTTP redirects

By default linky doesn't follow HTTP redirects. This way you're able to know which links do redirect.

$ linky --check example_site/path/to/follow.md
example_site/path/to/follow.md:2: HTTP_301 http://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md
example_site/path/to/follow.md:3: HTTP_301 http://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#non-existing

Enable the --follow option to make linky proceed with the resolution across HTTP redirects:

$ linky --check --follow example_site/path/to/follow.md
example_site/path/to/follow.md:2: OK http://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md
example_site/path/to/follow.md:3: NO_FRAG http://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#non-existing

URI fragment identifiers

Sometimes when Markdown headings are converted into HTML id attributes a prefix is added to the id attribute. E.g. Github adds a "user-content-" prefix.

First, let's just try out the example file without specifying a prefix:

$ linky --check example_site/path/to/fragment.md
example_site/path/to/fragment.md:2: NO_FRAG https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#existing
example_site/path/to/fragment.md:3: NO_FRAG https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#non-existing

Now, let's try adding that prefix:

$ linky --check --prefix='user-content-' example_site/path/to/fragment.md
example_site/path/to/fragment.md:2: PREFIXED https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#existing
example_site/path/to/fragment.md:3: NO_FRAG https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#non-existing

Transforming links before resolution

If you, for example, want to check links against a development version of a sister site you can pipe your links through sed to transform the base URL.

First, let's just extract all links from the example file:

$ linky example_site/path/to/transform.md
example_site/path/to/transform.md:2:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/non-existing.md
example_site/path/to/transform.md:3:  https://github.com/mattias-p/linky/blob/master/example_site/path/to/only-on-example-branch.md

Use sed to edit the links so they point to the sister site:

$ linky example_site/path/to/transform.md | sed 's,/master/,/example/,'
example_site/path/to/transform.md:2:  https://github.com/mattias-p/linky/blob/example/example_site/path/to/non-existing.md
example_site/path/to/transform.md:3:  https://github.com/mattias-p/linky/blob/example/example_site/path/to/only-on-example-branch.md

Note: You may need to be careful with your sed expressions so you don't inadvertently transform the path prefixes.

Finally, pipe the edited linky output into another linky process that actually checks the links:

$ linky example_site/path/to/transform.md | sed 's,/master/,/example/,' | linky --check
example_site/path/to/transform.md:2: HTTP_404 https://github.com/mattias-p/linky/blob/example/example_site/path/to/non-existing.md
example_site/path/to/transform.md:3: OK https://github.com/mattias-p/linky/blob/example/example_site/path/to/only-on-example-branch.md

Resolution speed

Linky uses a thread pool for link resolution. The size of the thread pool affects throughput, but the output itself remains unaffected.

You set the size of the thread pool using the RAYON_NUM_THREADS environment variable:

$ env RAYON_NUM_THREADS=16 linky --check example_site/path/to/example.md

The default size of the thead pool is equal to the number of logical CPU cores available to the process. You may be able to achieve much higher throughputs with a larger than default thread pool. Use the time command to benchmark a suitable size.

Resolution details

In case you ever wonder why a certain link resolved to whatever status token it got, set the RUST_LOG environment variable to warn.

$ env RUST_LOG=warn linky --check example_site/path/to/example.md
example_site/path/to/example.md:3: OK https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md
 WARN  linky > Fragment not found
 WARN  linky >   context: link = https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md
 WARN  linky >   context: fragment = #existing
example_site/path/to/example.md:4: NO_FRAG https://github.com/mattias-p/linky/blob/master/example_site/path/to/other.md#existing
example_site/path/to/example.md:5: OK other.md
 WARN  linky > Document not found
 WARN  linky >   context: link = /tmp/linky/example_site/path/to/non-existing.md
 WARN  linky >   caused by: No such file or directory (os error 2)
example_site/path/to/example.md:6: NO_DOC non-existing.md
example_site/path/to/example.md:7: OK other.md#existing
 WARN  linky > Fragment not found
 WARN  linky >   context: link = /tmp/linky/example_site/path/to/other.md
 WARN  linky >   context: fragment = #non-existing
example_site/path/to/example.md:8: NO_FRAG other.md#non-existing
example_site/path/to/example.md:9: OK #heading
 WARN  linky > Fragment not found
 WARN  linky >   context: link = /tmp/linky/example_site/path/to/example.md
 WARN  linky >   context: fragment = #non-existing
example_site/path/to/example.md:10: NO_FRAG #non-existing
example_site/path/to/example.md:11: OK #heading-with-code
 WARN  linky > Fragment not found case-sensitively
 WARN  linky >   context: link = /tmp/linky/example_site/path/to/example.md
 WARN  linky >   context: fragment = #HEADING
 WARN  linky >   context: anchor = #heading
example_site/path/to/example.md:12: CASE_FRAG #HEADING

Link resolution

Local links are resolved to readable ordinary files and directories in the local filesystem. HTTP(S) links are resolved using GET requests to 200-responses, optionally following redirects. Target documents are read and decoded into character strings.

For HTTP(S) links fragments are resolved to HTML anchors. For local links fragments are resolved to Markdown headings. Fragment resolution is attempted first without any prefix and then, if that fails, with each of the prefixes, if any were provided.

Issues

In case you experience an issue using linky, it would be most helpful if you provide a test Markdown document and the verbose output of the execution.

For example:

$ env RUST_LOG=debug RUST_BACKTRACE=1 linky --check  test.md 2&> linky_err.log

Note: RUST_LOG controls the logging verbosity. RUST_BACKTRACE controls the printing of the stack trace on panic.

Simply drag-and-drop the resulting linky_err.log file into the issue editor of Github. Unfortunately, as of February 2018, Github does not allow to drag-and-drop of Markdown files (*.md). You can either:

• rename your test.md file to test.md.txt

• use a third party paste service (e.g. Hastebin)

• if the file is not large, inline it into the issue inside code blocks:

  ´´´Markdown

  your test markdown here ...

  ´´´

License

Copyright 2017-2019 Mattias Päivärinta

Licensed under the Apache License, Version 2.0 (the "License"); you may not use any of the files in this distribution except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.