peckish

peckish (case-sensitive) is a tool for repackaging Linux software artifacts.

For example, suppose you're an application developer. You just made something cool and want to distribute it. However, packaging is hard. Different package formats do things differently -- ex. Arch has x86_64 and any as architectures, but Debian has over a dozen and calls "x86_64" "amd64" -- and it's hard to remember all the specifics. This is compounded by having to figure out the appropriate CLI flags for each package format. How many people can write a valid tar command on the first try? :P

This problem is fairly common:

It seems more and more common, at least in the self-hosted community, to provide two methods of installations: Docker or a shell script installer. It seems that many new projects don’t even bother with a native package manager, such as dpkg or rpm, which makes me reluctant to use them.

https://yotam.net/posts/the-audacity-of-piping-curl-to-bash/

It's easier to compile the linux kernel from scratch than it is to build a debian package from scratch.

It's mind-blowing how complicated it is; at the end of the day it's just an archive. I hate having to do it.

https://news.ycombinator.com/item?id=34787228

The impact of having frictionless package building cannot be understated. I'm publishing Arch Linux packages for all my applications because it takes just a few minutes to write up a PKGBUILD. Then one time, I tried providing a Debian package as well, but I gave up after several hours of trying to get through all the bureaucracy of the tooling.

https://news.ycombinator.com/item?id=34923960

[...] I've created my own RPM and DEB packages in the past as well; but, at least when I did it years ago, it wasn't as effective as a PKGBUILD on arch.

https://news.ycombinator.com/item?id=35014171

Official documentation is a problem. It is aimed at Debian maintainers building a fully policy compliant package that could be accepted to the official repository. You don't need any of that crap for making a .deb package. It puts off people from creating Debian packages for their own programs or private use.

https://news.ycombinator.com/item?id=34788350

If my package manager had an Oh My Zsh package

This is the author missing the point. The reason curl | bash is common is because devs don't like packaging for every distro under the sun, and MacOS, and FreeBSD, and... If you really think curl | bash is the problem, then you should be lining up to package the stuff you use for your distro. Instead, it is always someone else's problem.

Package managers are great... for the user. For everyone else, a polyglot system, with arcane technical policies, and even more arcane human policies is... not ideal.

https://news.ycombinator.com/item?id=33832401

peckish aims to solve this. Instead of mangling your files with various arcane tools, spinning up Docker containers or VMs to try to build packages and make sure they install or are even valid, and all the other pains, you can just write a basic YAML file, and you're done! peckish does the rest, without shelling out to distro-specific tools!

Additionally, peckish makes repackaging software easier. You can take a DEB and convert it into an RPM, or a Docker image to flat files on the filesystem, or turn a tarball into an installable package for DEB-based, RPM-based, or Arch-based distros.

peckish lets you convert back and forth between all of these formats:

• flat files on the filesystem
• a tarball
• an installable Debian package
• an installable Arch Linux package
• a runnable Docker image
• an installable RPM package
• an ext4 image (wip)

peckish is based around the concept of "artifacts" and "producers." An artifact is some metadata about a package that exists on your system, and a producer is something that takes in an artifact and produces a new artifact. For example, taking a tarball and producing a DEB or RPM package.

• peckish concepts
• artifact and producer docs

features

• convert between package formats
  • flat files
  • tarballs
  • .deb packages
  • Arch Linux packages
  • Docker images
  • .rpm packages
  • ext4 filesystem images
• GitHub Actions support
• library support
• (WIP) reproducible builds support
• statically-linked release binaries

caveats

• peckish is 0.x.y! Treat it accordingly. Maybe don't run it in prod.
• Documentation is hard, and I am not the best at it. Some questions may only be able to be answered by reading the source or opening an issue. Reporting missing and/or broken docs helps everyone!
• peckish is not a build system or a package manager. It doesn't care about how your code is built or installed, just about getting it from one package format to another.
• peckish does not guarantee that files are placed in the correct locations in the package, just that the package is valid. For example, if you convert a DEB to an RPM, you'll need to make sure that the files in the RPM are installed to the correct locations.
• peckish cannot guarantee all dependencies are placed into the package correctly. It's up to you to make sure your package is statically linked, or that all dependencies are included in the package.
• peckish is primarily tested for my use-cases. Please please please please report bugs!
• Packages produced with peckish almost certainly are not of sufficient quality to be accepted into a distro's official repositories. PLEASE don't waste the time of distro maintainers by submitting a bunch of peckish-generated packages.

community

peckish is a part of the amyware discord server.

If you like what I make, consider supporting me on Patreon:

usage

Create a peckish.yaml file in the root of your project.

# whether to chain outputs, ie each artifact output is the input to the next
# producer. defaults to `false` if not specified.
chain: false

# metadata about the package. required, even if you're only producing a file or
# a tarball. this is because it's just easier than trying to play the "is there
# enough metadata to build the package" game.
metadata:
  name: "my-cool-pkg"
  # many distros want versions that end in -#, which is a revision number for
  # the package. this is required if you are producing an Arch/Deb/RPM/similar
  # package.
  version: "0.1.0-1"
  description: "a package"
  # suggested format: "me <me@example.com>"
  author: "me"
  # the architecture of the system the package is built for. this is usually
  # the same as the architecture of the system you're building on. will be
  # automatically set to the correct value for the target package format, ex.
  # x86_64 -> amd64 for debian.
  arch: "amd64"
  license: "Apache-2.0"

# the artifact being used as input to the pipeline.
input:
  name: "some file"
  type: "file"
  paths:
    - "./path/to/file"

# the producers being used as outputs. see `docs/` for more info about each
# producer.
output:
  - name: "tarball"
    type: "tarball"
    path: "./whatever.tar"

  - name: "debian package"
    type: "deb"
    path: "./whatever.deb"
    # a list of changes to inject into the filesystem. this lets you move,
    # copy, symlink, etc. files and directories within the artifact before it's
    # written to disk. see `docs/injections.md` for more info.
    injections:
      - "move-file"
      - "cleanup"

# the actual injections that are applied to output artifacts. these are
# specified in their own group to allow for reuse between multiple producers.
injections:
  move-file:
    type: "move"
    src: "/path/to/file"
    dest: "/new/path/to/file"
  cleanup:
    type: "delete"
    path: "/path"

suggested use-cases

• Package your software for more distros with less pain
• Extract packages without having to memorise arcane CLI flags
• Create Docker images without a Dockerfile
• Make a package for one distro installable on others, without having to repackage by hand
• Programmatically create/manipulate packages in Rust

library

crates.io: https://crates.io/crates/peckish

MSRV 1.71.

// artifacts
use peckish::prelude::builder::*;
use peckish::prelude::*;

let file_artifact = FileArtifactBuilder::new("example file artifact".into())
    .add_path("./examples/a".into())
    .build()?;

let tarball_producer = TarballProducerBuilder::new("example tarball producer".into())
    .path("test.tar.gz".into())
    .build()?;

let tarball_artifact = tarball_producer.produce(&file_artifact).await?;

// pipelines
use peckish::prelude::pipeline::*;
use peckish::prelude::*;

let file_artifact = ...;

let tarball_producer = ...;

let debian_producer = ...;

let config = PeckishConfig {
    input: ConfiguredArtifact::File(file_artifact),
    output: vec![
        ConfiguredProducer::Tarball(tarball_producer),
        ConfiguredProducer::Deb(debian_producer),
    ],
    chain: false,
};

let pipeline = Pipeline::new();
let out = pipeline.run(config).await?;
println!("produced {} artifacts", out.len());

GitHub Actions

GitHub Actions for peckish can be found at queer/actions.

- name: "install peckish!"
  uses: "queer/actions/peckish_install@mistress"
  with:
    token: "${{ secrets.GITHUB_TOKEN }}"
- name: "run peckish!"
  uses: "queer/actions/peckish_run@mistress"

reproducible builds

peckish tries to respect SOURCE_DATE_EPOCH. Please open issues or PRs if you find places where it doesn't!

roadmap

package/artifact formats

The goal/hope is to be able to support all of these. Formats not listed may end up on the list in the future. Formats that are not currently supported may never be supported.

• flat files "file"
• arch package "arch"
• tarball "tarball"
• debian package "deb"
• docker image "docker"
• rpm package "rpm"
• Filesystem and VM images "ext4"
  • WIP
• OCI images "oci"
  • WIP
• appimage "appimage"
  • squashfs: https://crates.io/crates/backhand
  • unpacker: ???
• Flatpak?
  • More research needed

other

• compression handled transparently
• [WIP] SOURCE_DATE_EPOCH support for reproducible builds
• better docs
• GitHub Actions support for ease-of-use
  • https://github.com/queer/actions
• better error messages
• maybe someday hit 1.0.0?

concepts

peckish is built around the concepts of artifacts and producers.

Artifacts are some sort of data that exists on your system that can be packaged; artifacts themselves do not contain any of that data, just metadata. For example, a FileArtifact is a list of paths to files on your system. A TarballArtifact is a path to a tarball. A DebArtifact is a path to a .deb file. So on and so forth.

Producers are a bit more interesting. Producers are the things that actually do the packaging: they take an artifact as input and produce a new artifact as output. For example, a TarballProducer may take a FileArtifact as input and produce a TarballArtifact as output, a DebProducer may take a TarballArtifact as input and produce a DebArtifact as output, and so on.

peckish artifacts and producers are centred around the idea of an in-memory filesystem. Rather than having to mangle things on the disk, peckish moves everything into memory, manipulates it, then flushes it back to disk. This allows for trivial manipulation of software artifacts, as changing them is simply injecting some changes into the in-memory filesystem and repackaging with the metadata in the producer. No knowledge of the previous artifact is needed beyond its in-memory filesystem representation.

misc

why is it called peckish?

If you pretend really hard, "peckish" sounds kinda sorta a bit like "package."

related libraries

• floppy-disk: async filesystem facade
• disk-drive: multi-floppy-disk utils
• flop: floppy-disk archive facade
• smoosh: automagic async (re)compression
• nyoom: filesystem walker for floppy-disk
• flail: floppy-disk facade for ext4

license

Copyright 2023-present amy null

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.