package manager

[andrewrk]

Latest Proposal

---

Zig needs to make it so that people can effortlessly and confidently depend on each other's code.

Depends on #89

[phase]

My thoughts on Package Managers:

• Packages should be immutable in the package repository (so the NPM problem doesn't arise).

• Making a new release with a nice changelog should be simple. Maybe integration for reading releases from GitHub or other popular source hosts?

• Packages should only depend on packages that are in the package repository (no GitHub urls like Go uses).

• Private instances of the package repository should be supported from the get go (I believe this was a problem for Rust or some other newer language).

---

Thoughts that might not be great for Zig:

• Enforcing Semver is an interesting concept, and projects like Elm have done it.

[andrewchambers]

This is a good reference for avoiding the complexity of package managers like cargo, minimal version selection is a unique approach that avoids lockfiles, .modverify avoids deps being changed out from under you.

https://research.swtch.com/vgo

The features around verifiable builds and library verification are also really neat. Also around staged upgrades of libraries and allowing multiple major versions of the same package to be in a program at once.

[bnoordhuis]

Packages should be immutable in the package repository (so the NPM problem doesn't arise).

I assume you mean authors can't unpublish without admin intervention. True immutability conflicts with the hoster's legal responsibilities in most jurisdictions.

minimal version selection

I'd wait a few years to see how that pans out for Go.

[andrewchambers]

Note that by minimal, they mean minimal that the authors said was okay. i.e. the version they actually tested. The author of the root module is always free to increase the minimum. It is just that the minimum isn't some arbitrary thing that changes over time when other people make releases.

[BraedonWooding]

My top three things are;

• No lockfiles
• KISS, I don't want to fight with the package manager, also should fully be integrated into build.zig no external programs.
• Allow a file to be on each dependency to allow that dependency to go and download its own dependencies, this file should also maintain a change-log.

A good package manager can break/make a language, one of the reasons why Go has ditched atleast one of its official package managers and completely redid it (it may even be two, I haven't kept up to date with that scene).

[andrewrk]

The first thing I'm going to explore is a decentralized solution. For example, this is what package dependencies might look like:

const Builder = @import("std").build.Builder;
const builtin = @import("builtin");

pub fn build(b: &Builder) void {
    const mode = b.standardReleaseOptions();

    var exe = b.addExecutable("tetris", "src/main.zig");
    exe.setBuildMode(mode);

    exe.addGitPackage("clap", "https://github.com/Hejsil/zig-clap",
        "0.2.0", "76c50794004b5300a620ed71ef58e4444455fd72e7f7e8f70b7d930a040210ff");
    exe.addUrlPackage("png", "http://example.com/zig-png.tar.gz",
        "00e27a29ead4267e3de8111fcaa59b132d0533cdfdbdddf4b0604279acbcf4f4");

    b.default_step.dependOn(&exe.step);
}

Here we provide a mapping of a name and a way for zig to download or otherwise acquire the source files of the package to depend on.

Since the build system is declarative, zig can run it and query the set of build artifacts and their dependencies, and then fetch them in parallel.

Dependencies are even stricter than version locking - they are source-locked. In both examples we provide a SHA-256 hash, so that even a compromised third party provider cannot compromise your build.

When you depend on a package, you trust it. It will run zig build on the dependency to recursively find all of its dependencies, and so on. However, by providing a hash, you trust only the version you intend to; if the author updates the code and you want the updates, you will have to update the hash and potentially the URL.

Running zig build on dependencies is desirable because it provides a package the ability to query the system, depend on installed system libraries, and potentially run the C/C++ compiler. This would allow us to create Zig package wrappers for C projects, such as ffmpeg. You would even potentially use this feature for a purely C project - a build tool that downloads and builds dependencies for you.

and potentially run the C/C++ compiler
cmd/go: arbitrary code execution during “go get” #23672

although you might argue

Dependencies are even stricter than version locking - they are source-locked. In both examples we provide a SHA-256 hash, so that even a compromised third party provider cannot compromise your build.

in that case you'd have to check all the reps of all your reps recursively (manually?) on each shape change though to be really sure

[andrewrk]

in that case you'd have to check all the reps of all your reps recursively (manually?) on each shape change though to be really sure

This is already true about all software dependencies.

[BraedonWooding]

I've been considering how one could do this for the past few days, here is what I generally came up with (this is based off @andrewrk 's idea), I've kept out hashes to make it easier, I'm more talking about architecture then implementation details here;

• No lock files, purely source driven
• Have a central repository of all downloaded files (like under /usr/local/zig-vendor and have a built in to access them like @vImport("BraedonWooding/ZigJSON"), or have a unique vendor location for each 'zig' build file or rather each project, in which case we autogenerate a nice index.zig file for you to access like const vendor = @import("vendor/index.zig"); const json = vendor.ZigJSON.
• Utilise them during building, we can go and download any new dependencies.
• Automatically download the latest dependency as per the user requirements that is either (for structure x.y.z);
  • Keep major version i.e. upgrade y and z but not x
  • Keep minor and major i.e. upgrade all 'z' changes.
  • Keep version explicitly stated
  • Always upgrade to latest

This would also solve the issue of security fixes as most users would keep the second option which is intended for small bug fixes that don't introduce any new things, whereas the major version is for breaking changes and the minor is for new changes that are typically non-breaking.

Your build file would have something like this in your 'build' function;

...
builder.addDependency(builder.Dependency.Git, "github.laiyagushi.com.au", "BraedonWooding", "ZigJSON", builder.Versions.NonMajor);
// Or maybe
builder.addDependency(builder.Dependency.Git, "github.laiyagushi.com.au/BraedonWooding/ZigJSON", builder.Versions.NonMajor);
// Or just
builder.addGitDependency("github.laiyagushi.com.au/BraedonWooding/ZigJSON", builder.Versions.NonMajor);
...

Keeping in mind that svn and mercurial (as well as plenty more) are also used quite a bit :). We could either use just a folder system of naming to detect what we have downloaded, or have a simple file storing information about all the files downloaded (note: NOT a lock file, just a file with information on what things have been downloaded). Would use tags to determine versions but could also have a simple central repository of versions linking to locations like I believe what other things have.

[isaachier]

How would you handle multiple definitions of the same function? I find this to be the most difficult part of C/C++ package management. Or does Zig use some sort of package name prefixing?

[BraedonWooding]

@isaachier Well you can't have multiple definitions of a function in Zig, function overloads aren't a thing (intended).

You would import a package like;

const Json = @Import("JSON/index.zig");

fn main() void {
    Json.parse(...);
    // And whatever
}

When you 'include' things in your source Zig file they are exist under a variable kinda like a namespace (but simpler), this means that you should generally never run into multiple definitions :). If you want to 'use' an import like using in C++ you can do something like use Json; which will let you use the contents without having to refer to Json for example in the above example it would just be parse(...) instead of Json.parse(...) if you used use, you still can't use private functions however.

If for some reason you 'use' two 'libraries' that have a dual function definition you'll get an error and will most likely have to put one under a namespace/variable, very rarely should you use use :).

[isaachier]

I don't expect a clash in the language necessarily, but in the linker aren't there duplicate definitions for parse if multiple packages define it? Or is it automatically made into Json_parse?