Using packages in air-gapped environments

In some cases, Pkl evaluation needs to occur in environments that do not have internet access. At the same time, these evaluations can use packages that are published to the internet. To enable using such packages, Pkl provides three approaches: HTTP proxying, mirroring, and vendoring.

HTTP proxying

Pkl can connect through an HTTP CONNECT proxy server when downloading packages (and making any other HTTP requests).

For users of the CLI, the HTTP proxy can either be configured on the OS user level, (in ~/.pkl/settings.pkl), on the project level (in PklProject), or using the --http-proxy and --http-no-proxy CLI flags.

  • OS user

  • project

~/.pkl/settings.pkl
amends "pkl:settings"

http {
  proxy {
    address = "http://my.proxy.address"
    noProxy {
      // don't proxy if the host is `localhost`
      "localhost"
      // don't proxy if the IP address starts with `127.0`
      "127.0.0.0/16"
    }
  }
}
PklProject
amends "pkl:Project"

evaluatorSettings {
  http {
    proxy {
      address = "http://my.proxy.address"
      noProxy {
        // don't proxy if the host is `localhost`
        "localhost"
        // don't proxy if the IP address starts with `127.0`
        "127.0.0.0/16"
      }
    }
  }
}

Those that use Pkl as a library in a host language have similarly named options when building the evaluator.

To read more about proxying, see the documentation.

Mirror server

It’s also possible to set up a server that mirrors the assets available from the internet.

To connect to a mirror server, the HTTP rewrites setting is used. Like proxying, this can be configured on either the OS user level, on the project level, or using the --http-rewrite CLI flag.

For example, this configures rewrites for packages from pkg.pkl-lang.org:

  • OS user

  • project

~/.pkl/settings.pkl
amends "pkl:settings"

http {
  rewrites {
    ["https://pkg.pkl-lang.org/"] = "https://my.mirror/pkg.pkl-lang.org/"
    // github rewrites are also required in order to fetch package ZIP assets.
    ["https://github.com/"] = "https://my.mirror/github.com/"
    ["https://www.github.com/"] = "https://my.mirror/github.com/"
  }
}
PklProject
amends "pkl:Project"

evaluatorSettings {
  http {
    rewrites {
      ["https://pkg.pkl-lang.org/"] = "https://my.mirror/pkg.pkl-lang.org/"
      // github rewrites are also required in order to fetch package ZIP assets.
      ["https://github.com/"] = "https://my.mirror/github.com/"
      ["https://www.github.com/"] = "https://my.mirror/github.com/"
    }
  }
}

Beneath the hood, the pkg.pkl-lang.org service simply redirects to GitHub. So, a mirror server for such packages only needs to mirror GitHub assets. Here is a simple nginx configuration that handles mirroring given the above settings.

nginx.conf
server {
  proxy_set_header Host "github.com";
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_redirect off;

  location ~ ^/pkg\.pkl-lang\.org/github\.com/(?<org>[^/]+)/(?<repo>[^/]+)/(?<asset>.*)$ {
    proxy_pass https://github.com/$org/$repo/releases/download/$asset/$asset;
  }

  location ~ ^/pkg\.pkl-lang\.org/(?<repo>[^/]+)/(?<asset>.*)$ {
    proxy_pass https://github.com/apple/$repo/releases/download/$asset/$asset;
  }

  location ~ ^/github\.com/(?<path>.*)$ {
    proxy_pass https://github.com/$path;
  }
}

Note that pkg.pkl-lang.org is not a straight-up prefix replacement. Therefore, the server needs to mirror according to the same redirect scheme. Effectively, the two redirects are:

/:repo/:pkg                 -> https://github.com/apple/:repo/releases/download/:pkg/:pkg
/github.com/:org/:repo/:pkg -> https://github.com/:org/:repo/releases/download/:pkg/:pkg

Vendoring dependencies

Strictly speaking, Pkl does not have a feature for vendoring dependencies. However, it provides derivative tools to determine the coordinates of dependencies, to download these dependencies to a set path, and to tell Pkl to load packages from there.

Downloading dependencies

When using projects, the PklProject.deps.json file describes all the dependencies of the project. Using the jq, xargs, and curl tools, these dependencies can be downloaded to a vendor directory.

Here is a sample script:

#!/usr/bin/env bash

VENDOR_DIRECTORY=vendor

# Download all project dependencies based on the generated `PklProject.deps.json` file
cat PklProject.deps.json \
  | jq -r '.resolvedDependencies[] | select(.type == "remote") | .uri[7:] + "::sha256:" + .checksums.sha256' \
  | xargs -I {} pkl download-package --no-transitive {} --cache-dir "$VENDOR_DIRECTORY"

Handling directly imported packages

Sometimes, Pkl code is not evaluated within the context of a PklProject. In these cases, modules within packages are imported directly using the full package URI (e.g. import "package://…​" instead of import "@myDep/…​").

Without a project, there is no PklProject.deps.json. So, the pkl analyze imports command should be used instead.

It’s uncommon and generally not advised to import packages directly if there is a PklProject present. Instead, these should be imported as project dependencies.
#!/usr/bin/env bash

VENDOR_DIRECTORY=vendor

pkl analyze imports -f json config.pkl \
  | jq -r '[.resolvedImports[] | split("#/")[0] | select(startswith("package://"))] | unique | .[]' \
  | xargs -I {} pkl download-package {} --cache-dir "$VENDOR_DIRECTORY"

Configuring the cache directory

For those using the Pkl CLI, the most straightforward way to configure the cache directory is through the --cache-dir flag.

pkl eval --cache-dir vendor myModule.pkl

For those using projects, the cache directory can be set in the PklProject file. This means that you do not need to specify the --cache-dir flag during evaluation.

PklProject
amends "pkl:Project"

evaluatorSettings {
  moduleCacheDir = "vendor"
}

Those that use Pkl as a library within a host language have similarly named settings when constructing the evaluator.