Testing in Pkl

by Dan Chao on April 9th, 2024

Pkl files are programs that get evaluated to produce a result. Like any other program, it can be useful to have tests that verify their business logic. To address this, Pkl provides tooling for writing and running tests.

Writing Tests

Tests in Pkl are modules that amend standard library module pkl.test.

These modules can describe facts, and examples.

Facts

Facts are boolean assertions. They are useful for unit tests, where the behavior of functions, types, or any expression can be tested.

If an expression evaluates to false, the test case is considered failing.

Below is a naïve test about the qualities of the integer 1, where the last expression fails.

math.pkl

amends "pkl:test"

facts {
  ["one"] {
    1.isOdd
    1.isBetween(0, 3)
    1 == 2
  }
}

Running this test provides a report about the failing test.

$ pkl test math.pkl
module test
  facts
    ✘ one
       1 == 2 (file:///math.pkl, line 7)

0.0% tests pass [1/1 failed], 66.7% asserts pass [1/3 failed]

facts in Pkl are most useful for writing fine-grained assertions about specific behavior of your code.

The tests for module pkl.experimental.uri are a good example of facts in practice. There are multiple facts, where each fact contains multiple assertions which cover different cases of business logic:

URI.pkl#L27-L36

facts {
  ["encode"] {
    URI.encode("https://example.com/some path") == "https://example.com/some%20path"
    URI.encode(alphaLower) == alphaLower
    URI.encode(alphaUpper) == alphaUpper
    URI.encode(nums) == nums

    local safeChars = "!#$&'()*+,-./:;=?@_~"
    URI.encode(safeChars) == safeChars
    URI.encode("\u{ffff}") == "%EF%BF%BF"
    URI.encode("🏀") == "%F0%9F%8F%80"
  }

Examples

Examples are values that get compared to expected values. When first testing an example, its value gets written to a sibling file that ends in pkl-expected.pcf. The next time the test is run, the example value is then asserted to be equal to that expected value. If they are not equal, the test fails.

Here is a quick tutorial for writing an example. First, we create a test module with our example. In this case, we are checking the behavior of an imagined Person class. For simplicity’s sake, we are just declaring it as a local class.

person.pkl

amends "pkl:test"

local class Person {
  firstName: String

  lastName: String

  fullName: String = "\(firstName) \(lastName)"
}

examples {
  ["person"] {
    new Person {
      firstName = "Johnny"
      lastName = "Appleseed"
    }
  }
}

This test then gets run:

pkl test person.pkl

On the first run, Pkl tells us that the corresponding expected value was written.

module test
  examples
    ✍️ person

1 examples written

This creates a file called person.pkl-expected.pcf in the same directory. The directory tree now looks like this:

.
├── person.pkl
└── person.pkl-expected.pcf

We can inspect the contents of the pkl-expected.pcf to see what the expected value is.

person.pkl-expected.pcf

examples {
  ["person"] {
    new {
      firstName = "Johnny"
      lastName = "Appleseed"
      fullName = "Johnny Appleseed"
    }
  }
}

Running pkl test person.pkl again now shows that it passes.

module test
  examples
    ✔ person

100.0% tests pass [1 passed], 100.0% asserts pass [1 passed]

Now, we’ll change "Johnny" to "Sally", to demonstrate a test failure.

 examples {
   ["person"] {
     new Person {
-      firstName = "Johnny"
+      firstName = "Sally"
       lastName = "Appleseed"
     }
   }
 }

Running pkl test person.pkl again fails.

module test
  examples
    ✘ person
       #0: (file:///test.pkl, line 13)
         Expected: (file:///test.pkl-expected.pcf, line 3)
         new {
           firstName = "Johnny"
           lastName = "Appleseed"
           fullName = "Johnny Appleseed"
         }
         Actual: (file:///test.pkl-actual.pcf, line 3)
         new {
           firstName = "Sally"
           lastName = "Appleseed"
           fullName = "Sally Appleseed"
         }

0.0% tests pass [1/1 failed], 0.0% asserts pass [1/1 failed]

Because the test failed, Pkl writes a new file to the same directory. The directory tree now looks like this:

.
├── person.pkl
├── person.pkl-actual.pcf
└── person.pkl-expected.pcf

It can be especially useful to use diff to compare the pkl-actual.pcf file with the pkl-expected.pcf file.

diff -c person.pkl-actual.pcf person.pkl-expected.pcf

--- person.pkl-actual.pcf    2024-03-28 15:33:15
+++ person.pkl-expected.pcf  2024-03-28 15:15:00
@@ -1,9 +1,9 @@
 examples {
   ["person"] {
     new {
-      firstName = "Sally"
+      firstName = "Johnny"
       lastName = "Appleseed"
-      fullName = "Sally Appleseed"
+      fullName = "Johnny Appleseed"
     }
   }
 }

For intentional changes, add the --overwrite flag. This will overwrite the expected output file, and also remove the pkl-actual.pcf file.

$ pkl test person.pkl --overwrite
module test
  examples
    ✍️ person

1 examples written

Pattern: Testing JSON, YAML and other module output

A common pattern is to use examples to test how Pkl renders into static configuration.

Pkl is idiomatically split between schema and data. Base Pkl modules define schema and rendering logic (colloquially called templates), and downstream modules amend those base modules with just data.

Here is an imagined Pkl template for configuring a logger. It defines some converters for DataSize and Duration, and also sets the output renderer to YAML.

Logger.pkl

module Logger

/// The list of targets to send log output to.
targets: Listing<LogTarget>

abstract class LogTarget {
  /// The logging level to write at.
  logLevel: "info"|"warn"|"error"
}

class RotatingFileTarget extends LogTarget {
  /// The max file size
  maxSize: DataSize?

  /// The directory to write log lines to.
  directory: String
}

class NetworkLogTarget extends LogTarget {
  /// The network URL to send log lines to.
  connectionString: Uri

  /// The timeout before the connection gets killed.
  timeout: Duration?
}

output {
  renderer = new YamlRenderer {
    converters {
      [DataSize] = (it) -> "\(it.unit)\(it.value)"
      [Duration] = (it) -> it.isoString
    }
  }
}

After having written this template, we’d like to test to see what our YAML output actually looks like. We’d also like to provide some sample code for our users, that demonstrate how to use our template.

To do this, we’ll first create some examples modules, in an examples/ directory.

examples/rotatingLogger.pkl

amends "../Logger.pkl"

targets {
  new RotatingFileTarget {
    maxSize = 5.mb
    directory = "/vat/etc/log"
  }
}

examples/networkLogger.pkl

amends "../Logger.pkl"

targets {
  new NetworkLogTarget {
    timeout = 5.s
    connectionString = "https://example.com/foo/bar"
  }
}

With this set up, we can now use them as test examples.

tests/Logger.pkl

module tests.Logger

amends "pkl:test"

import* "../examples/*.pkl" as allExamples

examples {
  for (key in allExamples.keys) { (1)
    [key.drop("../examples/".length).replaceLast("pkl", "yml")] { (2)
      allExamples[key].output.text
    }
  }
}

1	Iterates over the keys only as a workaround for a current bug where for-generators are eager in values. This ensures that any errors related to loading the module are captured as related to that specific example.
2	Sets the test name to `<filename>.yml`

These tests are defined as evaluating each module’s output.text property. This emulates the behavior of the Pkl CLI when it evaluates a module through pkl eval.

Furthermore, the tests uses a glob import to bulk-import these example modules. This means that if we add new modules to the examples/ directory, they are automatically added as a new test.

Running this test creates expected output:

$ pkl test tests/Logger.pkl
module tests.Logger
  examples
    ✍️ networkLogger.yml
    ✍️ rotatingLogger.yml

2 examples written

Reporting

By default, Pkl writes a simple test report to console. Optionally, it can also produce JUnit-style reports by setting the --junit-reports flag.

Example:

pkl test --junit-reports .out

Interaction with `pkl:Project`

In Pkl, a project is a directory of Pkl modules that is tied together with the presence of a PklProject file.

There are many reasons for wanting to define a project. One reason is to simplify the pkl test command. If pkl test is run without any input source modules, it will run all tests defined in the PklProject.

PklProject

amends "pkl:Project"

tests {
  ...import*("tests/**.pkl").keys
}

A note on `apiTests`

When creating a package, it is also possible to specify apiTests. These are tests for the external API of this package. The intention of this is to allow checking for breaking changes when updating a version. When publishing a new version of the same package, running apiTests of a previous package can inform whether the package’s major version needs to be bumped or not.

The apiTests are also run when the package is created via pkl project package.

Future improvements: power assertions

Testing in Pkl is already tremendously useful. With that said, there is still room improvements.

One feature that we would like to implement is power assert style reporting. Power assertions are a form of reporting that displays a diagram that shows parts of the syntax tree, and their resolved values.

A power-assertion report might look like:

module test
  facts
    ✘ math
        num1 == num2
         |   |   |
         | false |
         1       2

This feature improves the developer experience when testing facts. Currently, only the expression is printed, as well as a report that the test failed.

In lieu of this feature, trace() expressions can be used to add more debugging output.

Frameworks that provide power assertions include Spock, power-assert-js, and swift-power-assert.