Using the Community Chart Testing Tools Yourself

Tue, Sep 25, 2018

The Helm community charts, available as the stable and incubator repositories, have long had testing. That testing has grown and improved a significant amount in the past year; from Helm linting and testing if an application runs in a cluster to now include YAML linting, some validation on maintainers, Chart.yaml schema validation, tests on chart version increments, and more.

These testing tools are useful for more than the community charts. They could be used in development workflows, in other testing systems, and for private charts. To make the testing more accessible we (mostly Reinhard Nägele) refactored the tools into a container image that can be run outside of the community charts testing infrastructure.

This new image is now available as the Chart Testing project. This project is built and maintained by the Helm Charts Maintainers, powers the community chart testing process, and is being used elsewhere.

Example: Locally on Mac

One of the easiest ways to take a look at it is to try it out locally. To aid with that, one of the examples provided by the project shows you how to use it with Docker for Mac with the charts repository. An easy way to try it out is to make a change to a chart and run the following command from the root of the charts directory:

$ /path/to/chart-testing/examples/docker-for-mac/my_test.sh

To illustrate this I added a tag in the Chart.yaml file of the mariadb chart without incrementing the chart version. Running the test produced the following output:

Cluster "docker-for-desktop-cluster" set.
Cluster "docker-for-desktop-cluster" set.
Switched to context "docker-for-desktop".

--------------------------------------------------------------------------------
Environment:
REMOTE=k8s
TARGET_BRANCH=master
CHART_DIRS=stable incubator
EXCLUDED_CHARTS=common
CHART_REPOS=incubator=https://kubernetes-charts-incubator.storage.googleapis.com/
TIMEOUT=600
LINT_CONF=/testing/etc/lintconf.yaml
CHART_YAML_SCHEMA=/testing/etc/chart_schema.yaml
VALIDATE_MAINTAINERS=true
GITHUB_INSTANCE=https://github.com
CHECK_VERSION_INCREMENT=true
--------------------------------------------------------------------------------

Charts to be installed and tested: stable/mariadb
Initializing Helm client...
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!
"incubator" has been added to your repositories

--------------------------------------------------------------------------------
Processing chart 'stable/mariadb'...
--------------------------------------------------------------------------------

Validating chart 'stable/mariadb'...
Checking chart 'stable/mariadb' for a version bump...
Chart version on k8s/master : 5.0.3
New chart version:  5.0.3
ERROR: Chart version not ok. Needs a version bump.
Linting 'stable/mariadb/Chart.yaml'...
Linting 'stable/mariadb/values.yaml'...
Validating Chart.yaml
Validating /workdir/stable/mariadb/Chart.yaml...
Validation success! 👍
Validating maintainers
Verifying maintainer 'bitnami-bot'...
ERROR: Chart validation failed.
Building dependencies for chart 'stable/mariadb'...
No requirements found in stable/mariadb/charts.
Chart does not provide test values. Using defaults...
Linting chart 'stable/mariadb'...
==> Linting stable/mariadb
Lint OK

1 chart(s) linted, no failures
--------------------------------------------------------------------------------
✖︎ stable/mariadb
--------------------------------------------------------------------------------

You’ll notice the chart failed to pass testing because the version was not incremented.

Configurable

While the testing image contains defaults, it is configurable so it can be used without any association to the community charts setup. The configuration is handled via environment variables which are documented in the README.md file.

For example, if you wanted to skip checking for a version increment on the chart for every change you can set CHECK_VERSION_INCREMENT to false. This will skip that check and is useful for cases where every change to a chart is not released.

Example: Using It with CircleCI

Linting, without trying to operate the chart, is easy to incorporate into a workflow. The following is a simple example CircleCI configuration to do so:

version: 2
jobs:
  lint-charts:
    docker:
      - image: quay.io/helmpack/chart-testing:v1.1.0
    steps:
      - checkout
      - run:
          name: lint
          command: |
            chart_test.sh --config .testenv --no-install
workflows:
  version: 2
  lint:
    jobs:
      - lint-charts

In this case the environment variables for the configuration are stored in a file name .testenv. This file holds the environment variables and is sourced into the environment. The following is an example from the community charts:

# The name of the Git remote
REMOTE=k8s

# The name of the Git target branch
TARGET_BRANCH=master

# Chart directories separated by a space
CHART_DIRS=(
    stable
    incubator
)

# Charts that should be skipped
EXCLUDED_CHARTS=(
    common
)

# Additional chart repos to add (<name>=<url>), separated by a space
CHART_REPOS=(
    incubator=https://kubernetes-charts-incubator.storage.googleapis.com/
)

TIMEOUT=600

Try It in Your Workflow

This toolchain, wrapped in a container image, is meant to be used in a wide variety of workflows. Please take it for a spin, give it a try, use it in your workflows, and provide feedback.

Matt Farina