# How to diff rendered manifests per environment before you merge > Render each overlay from a pull request and from its base, diff every environment with dyff, and post the result as one comment before anything merges. Source: https://voxgig.com/howto/diff-rendered-manifests-between-environments-in-ci - Audience: platform-team - Level: intermediate - Languages: yaml, bash - Verified: 2026-10-21 - Published: 2026-10-20 ## Short answer Render every overlay with kustomize build twice, from the pull request's merge commit and from the base it merges into. Compare each environment's pair with dyff between, and post the result as one pull-request comment. Keep kubectl diff for the deploy step, because an environment not yet deployed diffs as whole objects, defaulted fields included. --- ## You will need Kustomize overlays for two or more environments, a CI system that can comment on pull requests, and Node 22 or later to run the `npm ci` that fetches the tools. Verified 2026-10-21 against Node 22.22.0, Kustomize 5.8.3, dyff 1.12.0, and `kubectl`, `kube-apiserver`, and `etcd` from envtest 1.37.0. `npm ci` downloads each tool from its GitHub release into `node_modules/.bin` and checks it against a hash pinned in `install-tools.sh`. The API server and `etcd` come from the [envtest](https://book.kubebuilder.io/reference/envtest) bundle that controller tests use, and only the cluster diff needs them. The sample keeps the two checkouts as plain directories, `main/` and `pr/`, so it runs without a repository. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [dyff](https://github.com/homeport/dyff) | Two rendered files and no cluster, for a comment people read | A second binary to pin, and no view of the cluster, so drift stays out of the comment | You need to know what an apply would change in the cluster | | [helm diff plugin](https://github.com/databus23/helm-diff) | Helm releases, compared with what the cluster has deployed | Helm only, a plugin on every runner, and cluster credentials in the CI job | The manifests come from Kustomize, or CI must not reach the cluster | | [kubectl diff](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_diff/) | The deploy step, run against the cluster it is about to change | Cluster credentials in CI, and whole objects with defaulted fields for anything not yet deployed | Pull requests from forks, or a comment a reviewer must read quickly | `kubectl diff` answers the deploy step's question, what this apply changes in this cluster, at the price of credentials and server-side detail nobody wrote. The helm diff plugin answers the same question for Helm releases, with the same credentials. dyff compares two files, so it needs nothing but the renders and cannot see drift in the cluster. Use dyff in the pull-request check and `kubectl diff` in the deploy step. ## Render both sides of the pull request The check needs two renders of every environment: the pull request as it would merge, and the base it merges into. `render.sh` builds each overlay of one checkout into its own file: ```bash title="render.sh" set -euo pipefail export PATH="$(cd "$(dirname "$0")" && pwd)/node_modules/.bin:$PATH" checkout=$1 out=$2 mkdir -p "$out" for overlay in "$checkout"/overlays/*/; do env=$(basename "$overlay") kustomize build "$overlay" > "$out/$env.yaml" done ``` [`kustomize build`](https://kubectl.docs.kubernetes.io/references/kustomize/cmd/build/) prints the overlay as the cluster would receive it. For a Helm chart, run [`helm template`](https://helm.sh/docs/helm/helm_template/) with each environment's values file instead, and the rest of the check stays the same. In CI, the pull request's side is the merge commit GitHub builds for the run, and the base side is the first parent of that commit.[^1] The diff then shows what merging would change, not what the branch changed since it forked. ## Diff each environment and write the comment `diff-envs.sh` renders both checkouts, compares each environment's pair with `dyff between`, and prints a Markdown comment.[^2] An environment that exists on one side only is compared with an empty file, so a new overlay shows as documents added. dyff exits 0 for no difference, 1 for differences, and 255 for an error, so only an exit code greater than 1 stops the script: ```bash title="diff-envs.sh" envs=$(cd "$work" && ls base head | grep '\.yaml$' | sed 's/\.yaml$//' | sort -u) for env in $envs; do touch "$work/base/$env.yaml" "$work/head/$env.yaml" set +e dyff between --omit-header --color off --output github --set-exit-code \ "$work/base/$env.yaml" "$work/head/$env.yaml" > "$work/$env.diff" status=$? set -e if [ "$status" -gt 1 ]; then echo "dyff failed on $env" >&2 exit 1 fi done ``` dyff matches documents by their Kubernetes identity and list entries by name, so a change reads as a path and two values rather than as shifted lines. Run it on the pull request that turns on debug logging: ```bash bash diff-envs.sh main pr ``` ```text output ### Rendered manifests: this pull request against its base | Environment | Changes | | --- | --- | | production | 1 | | staging | 1 | #### production ~~~diff @@ spec.template.spec.containers.checkout.env.LOG_LEVEL.value @@ # apps/v1/Deployment/production/checkout ! ± value change - info + debug ~~~ #### staging ~~~diff @@ spec.template.spec.containers.checkout.env.LOG_LEVEL.value @@ # apps/v1/Deployment/staging/checkout ! ± value change - info + debug ~~~ ``` The table is the review. `production | 1` says the change meant for staging reaches production, because the edit sits in the shared base. Moving the variable into the staging overlay leaves the production render as it was. ## Post the comment on every push The workflow builds the two checkouts from real commits and posts the comment with the GitHub CLI: ```yaml title="manifest-diff.yml" # The merge commit GitHub builds for the pull request. Its first parent # is the commit of the base branch it was merged onto. - uses: actions/checkout@v6 with: path: pr fetch-depth: 2 - name: Check out the base beside it run: git -C pr worktree add ../main HEAD^1 - name: Fetch the pinned kustomize and dyff working-directory: pr/ci env: SKIP_ENVTEST: 1 run: npm ci --no-audit --no-fund - name: Render and diff every environment run: bash pr/ci/diff-envs.sh main pr > comment.md # One comment per pull request, edited on every push. - name: Post the comment env: GH_TOKEN: ${{ github.token }} PR: ${{ github.event.pull_request.number }} run: gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --edit-last --create-if-none --body-file comment.md ``` `--edit-last` with `--create-if-none`, both documented in the [`gh pr comment` manual](https://cli.github.com/manual/gh_pr_comment), keeps one comment per pull request instead of one per push. It edits the newest comment by the token's user, and every workflow's `GITHUB_TOKEN` posts as the same bot, so this holds only while no other workflow comments that way. Two quick pushes start two runs, and the comment ends up holding whichever render finished last. A `concurrency` group at the top of the workflow cancels the older run: ```yaml title="manifest-diff.yml" concurrency: group: manifest-diff-${{ github.event.pull_request.number }} cancel-in-progress: true ``` `SKIP_ENVTEST=1` leaves out the API server, which only the cluster diff below needs. The job declares `pull-requests: write`, and GitHub grants it only to pull requests from branches in the same repository. ## Check it worked A pull request that changes a production-only value must produce a comment showing that change and nothing else. `check.sh` builds that pull request from a copy of `main/`, with production's replicas raised from three to four, then checks two more cases and the workflow's `concurrency` setting: ```bash bash check.sh ``` ```text output ok 1 - a production-only change shows one change in production ok 2 - and none in staging ok 3 - and names the field and both values, line for line ok 4 - an unchanged tree shows no changes anywhere ok 5 - a base edit shows up in every environment ok 6 - the workflow cancels the run for an older push 6 of 6 checks passed ``` Check 3 compares the production section line for line, from `@@ spec.replicas @@` to `- 3` and `+ 4`, so a comment that grows unrelated lines fails it. ## When it goes wrong `kubectl diff` against a cluster looks like the natural check, but on its first run it buries the change. `cluster-diff.sh` starts a throwaway API server, sets dyff as kubectl's external differ, and diffs production before anything is deployed: ```bash bash cluster-diff.sh fresh ``` ```text output spec: progressDeadlineSeconds: 600 replicas: 3 revisionHistoryLimit: 10 selector: matchLabels: app: checkout strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 25% template: metadata: labels: app: checkout spec: containers: - name: checkout image: "registry.example.com/checkout:1.4.2" imagePullPolicy: IfNotPresent terminationMessagePath: /dev/termination-log terminationMessagePolicy: File env: - name: LOG_LEVEL value: info ports: - containerPort: 8080 protocol: TCP resources: limits: memory: 512Mi dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 status: {} ``` The namespace holds nothing yet, so `kubectl diff` prints both objects whole. Sixteen of the Deployment's spec lines are fields no manifest set, from `progressDeadlineSeconds` to `terminationGracePeriodSeconds`, filled in by the API server's defaults.[^3] Diff renders against each other in the pull request, and keep the cluster diff for the deploy step, where the objects exist: ```bash bash cluster-diff.sh deployed ``` ```text output service/checkout created deployment.apps/checkout created kubectl diff -f the pull request's production render metadata.generation (apps/v1/Deployment/production/checkout) ± value change - 1 + 2 spec.template.spec.containers.checkout.env.LOG_LEVEL.value (apps/v1/Deployment/production/checkout) ± value change - info + debug kubectl diff exited 1 ``` With production applied, the same pull request diffs as the one real change, plus `metadata.generation`, a counter the server raises on every spec change. The comment step also fails on a pull request from a fork, because GitHub gives a fork's run a [read-only token](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflows-in-forked-repositories). Upload `comment.md` as an artifact, and post it from a second workflow triggered by `workflow_run`, which never runs the fork's code, as [GitHub Security Lab describes](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/). ## When not to do this Do not run `kubectl diff` against production from the pull-request job. It needs production credentials in a job that runs the branch's own scripts, and a run from a fork cannot have them. Do not switch the trigger to `pull_request_target` to get a write token for forks while checking out the pull request. That runs the fork's scripts with the base repository's permissions. Do not read a clean render diff as a clean deploy. The renders cannot see drift in the cluster, such as a manual `kubectl edit` or replicas that a `HorizontalPodAutoscaler` owns. The `kubectl diff` in the deploy step is the check for that. Do not build this for one environment with no shared base. The pull request's own diff already shows every line that reaches the cluster. ## Related how-tos - [Merge nested config objects with predictable precedence](/howto/merge-nested-config-objects-with-precedence) - [Explain why a unified config has a given value](/howto/explain-why-a-unified-config-value) ## Last verified Verified 2026-10-21 against Node 22.22.0, Kustomize 5.8.3, dyff 1.12.0, and `kubectl`, `kube-apiserver`, and `etcd` from envtest 1.37.0. Every output block is what the command preceding it printed. The cluster is a local API server with no controllers or nodes, started by `cluster-diff.sh` and deleted when it exits. The GitHub Actions workflow was not run: `main/` and `pr/` stand in for its two checkouts, and the comment it would post is the output of `diff-envs.sh`. The helm diff plugin was not run, because the sample renders with Kustomize. [^1]: [GitHub's documentation](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request) sets `GITHUB_REF` to `refs/pull//merge` for a `pull_request` run. The page gives the reason in a sentence of its own: tests "run against the merged result" instead of the branch alone. The commit the check renders as the pull request's side is a trial merge that no branch contains and nobody wrote. [^2]: The project's [README](https://github.com/homeport/dyff) spells the name with a Greek delta and two florin signs, and gives its pronunciation in the International Phonetic Alphabet as /ˈdʏf/. The line under the title is plainer: "A diff tool for YAML files, and sometimes JSON." [^3]: The two at the top of the list each get a sentence in the [Deployment documentation](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/). Ten old `ReplicaSet` objects are kept for rollback by default, and the documentation adds that the ideal number depends on how often and how reliably you deploy. A rollout that stops making progress is reported as stalled once `progressDeadlineSeconds` runs out, and on that field the documentation is brief: "This defaults to 600."