Skip to content

5.3. CI/CD Workflows

What is CI/CD?

CI/CD is a cornerstone of modern software development that combines Continuous Integration and Continuous Delivery or Deployment. It automates the process of integrating code from multiple contributors, testing it, and preparing it for release.

  • Continuous Integration (CI) is the practice of frequently merging all developers' code changes into a central repository. After each merge, an automated build and a series of automated tests are run to detect integration issues early.
  • Continuous Delivery (CD) extends CI by automatically deploying all code changes to a testing and/or production environment after the build stage.
  • Continuous Deployment is a step further, where every change that passes all stages of the pipeline is automatically released to customers.

The primary goal is to make software development faster, more reliable, and less error-prone by automating the entire release process.

What is a CI/CD workflow?

A CI/CD workflow, often called a pipeline, is the automated sequence of steps that takes code from a developer's machine to the production environment. This pipeline typically includes stages for building the application, running a comprehensive suite of automated tests (unit, integration, security), and deploying the application. By automating this path, teams can release new features and fixes to users with speed and confidence.

Why are CI/CD workflows essential for MLOps?

In MLOps, CI/CD workflows are critical for managing the complexity of machine learning systems. They provide several key benefits:

  • Ensure Code and Model Quality: CI/CD acts as a gatekeeper, enforcing quality standards for both code and models. By running automated checks for code style, typing, security, and test coverage, it prevents regressions and maintains a healthy codebase.
  • Automate Repetitive Tasks: Workflows automate tedious but crucial tasks like dependency installation, testing, packaging, and publishing. This frees up AI/ML engineers to focus on higher-value activities like model development and performance tuning.
  • Enhance Reproducibility: By codifying the build, test, and deployment process, CI/CD ensures that every version of your ML system is built and deployed in a consistent, reproducible manner. This is vital for tracking experiments and complying with regulatory requirements.
  • Improve Collaboration and Visibility: Centralized workflows provide a clear, shared understanding of the project's health. They generate reports on code quality, test results, and deployment status, making it easier for team members to collaborate and maintain high standards.

Which CI/CD solution should you use?

While many CI/CD solutions exist, GitHub Actions is a powerful and convenient choice for projects hosted on GitHub. It is deeply integrated with the GitHub platform, allowing you to build, test, and deploy your code directly from your repository.

To create a workflow, you define a YAML file in the .github/workflows directory of your project. This file specifies the triggers (e.g., a pull request), the jobs to run, and the individual steps within each job.

What are the essential workflows for an MLOps project?

For a typical MLOps project, you should establish three workflows: one for verification on every change, one for publication on every release, and one scheduled deep security scan.

Continuous Integration Workflow

This workflow runs on every pull request (and every push to main) to ensure that code changes meet quality standards before being merged. Notice how little of it is CI-specific: the entire verification logic is one call to the all task defined in mise.toml, the same task you run in your terminal.

name: CI
on:
  push:
    branches: [main]
  pull_request:
permissions:
  contents: read
concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
  checks: # name kept as "checks" to satisfy the repository's required-status-check ruleset
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    steps:
      - name: Checkout repository
        uses: actions/checkout@v7
        with:
          persist-credentials: false # no step pushes back to the repository
      - name: Install mise toolchain
        uses: jdx/mise-action@v4
        with:
          cache: true
      - name: Run canonical gate
        run: mise run all
      - name: Verify no changes
        run: test -z "$(git status --porcelain)"

Workflow Breakdown:

  • name: The workflow's name, "CI," as it appears in the GitHub UI.
  • on: Triggers the workflow on any pull request and on pushes to main.
  • permissions: Grants the workflow read-only access by default, following the principle of least privilege.
  • concurrency: Ensures that only one run of this workflow per branch is active at a time. If a new commit is pushed to a branch, the previous run is canceled (but runs on main are never interrupted).
  • jobs.checks: The single verification job. Its name is load-bearing—see below.
    • runs-on: ubuntu-24.04: A pinned runner image rather than ubuntu-latest. latest moves under you: when GitHub promotes a new image, a build that passed yesterday can fail today for reasons no commit explains. Pinning makes the upgrade a deliberate, reviewable one-line change.
    • timeout-minutes: 20: A hung job—a test waiting on a socket, a scanner stuck on a network call—otherwise burns runner minutes until GitHub's six-hour default kills it. A tight timeout turns a hang into a fast, obvious failure.
    • actions/checkout@v7 with persist-credentials: false: By default, checkout leaves a credential in .git/config for the rest of the job. No step here pushes back to the repository, so leaving that token available only widens the blast radius if any tool the job runs is compromised.
    • jdx/mise-action@v4 with cache: true: Installs the toolchain pinned in mise.toml and mise.lock, and caches the downloaded binaries keyed on the lockfile. This single step replaces separate "setup Python" and "install uv" actions—uv is pinned here, and uv in turn provisions the interpreter named in .python-version.
    • mise run all: The whole gate—format, check, test, build—in one step.
    • test -z "$(git status --porcelain)": Fails the build if the gate modified or created anything.

Why one mise run all step instead of four?

An earlier version of this workflow spelled out its own sequence: mise run format, then mise run check, then mise run test. It looked complete. It was missing mise run build, and had been for a long time. Nothing failed, nothing warned—the packaging step simply never ran on any pull request.

That is the failure mode of a duplicated pipeline. It does not break loudly when it drifts from the project's real definition of "passing"; it just quietly does less. Calling a single named task removes the possibility: adding a step to all adds it to CI, to your terminal, and to every contributor's pre-push run at the same instant, with no YAML to remember.

Why test -z "$(git status --porcelain)" and not git diff --exit-code?

Both commands are meant to answer the same question—did running the gate change the working tree?—and if it did, the change belongs in the commit, not in CI.

But git diff only reports modifications to tracked files. A task that creates a new file leaves git diff perfectly empty. This is not hypothetical: moving the vulnerability scanner to uv run pip-audit --skip-editable --cache-dir .cache/pip-audit made it write a cache directory nobody had added to .gitignore, and git diff --exit-code passed happily on a checkout that had grown an untracked directory.

git status --porcelain lists tracked modifications and untracked files, so test -z on its output fails on both. It is the stricter, more honest question: "is this checkout still exactly what was committed?"

Why is the job named checks?

Branch protection—whether through the classic branch protection rules or a repository ruleset—requires a status check by name. That name is a plain string matched against the job name GitHub reports, and nothing validates that the string corresponds to a job that exists.

Get it wrong and the failure is spectacularly quiet: the ruleset waits forever for a status check named checks while the workflow dutifully reports one named check, and every pull request is blocked with "expected — waiting for status to be reported". No error, no log, no hint. If you codify your ruleset in a JSON file (.github/rulesets/main.json) and install it with a task, keep the job name and the required context in sync, and verify on a throwaway pull request that the check actually appears.

Continuous Deployment Workflow

This workflow is triggered when a new release is published. It handles building and publishing the project artifacts—the documentation site and a Docker image. The documentation deploys through the official GitHub Pages Actions, which upload an artifact and deploy it to the github-pages environment. This is the modern, recommended flow and replaces the legacy approach of pushing to a gh-pages branch.

name: CD
on:
  release:
    types: [published]
permissions:
  contents: read
jobs:
  pages:
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    permissions:
      contents: read
      pages: write
      id-token: write
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v7
        with:
          persist-credentials: false # no step pushes back to the repository
      - name: Install mise toolchain
        uses: jdx/mise-action@v4
        with:
          cache: false # a poisoned tool cache must never reach published artifacts
      - name: Build API documentation
        run: mise run docs
      - name: Configure Pages
        uses: actions/configure-pages@v6
      - name: Upload Pages artifact
        uses: actions/upload-pages-artifact@v5
        with:
          path: docs/
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v5
  image:
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    permissions:
      contents: read
      packages: write
    steps:
      - name: Checkout repository
        uses: actions/checkout@v7
        with:
          persist-credentials: false # the registry login below carries its own token
      - name: Set lower-case image path
        # GitHub expands `${{ }}` before the shell runs, so repository references reach
        # the script through `env:` rather than being interpolated into the command.
        env:
          REPOSITORY: ${{ github.repository }}
        run: echo "IMAGE=$(echo "ghcr.io/$REPOSITORY" | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_ENV"
      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4
      - name: Build and push image
        uses: docker/build-push-action@v7
        with:
          context: .
          push: true
          tags: |
            ${{ env.IMAGE }}:latest
            ${{ env.IMAGE }}:${{ github.ref_name }}

Workflow Breakdown:

  • on: Triggers the workflow when a release is published.
  • jobs.pages: Builds the documentation with mise run docs and deploys it with the official Pages pipeline.
    • permissions: Grants contents: read, pages: write, and id-token: write, which the Pages deployment requires.
    • environment: github-pages: Deploys into the protected github-pages environment and exposes the published URL.
    • actions/configure-pages → actions/upload-pages-artifact → actions/deploy-pages: The three official steps that configure Pages, package the docs/ folder as an artifact, and deploy it—no gh-pages branch involved.
  • jobs.image: Builds and publishes the Docker image.
    • permissions: Grants contents: read and packages: write, allowing it to publish to the GitHub Container Registry (ghcr.io).
    • env: REPOSITORY: The repository name reaches the shell through an environment variable instead of being interpolated into the command string—see the script-injection note below.
    • docker/login-action: Logs into the container registry using the automatically provided GITHUB_TOKEN.
    • docker/build-push-action: Builds the image, tags it with latest and the release version, and pushes it. Using a container ensures a consistent, portable environment for running the ML model.

A job's permissions block replaces the workflow's—it does not merge

This is the single most common way a hardened workflow accidentally breaks itself. The workflow above declares permissions: contents: read at the top level, and the image job needs to push a package, so it adds packages: write.

If the job had written only:

permissions:
  packages: write

it would have packages: write and nothing else—no contents: read at all. A job-level permissions block is a full replacement of the workflow-level one, not an addition to it. The job would then fail at actions/checkout, and the error message ("could not read from remote repository") points nowhere near the YAML that caused it.

The habit that avoids this: whenever you add a job-level permissions block, restate every scope the job needs, including the read scopes the workflow already granted. Both jobs above spell out contents: read for exactly that reason.

Why cache: false on release jobs but cache: true on CI?

jdx/mise-action can cache the tool binaries it downloads, and on CI that is a straightforward win: the cache is keyed on mise.lock, so a run that changes no tool reuses the previous download.

On a release job, the trade-off inverts. GitHub Actions caches are writable by any workflow run in the repository, including runs triggered from a pull request branch. A cache entry poisoned by one of those runs would be restored here—inside the job that signs, builds, and publishes what your users install. Rebuilding the toolchain from its pinned, checksummed sources costs a minute; letting attacker-controlled bytes into a published container image costs far more. The comment in the YAML records the reasoning, because the next person to read that line will otherwise "optimize" it back.

This is not a hypothetical worry you have to spot by eye—a workflow auditor flags it, as the next section shows.

Scheduled Security Workflow

The CI workflow scans what you just changed. It cannot scan what you removed a year ago.

Push and pull-request CI runs on a shallow checkout, and the secret scanner inside mise run check is deliberately bounded (gitleaks git --log-opts="--max-count=100") to keep the gate fast. Both choices are right for a gate and wrong for an audit: a credential that was committed in March and deleted in April still sits in the object database, still works, and will never appear in either view again.

The answer is a separate, scheduled workflow that trades speed for depth:

name: Security
# Push CI scans only the latest commits, so a secret committed earlier and later
# removed would never be seen again. This is the full-history counterpart: same
# pinned scanners, whole history, no reports to parse — a finding fails the job.
on:
  schedule:
    - cron: "17 3 * * 1"
  workflow_dispatch:
permissions:
  contents: read
concurrency:
  group: security
  cancel-in-progress: true
jobs:
  scan:
    name: Full-history scan
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    steps:
      - name: Checkout complete history
        uses: actions/checkout@v7
        with:
          fetch-depth: 0
          persist-credentials: false # scanners only read the checkout
      - name: Install mise system
        uses: jdx/mise-action@v4
      - name: Scan complete Git history
        run: gitleaks git --redact=100 --verbose
      - name: Scan full checkout
        run: trivy fs .

Workflow Breakdown:

  • on.schedule: A weekly cron (17 3 * * 1—Mondays at 03:17 UTC). The odd minute is intentional: GitHub delays workflows scheduled on the hour, when everyone else's crons fire.
  • on.workflow_dispatch: Lets you run the audit on demand from the Actions tab, which is how you confirm it works without waiting a week.
  • fetch-depth: 0: Clones the complete history. This one line is the entire reason the workflow exists—without it, the deep scan sees no more than CI does.
  • --redact=100: Redacts the matched secret in the log output. Findings appear in a public run log; a scanner that prints the credential it found has published it a second time.
  • trivy fs .: Re-runs the filesystem scan against the whole checkout, unbounded.
  • timeout-minutes: 30: Deep scans are slow, but they are not unbounded—a full-history scan that has run for half an hour is stuck, not thorough.
  • runs-on: ubuntu-24.04: The same pinned image as CI. Pinning matters most for a job whose result blocks a merge, and a scheduled audit could tolerate a moving image—but a scanner that changes underneath you turns "a new finding appeared" into a question about the runner rather than about your code.

The reason this does not belong in push CI is simple arithmetic. A full-history clone plus a full-history scan takes minutes and grows with the age of the repository, and it would run on every commit of every pull request to discover the same thing it discovered yesterday. Findings in old history are not caused by your pull request and cannot be fixed by it. Put deep, slow, whole-repository audits on a schedule; keep the pull-request gate fast enough that nobody wants to bypass it.

How do you lint the workflows themselves?

Everything above is code, and it is code that runs with your repository's credentials—yet it is usually the only code in the project that no linter, type checker, or test ever inspects. A typo in a step name is harmless; a typo in an if: expression can silently make a security step never run.

Two tools cover the two distinct failure classes, and the check:actions task runs both:

[tasks."check:actions"]
description = "Lint and audit GitHub Actions workflows (actionlint + zizmor)"
run = ["actionlint", "zizmor --offline .github/workflows/"]
  • actionlint checks correctness. It validates the workflow schema, checks that expression syntax and context references are real (github.evnt.name is caught, not silently empty), verifies runner labels exist, and even runs shellcheck over your run: blocks—so an unquoted shell variable in a workflow is flagged just as it would be in a script.
  • zizmor checks security. It audits for the attack patterns specific to CI: template injection, over-broad permissions, credential persistence, cache poisoning, and unpinned actions.

Both are standalone binaries, so they are pinned in mise.toml alongside the rest of the toolchain and run offline as part of mise run check—no network, no API token, no separate CI job.

What does zizmor actually catch?

Two findings from the workflows above are worth walking through, because neither is obvious by inspection.

Template injection. The image job originally built its lower-case image name like this:

run: echo "IMAGE=$(echo "ghcr.io/${{ github.repository }}" | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_ENV"

GitHub expands ${{ }} by pasting the value into the script text before the shell ever sees it. With a repository name that is safe; with a context an outsider controls—a branch name, an issue title, a pull request body—the pasted text becomes shell code running with your job's token. Passing the value through env: and referring to $REPOSITORY fixes it, because the shell then receives the value as data rather than as source code.

Cache poisoning. Running zizmor against the CD workflow with cache: true on the mise-action step produces an error-level finding:

error[cache-poisoning]: runtime artifacts potentially vulnerable to a cache poisoning attack
  --> .github/workflows/cd.yml:24:9
   |
 2 | / on:
 3 | |   release:
 4 | |     types: [published]
   | |______________________- generally used when publishing artifacts generated at runtime
...
24 |           uses: jdx/mise-action@v4
   |           ^^^^^^^^^^^^^^^^^^^^^^^^ this step
25 | /         with:
26 | |           cache: true
   | |_____________________- enables caching explicitly here

Read what it reasoned about: it connected the workflow's trigger (release: published, a workflow that publishes artifacts) to a step several jobs down that restores a mutable cache, and concluded that attacker-influenced cache contents could reach a published container image. That is a two-part inference across the file that a human reviewer skims straight past—and exactly why cache: false sits on the release jobs.

Why relax zizmor's pinning rule?

Out of the box, zizmor requires every third-party action to be pinned to a full commit SHA, and flags uses: actions/checkout@v7 as unpinned. That is a defensible policy, but it is not the only defensible one, and it is not the one these repositories chose: actions are pinned to major-version tags so that security patches within a major version arrive automatically, at the cost of trusting the tag owner not to move it maliciously.

The wrong response to a rule you disagree with is to disable the tool, or to sprinkle inline suppressions until it goes quiet. The right response is to configure the rule to enforce the policy you actually hold, in .github/zizmor.yml:

# Actions are deliberately pinned to major-version tags: tags track security
# patches within a major, at the cost of trusting the tag. zizmor's default policy
# demands hash-pins; relax it to ref-pins so the audit enforces the actual policy
# instead of fighting it.
rules:
  unpinned-uses:
    config:
      policies:
        "*": ref-pin

With this file, uses: actions/checkout@v7 passes and uses: actions/checkout@main still fails—a floating branch reference is caught, a version tag is accepted. The audit now encodes a decision instead of a default, and the comment explains it to whoever revisits the choice. Note the general principle, which applies to every linter in this course: tune the rule, do not silence the tool.

How can you avoid repeating steps in CI/CD workflows?

The best way to follow the DRY (Don't Repeat Yourself) principle in MLOps is to push logic down into your mise tasks rather than duplicating shell commands across workflows. Both the CI and CD examples above start with the same two lines:

- uses: actions/checkout@v7
- uses: jdx/mise-action@v4

A single jdx/mise-action@v4 step reads your mise.toml and mise.lock and installs the pinned toolchain (uv, dprint, gitleaks, trivy, hadolint, actionlint, zizmor, ...); the tasks it then runs use uv to provision the Python interpreter from .python-version. Because every job then calls high-level tasks like mise run all or mise run docs, there is nothing to re-declare per workflow—the definitions live in one place and are shared with your terminal and git hooks.

For repository-specific step sequences that are not tasks (for example, a bespoke sign-and-attest flow), you can still encapsulate them into a reusable composite action stored under .github/actions, then reference it with - uses: ./.github/actions/<name>. You can also find thousands of pre-built actions on the GitHub Marketplace to integrate with third-party services and streamline your workflows.

How do you keep action versions current?

Pinning actions to major tags only helps if somebody eventually moves the pin. Dependabot does that for you, and it deserves an explicit configuration rather than the defaults. Declare one entry per ecosystem in .github/dependabot.yml—here, GitHub Actions alongside the Python and Docker dependencies of the same project:

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
      day: "monday"
    commit-message:
      prefix: "chore(deps)"
    groups:
      actions:
        patterns: ["*"]
        update-types: ["minor", "patch"]

Three choices are worth copying:

  • groups collapses every minor and patch bump into one pull request per ecosystem. Because update-types deliberately excludes major, a breaking upgrade still arrives on its own, where it gets the review it deserves.
  • schedule.day: monday puts the review in a predictable slot instead of scattering it across the week.
  • commit-message.prefix: "chore(deps)" makes the bot's commits match the convention your changelog filter expects—see the Pre-Commit Hooks chapter for why the default prefix quietly breaks git-cliff.

Note also that the docker ecosystem only tracks base images pinned to a concrete tag. A FROM or COPY --from line referring to :latest is invisible to Dependabot, which is one of the reasons the Dockerfile pins its uv image.

What are some best practices for CI/CD in MLOps?

  • Automate Everything: Automate all manual steps in your ML lifecycle, including data validation, model training, evaluation, and deployment, to reduce human error and increase velocity.
  • Call One Task, Not a List of Steps: Let the workflow run mise run all and keep the definition of "passing" in mise.toml. A hand-written list of CI steps drifts by omission, and nothing tells you when it does.
  • Lint Your Workflows: Run actionlint and zizmor on .github/workflows/ as part of mise run check. Workflow YAML is privileged code; treat it like the rest of your source.
  • Pin the Runner and Bound the Job: Prefer runs-on: ubuntu-24.04 over ubuntu-latest, and give every job a timeout-minutes. Upgrades then happen when you choose, and a hang fails fast instead of burning an hour.
  • Grant the Least Privilege, Job by Job: Default the workflow to permissions: contents: read, add persist-credentials: false to checkout in jobs that never push, and remember that a job-level permissions block replaces the workflow-level one.
  • Manage Secrets Securely: Use encrypted secrets to store sensitive information like API keys, passwords, and cloud credentials. GitHub Actions provides a secure way to manage secrets at the repository or organization level. Never interpolate ${{ }} into a run: script—pass values through env: so the shell treats them as data.
  • Master GitHub Actions Syntax: A deep understanding of the workflow syntax, including contexts, expressions, and triggers, will allow you to build highly dynamic and powerful pipelines.
  • Use Concurrency Strategically: The concurrency key is essential for managing workflow runs efficiently, preventing race conditions, and saving resources by canceling outdated jobs.
  • Leverage the GitHub CLI: Use the gh command-line tool to interact with your workflows, check run status, and trigger them manually (e.g., gh workflow run ...), streamlining your development loop.
  • Implement Branch Protection Rules: Protect your main branch by requiring status checks (like your verification workflow) to pass before pull requests can be merged. This is a critical safeguard for maintaining a stable and deployable project.

Additional Resources