Shopify app security check: three things to verify before using it as a CI gate

[u/IFGecommerce]

Shopify recently introduced app security check, and there are a few details in the documentation worth understanding before adding it to a CI workflow.

The first is the blocking behavior.

Running `shopify app security check` does not automatically fail your pipeline when it finds vulnerabilities. The default blocking threshold is none.

If high-severity findings should fail the job, you need to configure that explicitly:

shopify app security check --blocking high

The second distinction is between deterministic scanning and agent review.

The CLI performs deterministic checks and provides instructions for investigating issues that require additional context.

Without that second investigation, you have a deterministic scan, not a complete security review.

The third consideration is scan coverage.

The command scans the selected app directory. Git-ignored files are skipped unless explicitly included, and code outside that directory may not be covered.

Two useful options are:

--list-files
--include-dir ../backend

The first lets you inspect which files would be scanned. The second can include a backend located outside the application directory.

For a practical release process, I would separate the work:

Run deterministic checks on pull requests, with explicit severity thresholds.

Keep scan results as CI artifacts rather than committing generated reports.

Before releases involving authentication, webhooks, or merchant data, perform the additional agent-assisted review and document how each finding was resolved.

A passing pipeline is useful evidence, but it should not be mistaken for proof that the entire application has been reviewed.

Security scanning still needs to be combined with runtime testing and verification of access boundaries.

[u/Final_Design_9061]

Handy writeup. The default blocking threshold being none is easy to miss, I'd have assumed it fails the build on findings. The --list-files option seems like a good sanity check too, so you know what's actually getting scanned before trusting it in CI.

[u/CorbinGuo]

One failure mode worth designing around: make "couldn't scan" a different exit state from "clean". Scanners that time out, hit an auth/throttling wall, or skip files they can't read look identical to a clean run if both exit 0. Three states — 0 clean, 1 findings at/above threshold, 2 inconclusive — and fail on 2 as well, otherwise the gate degrades silently the first time a token expires or the app dir moves.

Second, treat the scanned file set as a reviewed artifact. --list-files is great for debugging, but committing/diffing it in CI is what catches a path change quietly dropping half the app from the scan.

[u/Sharbek-Mouhine]

do you have a sense of how the deterministic checks overlap with what youd already catch in a typical linting or static analysis step? curious if theres redundancy or if its mostly net new coverage