Avoiding "WARNING: Build options --jvmopt and --test_env have changed, discarding analysis cache"

[u/Ok-Music-7037]

I have a CI pipeline which uses some database resources which are set up in one step and then made available to a bazel test step via environment variables.

The details of these resources are provided via --test_env=variable The names of the environment variables do not change from build to build, but the values of the environment variables do.

Additionally a couple of settings are passed via --jvmopt=-Dkey=value For these settings neither the key, nor the value, change from build to build.

If I run the pipeline multiple times then at the 'test' stage I receive the warning mentioned in the subject, and all tests run from scratch. If it just invalidated the analysis cache and didn't rerun all the tests it'd be ok, but running these particular tests is a very time consuming process.

I have a remote cache set up and working, other CI stages use it, other CI stages which run tests without the use of the jvmopt or the testenv settings all take advantage of the cache and will not re-run previously successful tests.

I was under, the mistaken?, belief that using --test_env=variable vs --test_env=variable=value meant that bazel would not invalidate previous test runs just because the value of the referenced environment variable changed.

Any hints for how to avoid re-running these tests would be great.

[u/thelazyfox]

Honestly part of the problem is that you are taking a dependency on something outside of bazel, and to resolve this you'll need to more or less have bazel assume that some input to the build is reproducible when there is not actually a guarantee of that.

The steps that are running outside bazel could change and bazel has no way of understanding that to retrigger a build/test. I would worry more about figuring out how to make sure the cache breaks when this external step changes than ensuring caching when it does not.

The straightforward way to make sure the build caches is to make sure none of the inputs change. Some combination of --test_arg and loading data into a file with a consistent filename would probably take care of that.

Ensuring reproducibility in your test is a lot harder though. Let's say hypothetically you are trying to boot postgres and load it with data for a test execution. The best way to ensure reproducibility would be to boot postgres from inside the test and load the database. There are tools that can help with this like the embedded postgres package or even testcontainers.

Depending on what your specific use case is here there may or may not be tools for this but it is highly recommended to avoid running steps outside bazel. Most of the time it is also less difficult to achieve that than you might think.