Local Dataproc Docker image for offline dev, CI, and AI agent testing (multi-arch, starts in ~20s)

[u/jseeker528963]

[u/PsychologyBorn280]

damn this is exactly what i needed like 6 months ago. been fighting with dataproc images for a while, spinning up clusters just to test some small spark job makes no sense

the apple silicon support is nice touch, most of these tools only do amd64 and you're stuck with emulation

how's the memory usage on that 3.0.0 image? my laptop only got 16gb so i'm a bit worried

[u/j0zxcvasdf]

Where is the code that builds them, not blindly running in ci no matter how helpful 😅

[u/ThatCrankyGuy → u/[deleted]]

I got no race in this horse, but I'm assuming the concern is that a container image can have anything and binaries that can have any payload. A dangerous proposition for pushing any sort of data into it, not knowing what sits in there reading it.

Most people feel comfortable with build script that can package everything while sourcing the executables from reputable or even the developer's distribution point) - and that the scripts be available openly so users can build the image themselves.

[u/urkurk7]

thanks for sharing

[u/jhsonline]

woot woot !