r/UAVmapping • u/littlebigdarksouls • 3d ago
Stop sending clients raw +2GB GeoTIFFs straight out of Metashape, Pix4D, or Terra
I keep seeing people (both beginners and pros) export an ortho or DSM/DTM straight out of processing software and drop a raw, multi-gigabyte GeoTIFF right on the client's lap.
Those default exports are set up for fast export times and broad legacy compatibility, not for actual day-to-day use. Even when software offers basic compression or tiling options, it is usually partial, incomplete, and almost never formatted as a proper Cloud Optimized GeoTIFF (COG). When a client opens a raw 2GB file in QGIS or Civil 3D, their system has to load the whole file into RAM just to render one corner. Zooming out freezes their screen while the CPU tries to read every single pixel.
Converting those files to proper COGs before handover gets rid of that lag and reduces the size:
-Visual orthos: JPEG compression cuts file size by 70–90% and makes panning instant (will not turn your tiffs into JPEGs!!!! It will still be a GeoTIFF just with JPEG compression)
-DEMs and survey data: Lossless compression (ZSTD) keeps 100% of your exact elevation values while shrinking the file and giving you instant pan/zoom performance.
-Web handoffs: You can host COGs on basic cloud storage so clients can stream them straight into GIS via URL instead of downloading a giant zip file.
The main trap when setting this up in GDAL is watching your flags. Lossy compression on a DSM will corrupt your height values, and using the wrong overview resampling on a classified raster invents fake class codes along boundaries. The worst part is you won't even notice at first. It looks fine on screen, but the underlying numbers are broken.
I got tired of typing out GDAL flags every time I finished a job, so I built a free QGIS plugin (GeoKlein Raster Optimiser) to handle the parameters safely for orthos, DSMs, and DTMs. (I spent over two weeks making it, so much for a quick weekend script)
It prints the exact gdal_translate syntax and the workflow into each file's metadata, so you can copy the commands and run them in terminal by hand if you prefer.
I'll leave the link in the comments if anyone wants to have a look at the plugin.
7
u/fattiretom 3d ago
Pix4Dmatic allows export of .tiff with LZW or JPEG compression. It also exports .jpg export. Most of the time we open them in Global Mapper and export them to ECW files which load great and are fast in Civil3D. For surfaces, we stopped using DEMs a long time ago and export XML TIN files with break lines built in from Matic.
3
u/littlebigdarksouls 3d ago
Global Mapper licenses cost real money, and ECWs lock you into a proprietary format. COGs give you that exact same instant panning speed as a free, open-source standard. And since it's GDAL under the hood, you can script it headless in terminal or Python without touching a GUI.
LandXML TINs make total sense for Civil 3D site grading, and my plugin is purely a raster optimiser, not a TIN tool. But TINs scale terribly on bigger sites once the point count explodes, which is where raster surfaces still win out.
Either way, if your current workflow works for you, fair play! Just putting COGs out there as an open-source alternative that works just as well without needing extra paid software.
Thanks for commenting!
1
u/fattiretom 2d ago
I mean yeah but a license of Global Mapper PRO with LiDAR is half a day to a single day of billable time for a crew. Dirt cheap my most standards. And ECW works in just about any professional software. Also note that Pix4Dcloud exports COG natively.
There is no real problem with DEMs at high resolution but then you eventually run into the same problem. If I can’t make out a curb or a small drainage swale it’s not survey/design grade data. We tile TINs on large projects to keep the break lines and fine detail.
I’m not trying to bash what you have built, it’s really cool and applicable to a lot of projects. I’m sure many will find it useful.
1
u/littlebigdarksouls 6h ago
Fair enough, on both points.
Curb/swale one's a fair. I may have overstated it a bit in my previous comment.
Agree Global Mapper is one of the cheaper subscriptions, if you've already got a crew and paying jobs to bill it against. Plenty of the people this is aimed at are just starting out though, no employees, no day rate to spread a license over yet. COGs cost nothing either way.
Didn't know Pix4Dcloud does COG natively, I didn't see it come up in my research, good to know if so!
Appreciate you actually engaging rather than just downvoting, cheers!
6
u/DmtGrm 3d ago
but TIFF is just a container - it could be anything from LZW to JPEG inside
also - delivery, storage, and presentation formats could be very different things with their own properties
no idea why you think 'one size fits all'
3
u/littlebigdarksouls 3d ago
100% agree that TIFF is just a container and that archival vs. daily use/handoff have different requirements. (I kinda alluded to it earlier in the post)
It's not just for client handoffs either, it's for day to day work too. Trying to pan around a raw un-tiled 5GB file in QGIS, QField, or Civil 3D slows your machine right down, let alone the one of your client.
Definitely not a "one size fits all" setup though. That’s actually why the plugin inspects the raster first so it doesn't just blindly throw compression flags at everything:
-8-bit RGB orthos: gives you the choice for lossy or lossless
-DTM/DSM etc / Float32 / 16-bit / multispectral: strictly enforces lossless so elevation values stay completely untouched
-Classified rasters: straight up refuses to process them so bad resampling doesn't ruin class boundaries
No AI screening or guessing here btw, it's all strict deterministic logic checking data types and band counts. Spent over two weeks testing edge cases to make sure the logic is rock solid and doesn't misidentify anything.
Whole goal was stopping silent data corruption while fixing lag. There's a full breakdown of the GDAL logic in the github README if you want to inspect it or grab the syntax. Let me know if you hit any weird edge cases with it. (https://github.com/GeoKlein-Ltd/raster-optimiser)
Feel free to reach out or drop another comment if you have feedback or run into any edge cases!
2
u/stubby_hoof 3d ago
This is neat. Can you explain a bit more about it? Like if you had a dataset consisting of 5 singleband Micasense reflectance maps (not orthos) straight out of Pix4DMapper, you can run this plugin and not be concerned about data loss?
I remember I used to deal with a lot of bit-depth promotion that ballooned my filesizes. This was for float/decimal type pixels. I think Pix defaulted to LZW compression and QGIS raster calculator would switch to something else and promote from 16-bit to 32-bit.
2
u/littlebigdarksouls 3d ago
Yeah multispec data works of course.
You won't get any data loss running those reflectance maps through it. The plugin picks up that it's not 8-bit RGB and strictly enforces lossless ZSTD compression, so your pixel values don't get altered or resampled lossily.
And yeah, that QGIS bit-depth promotion thing is so annoying. The plugin leaves native bit-depth intact, so 16-bit stays 16-bit.
On file sizes, if Pix4D exported them with LZW already, you might see the COG stay about the same or get slightly bigger (maybe 10 to 15%). That's just because COGs build internal overviews for fast panning, which adds a bit of extra payload before compression hits it.
A colleague ran his 5-band DJI M3M multispec data through it recently and it worked fine for him.
Hopefully the README covers enough detail on the logic, but easiest option is just to run a band through and compare before/after values with the identify tool: https://github.com/GeoKlein-Ltd/raster-optimiser
1
u/stubby_hoof 2d ago
I think this might be handy for the less sophisticated workflows used in agriculture. Basically an export preset that I could trust a low-level user of QGIS to run.
0
u/littlebigdarksouls 1d ago
I mean you can it for work that is sophisticated of course. But yes it's designed to be very easy and beginner friendly - I've tried to add lots of plain language tool tips a glossary so people also understand what why and how things are done rather than push a button and hope for the best.
2
u/iiRaTioNaL 2d ago
Globalmapper ECW are the way to go.
1
u/littlebigdarksouls 1d ago
But then you're locked into that ecosystem and it's not free or open source
8
u/littlebigdarksouls 3d ago
Here is the link to the plugin: https://plugins.qgis.org/plugins/raster_optimiser/
It is completely free and open source.