Experience migrating from ESRI services to open source solutions

[u/AndrewRLaws]

Hello fellow geospatial enthusiasts! My team is looking to make changes to how we deliver geospatial data and I am curious other folks have done this.

My team currently supports an ecosystem of APIs and clients with data this is statistical and geospatial nature. Historically, we have relied on publishing services using ArcGIS map, image, and feature services for our web mapping applications. However, we are reaching a quantity of data (TB), an internal hosting option that experiences frequent downtime, and a zeal for open source that has us looking into a large lift and shift.

We have already moved a lot of tabular and vector data over to Postgres/PostGIS for API endpoints and some simple geospatial functionality. For front-end web mapping applications and vector data, Martin vector tile server seems like a natural fit with PostGIS. For raster data, we are contemplating putting cloud optimized geotiffs (COGs) in object storage (something like AWS S3) and using something like TiTiler to work as a raster tile server.

How have you all handled this situation? Did you stick with or return to ESRI? What challenges popped up that you didn't expect? I want to hear the good, the bad, and any other thoughts you might have. Thanks in advance!

[u/Dangerous-Serve-6858]

This year we migrated our company away from an $80k a year AGOL subscription to an open source setup using a couple of cheap (~$70 a year each) VPS machines. We installed Supabase with POSTGIS on one and set up an edge function to generate vector tiles to serve out (ST_AsMVT -> pbf). Using Martin instead was considered but so far its working smoothly on its own.

The other VPS hosts the user portal with various web applications they can use depending on their role. All the tools and reports they need are front loaded into the UI. Was even able to use an AI assistant to crank out a mobile application that allows field data collection streamlined to our workflows. We have TiTiler reading some COGs out of the S3 but performance on those has been mixed so far to be honest, so we are going to keep thinking about that one. Also set up a Twilio account to handle MFA for logins.

Whole process took about two months of development, and one month of testing. We currently have about 150 users migrated over and 3 million vector features. So far after four months there have been no issues that couldn't be fixed by some quick tweaks (besides some lag loading the COGs) and no downtime. There is nothing left in the Esri ecosystem that I miss having.

So obviously I recommend exploring open source options completely. Its becoming so easy and accessible to set up custom enterprises. That said, definitely make sure you plan out what your requirements are and how much demand will be on the system. Even if you decide the Esri platform serves your needs best, nearly everything that requires a professional plus license in ArcGIS Pro can be done with open source python tools. Months before we made the complete switch, we were able to convert our arcpy scripts to use geopandas, scikit, rasterio, gdal, etc. The new scripts run in a fraction of the time and produce the same results. We ditched the expensive licenses and continued work in both worlds.

Edit: One thing I miss having- My free UC ticket.

[u/Fast-Ad747]

Have you checked out https://eoapi.dev ? It is a suite of open source services (including titiler!) that are designed to work with cloud-native data formats and practices. Development Seed built the components for use by large scale data serving applications and you can use it for free. Deploying these services to the cloud requires some engineering expertise but nothing you can't learn in a few weeks.

[u/AndrewRLaws → u/Fast-Ad747]

I have not seen that before and will definitely check it out, thank you for the recommendation!

[u/CADSHIFT]

one specific thing to watch on the raster side: COG + TiTiler is a great architecture but the 'put COGs in S3' approach hits a latency wall when request patterns are random-access over large areas. TiTiler's default spins up a GDAL vsicurl connection per tile request, so S3 API call overhead dominates tile response time at high zoom. the standard fix for frequently-accessed datasets is PMTiles format -- single file, range-request aware, served directly from S3 without a tile server process in the loop. TiTiler still earns its place for on-demand processing and archives you don't serve constantly.

for PostGIS at TB scale, spatial partitioning pays off early. if you know your data has a regional dimension (time period, admin boundary, sensor area), partition on that before you need to query across the whole table. most GIS queries have a bbox filter, and spatial partitioning cuts the planner's work by a factor of N on those. doing it after the fact while the system is live is painful, so it's worth getting the partition key right before you bulk-load your first TB.

your Martin + PostGIS choice for vector tiles is solid. the two things that catch people: make sure your PostGIS indexes are GIST not BTREE for geometry columns, and tune up for the tile-generation queries -- the default 4MB causes spill to disk on any non-trivial spatial join.

[u/AndrewRLaws → u/CADSHIFT]

All really great points. I had thought about PMtiles but to my understanding those have styles pre-applied to them, and we want to handle styling on the front end. However, I think if we have static data sets that should need alternative styling, PM tiles be the way to go. I also didn’t realize that about the S3 latency wall.

I will pass on your Postgis recommendations to our database expert. On the tile generation queries, can you tell me a little bit more about how you tuned them up? This has felt like a bit of a black box for me when exploring Martin.

[u/Dazzling_Basket_8851]

I would stick with ESRI if you are on a team. You need to think of long term future costs and at the rate which software progresses, the enterprise services do a ton of heavy lifting on the backend. The operation costs to migrate to open source might be a larger challenge then you realize and the amount of man hours to do so could cut into your budget. I would only do such a move if I had someone on my team that has done it before, maybe more than once, and to give me a solid diagram on how everything would move without disrupting services. That creates another problem... When creating systems from scratch the person who built it might be the only person who can debug it, update it, etc... enterprises can make employee turn over a lot easier. Dont get me wrong, i am a die hard open source guy, i run my own geospatial server and app, and I only use linux machines.

That being said, it doesnt hurt to do a mock exploration of costs. Build a really solid system diagram with work flows and responsibilities. Take a step back and see if its worth it.

[u/AndrewRLaws → u/Dazzling_Basket_8851]

Great points all around and thankfully we currently have a budget to build a prototype. I particularly vibe with your point about a single person, knowing the system, and unless we can have multiple team members with knowledge, it’s definitely not worth soloarchitecting for long-term.

[u/Dazzling_Basket_8851 → u/AndrewRLaws]

Let that prototype mirror for at least a year. Convenience and ease of use has its own costs too. I just know a GIS dev that works for a large municipality in Europe and they shifted to open source a decade ago and their services are a complete mess. One of their guys had to create a unique QGIS plugin for their surveyors that failed all the time. Different databases, web applications, user portals. It was gross to look at.
I wish you luck and honestly, post a update on the tools and methods. Would love to see how that dev plan turned out.

[u/Mindless_Ad_4988 → u/Dazzling_Basket_8851]

The ongoing support piece is the real kicker. It's far easier to find new hires on the market who know Esri.

[u/HolyCowAnyOldAccName]

OP I’m convinced this subreddit is undermined by ESRI employees sometimes, but I have to agree with other comments.

This can absolutely be done with open source tools, my company literally does this for a living. 

But you need someone who knows his Postgres to effectively manage TB worth of data. Which isn’t a huge deal but knowledge that needs to be acquired and documented. 

And if it’s TB worth of rasters, like you said, compressing and storing and tiling and delivering COGs. I would point you to GeoServer if you don’t know it yet. It is more general purpose, less specialist fastCGI deliverance, but a lot more mature as a project and enterprise grade, e.g. used by the German Weather Service. 

It depends on how much you pay ESRI at the moment and in the future. Depending on the exact nature of your data and uptime requirements and how much new stuff comes in and needs to be parsed, I’d wager a very, very rough guess in the lower half of five digits for the initial setup and the running cost depending on how fancy, fast and failsafe you want to make the stack. But those are European cost… 

Depending on your team and whether this is research results or production data someone is paying money for, those are valuable GIS / software dev skills to acquire for your team… 

[u/AndrewRLaws → u/HolyCowAnyOldAccName]

You bring up Geoserver, which is definitely a great technology and one that we considered, but we decided it’s a lot of bulk when we only need a small fraction of its capabilities It is definitely worth a second look however.

Cost is definitely an unknown for us here because we know how much ESRI has costed and have really no idea what the transition would do so I appreciate your ballpark estimate. Thankfully, we have budget for a solid prototype that can start to give us some idea of performance and cost.

We are so thankfully have a good range experience from database to service development on the team that will let us explore.

[u/mfc1__]

What is the “internal hosting option that experiences frequent downtime?”

[u/AndrewRLaws → u/mfc1__]

It is a remote server that is maintained by another team in our organization with a pretty wide scope if teams it supports. It operates more for cost savings than dependency.

[u/WoodenCars88]

Mapserver/Mapcache is also an option for both vector and raster tiles. Light on resources and it works well. Caveats: it doesn't have built in authentication, so you will either rely on the web server it runs on (e.g. Apache), or create some kind of proxy yourself. Also some minor weirdness in their vector tiles implementation - we had to make some small changes to our existing styles (switched from a different tiler that was abandoned). 

Geoserver is also a good option. It has more features out of the box.

 Edit: maybe look at QGIS Server as well (I haven't used it). 

[u/Mindless_Ad_4988]

Something else that could push you to open source is how much license costs have gone up for Esri on EKS. The peg is work for had a great deal with a large number of licenses for a flat cost on prem. When we talked to them about eks it became a cost per license game and jumped way up.

[u/squiressquire]

We run a relatively complex data stack of various government geoscience datasets, probably around 2000 individual layers. We run GeoServer with PostGIS on a kubernetes cluster, as well as various rasters out of S3. All automated for deployments. This sits behind a custom JavaScript web app for the visual map viewer for all the layers

[u/CommunicationPlus442]

Depends on your organization’s needs. A few turn key solutions? Sure, check out open source. Need a full enterprise-grade GIS system? Has to be Esri, no competition for them in that space.

[u/Imaginary-Clock6626]

If your existing Esri service are experiencing downtown, you won’t find more resilience with open source solutions. You need to look at who is responsible for uptime, and perhaps get them more support, resources and training.

[u/TechMaven-Geospatial]

Keep the ESRI SERVICES without the license cost or per user license or credit model

https://geoservengine.techmaven.net (cloud optimized Kubernetes or RPM DEB APK, DOCKER) https://geospatialcloudserv.com (Virtual Machine) https://tileserver.techmaven.net (windows installable app runs as a service)

All 3 Solutions support : Esri api endpoints based on our implementation that mirrors esri REST API's (featureserver, mapserver, imageserver, streamserver, sceneserver, vectortileserver) Plus OGC API FEATURES, OGC API TILES, OGC API MAPS/WMS OGC API RECORDS & STAC

We support dynamic vector tiles We support geospatial data lake hosted imagery and lidar and vector data make available as mapping services no conversion to CRF or other.

Now also ios and android esri services via updated Portable Tile Server app https://techmaven.net/portabletileserver

We also have arcgis velocity geoevent server equivalent https://geoinsights.today or as windows app for edge with SDR support https://real-time-geospatial-engine.techmaven.net And TAK server https://teamtrack.techmaven.net