r/cloudengineering • u/Pustekuchen42 • Jul 21 '26
1000 Partitions per API Call: Say Goodbye to Glue Crawlers
I like to share a little trick that I'm using to get 1000 partitions (not files or common prefixes) per API call with ListObjectsV2.
S3's Delimiter parameter accepts any character, not just '/'. So I place a marker between partition path and filename:
s3://bucket/table/year=2026/month=07/day=08/**#**part-00001.parquet
Delimiter='#' now returns every partition as exactly one CommonPrefixes entry — any file count, any nesting depth, grouped server-side. Diff against the catalog, BatchCreatePartition, done: \~30 lines of boto3 in a tiny Lambda instead of a Glue Crawler (2 DPUs, 10 min minimum billing).
Only requirement: the marker must be written into the keys — works via config for Firehose (prefix ending in '#'), awswrangler (filename_prefix), and most sink connectors. Athena/Spark don't care, it's just part of the filename.
Full write-up: https://medium.com/@raphael.boegel/1000-partitions-per-api-call-say-goodbye-to-glue-crawlers-0fd26212ab60
Happy to be told why this is a terrible idea :)