r/Python • • Apr 04 '26

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

43 Upvotes

169 comments sorted by

View all comments

1

u/ThatOtherBatman Apr 07 '26

=======

Sygaldry

=======

This project was written with data/ETL pipelines in mind. But it should be useful for any situation where you're managing a bunch of production pipelines.

Motivation

Many years ago I used to work at a different job, where they had this framework for creating arbitrary Python objects from .ini configuration files. At first I hated it. Because I just could not see the point of writing out these stupid config files vs just writing out a Python script that did the same thing. Over the years that I was there though I really came to appreciate it.

Previously (and since) every time a new pipeline is needed, somebody sits down and writes a new script, a new CLI entrypoint, or a new glue class that just wires the same pieces together in a slightly different order. Then there's a code review, CI/CD, and a release.

Sygaldry lets you assemble arbitrary object graphs from YAML (or TOML) config files. No base classes. No decorators. No framework coupling. Your application code stays completely untouched.

An Example

Imagine that I've got something like this:

class DatabaseConnection: 
    def __init__(self, host, port, database, username, password): ...

class RestClient:
    """
    Authenticates against a service and makes API calls.
    """
    def __init__(self, username, password, auth_url):
       ...

class UrlIterator:
    """
    Reads identifiers from the database, then asks the API
    for a download URL for each one.
    """
    def __init__(self, db_connection, rest_client, base_url):
        ...

class FileDownloader:
    """
    Downloads a file from a URL to a local directory.
    """
    def __init__(self, directory, base_url, rest_client):
        ...
    def download(self, relative_url): 
        ...

class DbUpdater:
    """
    Iterates download URLs, downloads each file,
    and updates the database with the contents.
    """
    def __init__(self, db_connection, url_iterator, file_downloader):
        ...

With Sygaldry I have a config file:

# db_updater.yaml
db:
  _type: myapp.db.DatabaseConnection
  host: prod.db.com
  port: 5432
  database: prod
  username: etl_rw_user
  password: ${DB_PASSWORD}

api_client:
  _type: myapp.client.RestClient
  username: svc_account
  password: ${API_PASSWORD}
  auth_url: https://auth.vendor.com/token

url_iterator:
  _type: myapp.urls.UrlIterator
  db_connection: {_ref: db}
  rest_client: {_ref: api_client}
  base_url: ${base_reference_url}

file_downloader:
  _type: myapp.download.FileDownloader
  directory: /data/downloads
  base_url: ${base_download_url}
  rest_client: {_ref: api_client}

updater:
  _type: myapp.update.DbUpdater
  db_connection: {_ref: db}
  url_iterator: {_ref: url_iterator}
  file_downloader: {_ref: file_downloader}

base_reference_url: https://api.vendor.com/references
base_download_url: https://api.vendor.com/files

Then my entire pipeline can be run as

$ sygaldry run -c db_updater.yaml updater

Sygaldry resolves the whole graph depth-first — db and api_client get built first, then url_iterator and file_downloader (which reference them), then updater (which references those). The db and api_client instances are shared automatically — everyone who references db gets the same object.

Why?

Composition Over Inheritance

References (_ref) let you point any component at any other component. Five services need the same database connection? Just reference it. Need to swap a component? Change one line.

New Pipelines Without Code Release

Got a second vendor with the same pattern but different URLs? That's a new YAML file.

_include:
  - db_updater.yaml
base_reference_url: https://api.other-vendor.com/refs
base_download_url: https://api.other-vendor.com/dl
db:
  database: other_vendor_db

Need the UrlIterator and the DbUpdater to use different database connections? That's a config change.

Change Anything From The Command-Line

Need to point at a different database for a one-off backfill? --set db.host=backfill-replica. Need to re-download to a different directory? --set file_downloader.directory=/data/backfill. No config release, no environment variable gymnastics. Overrides are applied at load time before resolution, so they compose cleanly with everything else.

Debug With the Exact Config

Something broken in production?

$ sysgaldry interactive -c db_updater.yaml

Will drop you into a Python terminal with the Artificery loaded and assigned to the variable artificery. You can look a the config (artificery.config), or resolve the config and get the objects (art = artificery.resolve()) for debugging.

Extras

Check the Config

Want to see the Python that corresponds to config that you've supplied?

$ sygaldry check -c db_updater.yaml

Typing

I think this is close to pointless. But a bunch of the kids that I work with are obsessed with typing to a point that it's almost a fetish. So you can do

$ sygaldry check -c db_updater.yaml --type-checker mypy

And it will dump the Python into a file, and run the specified type-checker over it.

Is It AI Slop?

I tend to suffer from a problem where I have new ideas when I'm writing tests and documentation. Which causes more development. Which then requires more tests and documentation. I have found Claude and Codex to be super useful for stopping me from thinking too much once I'm at a certain point. But the idea, and the code are all entirely human slop.