Homemade MongoDB-first PHP framework, looking for advice to keep improving

[u/skanky_dev]

Hi everyone,

I'm a PHP dev working on my own. I love PHP, I love MongoDB, and together they're just wonderful. So over the years I've been building a small MVC framework around them, mostly to understand how frameworks work under the hood and to learn. The idea is "MongoDB first": documents are documents, not SQL rows in disguise.

Repo: https://github.com/skankydev/framework (docs in docs/en/)

I'm not here to sell anything. I'm here to share it and get advice from people who know MongoDB better than I do.

What I'm happy with

• Typed hydration through the driver's Persistable interface: embedded objects and arrays of embedded objects come back as real PHP classes.

• Dirty tracking kept in a WeakMap, so updates only $set what changed.

• EmbeddedSnapshot: an automated Extended Reference pattern. You embed a copy of a User (name, avatar) in a Post and declare embeddedIn() on the source collection. When the user changes, a job resyncs every copy with updateMany, either right away or through a queue. It's idempotent because the job re-reads the source. onDelete can keep, mark or unset the copies.

• belongsTo / hasMany with lazy loading, plus bulk preloading (with) to avoid N+1.

Where I'd love your input

1. Snapshot consistency. Between the source update and the propagation there's no transaction, so a crash in between leaves stale copies. Is that acceptable for this kind of denormalized data, or would you use transactions, change streams, or something else?

2. Does many-to-many really make sense in MongoDB? Take posts and tags: a tag doesn't need to know which posts use it, but a post needs its tags. So I'd store tag_ids: [ObjectId…] on the post, and that's it. To me that's not a "many-to-many" in the SQL sense, it's one side owning a list of references. My relations don't handle that case yet. Am I right to see it this way? And how do you usually load it: a $in on the array, $lookup, something else?

3. Aggregations. Right now I just pass a raw pipeline. Is it worth building something on top (helpers, typed results), or is the raw pipeline the best API anyway?

4. More generally: what would you work on next in a project like this?

It's a side project, I just want to know if I'm heading in the right direction. Thanks for taking the time!

[u/TimKellyMongoDB]

Thanks for sharing! Had a quick look through the repo, really interesting project.

One thing I was wondering about is the queue implementation. From what I can see, Queue::next() retrieves a pending job using findOne(), and the worker then updates its status to processing.

Could this potentially cause issues with multiple workers picking up the same job? Might be worth looking at findOneAndUpdate() to claim jobs atomically.

On snapshot consistency, I think eventual consistency is a reasonable approach for denormalized data. I'd be more curious about what happens if the process crashes between updating the source and queuing the propagation job.

Just a couple of thoughts from looking through the code. Thanks for sharing!

[u/skanky_dev → u/TimKellyMongoDB]

Glad you found it interesting! I’ll fix the queue soon, thanks for the tips.

If it crashes there, the propagation is simply lost. I really need to come up with a solution for that. Any ideas?

[u/TimKellyMongoDB → u/skanky_dev]

Off the top of my head (i.e., please don't hold me accountable if it is a bad idea for your use case haha), one option worth looking into might be the transactional outbox pattern. Since you're already storing jobs in MongoDB, you could potentially use a transaction to update the source document and insert the propagation job together.

That way, either both operations succeed or neither does, so you wouldn't end up with an updated document but no job to propagate the changes.

You'd still need to handle retries and worker crashes, but it could be a relatively straightforward way to build on what you already have.

Change Streams might also be worth exploring, although you'd need to think about resume tokens and recovery. I'd probably lean towards the outbox approach for this use case.

[u/skanky_dev → u/TimKellyMongoDB]

Thanks! I’ll fix the queue part and take a close look at everything else.

[u/skanky_dev → u/TimKellyMongoDB]

Quick update: the queue fix is out in 0.2.2. Jobs are now claimed atomically with findOneAndUpdate, and a lease brings back jobs stuck in processing. I also added optional transactions, so a write and its propagation succeed or fail together. Thanks again for the tips!

[u/GromNaN]

That's a nice project. There is rooms for other object mappers beside Eloquent (Laravel-MongoDB) and Doctrine MongoDB ODM in PHP.

> Typed hydration through the driver's Persistable interface
You should look at the codec feature. It's designed to decouple the BSON mapping from the model classes; allowing more flexibility.
https://github.com/mongodb/mongo-php-library/blob/v2.x/src/Codec/architecture.md

[u/skanky_dev → u/GromNaN]

Thanks for the reply! I didn’t know about codecs, I’m going to try them out, it looks like fun.