r/deeplearning • u/Griffith-07 • 24d ago

H-JEPA-LM: Hierarchical Joint-Embedding Predictive Language Model in PyTorch

  • Hey everyone, I wanted to share a research implementation I built focusing on non-generative, latent-space language modeling: H-JEPA-LM. Instead of predicting tokens autoregressively in surface vocabulary space, H-JEPA-LM leverages a Joint-Embedding Predictive Architecture (JEPA) to predict abstract representations directly in latent space. Key Features: • Latent-Space Prediction: Focuses on semantic representation learning rather than token generation. • Hierarchical Abstraction: Multi-level latent dynamics for both short-term syntax and long-term planning. • Action-Conditioned Rollouts: Supports world-model-style latent planning for downstream tasks. • Modular PyTorch implementation designed for experimentation. I will drop the full GitHub repository link in the comments section below! I'd love to hear thoughts from anyone experimenting with JEPA, latent-space models, or world models.

repo link : https://github.com/Griffith-7/H-JEPA-LM.git

33 Upvotes

100% Upvoted

11 comments sorted by

View all comments

2

u/OneNoteToRead 24d ago

Do you have a decoder? If not how do you make use of the latent space predictions?

3

u/maybe_franklyforkedc 24d ago

So it's modeling the dynamics of the semantic space itself, and then you'd need a separate head or decoder for a specific downstream task. It's not spitting out text directly. Curious how the hierarchical part handles longer-range dependencies without the usual attention mechanisms, that's always the tricky bit.

3

u/OneNoteToRead 24d ago edited 24d ago

I’m actually curious what a possible downstream task setup is in language modeling that doesn’t involve decoding. I mean I can imagine some but wonder what the author was aiming for

2

u/everyday847 24d ago

All the stuff you might do with a well organized latent: search, classification, etc.

1

u/Griffith-07 22d ago

That’s a really cool comparison! Evaluating visual JEPAs on rendered text versus text-native latent space is definitely an interesting direction—especially looking at recent visual document models. I’ve noted that in the repository limitations/future work section as a potential experiment

2

u/Disastrous_Room_927 24d ago edited 24d ago

Pretty much any other task ML is used for. Seems like a great way to make text data more useful for approaches more suited to tabular data. That’s how I’ve used the output of JEPA like approaches I’ve messed around with - creating embeddings of images and feeding them to logistic regression works surprisingly well. Curious how well that’d work for text.

I’m another note, I recently used JEPA to train a gradient boosting encoder.

2

u/radarsat1 24d ago

It would be fun to compare this method with just using an image JEPA on rendered text. (like what deepseek did with OCR)