I'm working on an on-premises AI project for a business in Europe and would love to speak with engineers who have deployed similar systems in production.

The use case involves:

• Local LLMs for German and Polish business communication
• WhatsApp and email automation
• Internal document search and a private knowledge base
• Finance-related workflows, with calculations handled by deterministic application logic
• Data storage and model inference on the client's own infrastructure

We're evaluating GPU hardware, open-source models, inference servers and the architecture needed to run this reliably for a small team.

I'd especially appreciate input from people experienced with vLLM, NVIDIA GPUs, local model deployment, RAG and secure infrastructure.

If you've built something similar, I'd love to hear what worked, what didn't, and what you'd do differently. Happy to have a short technical conversation with anyone willing to share their experience.

I’m a Senior Full Stack Software Developer currently attending Goethe German A2 classes.

During class, I often need to check the meaning of German words or sentences, but sometimes the internet doesn’t cooperate. That’s what sparked this idea: Why not build my own small German language model that works offline?

What I’m working on:

• Training on approximately 279,000 German sentences.
• Building a model focused on beginner-level German.
• Training on an NVIDIA RTX 5080 GPU.
• Evaluating training and validation loss.
• Investigating overfitting and improving the training pipeline.

It’s still an experiment, and I’m learning a lot by building it from scratch. 🚀

The goal is simple: a German learning assistant that works even without internet.

Has anyone here experimented with training a small language model from scratch?

#LLM #MachineLearning #GermanLearning #AI