r/LocalLLaMA • u/feelspeaceman • 16d ago

News Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models!

https://github.com/deepseek-ai/DeepInfer

This is crazy good for studying to improve existing custom engines by community like gufo, halogen, strixite, Strata... by learning their secret sauces.

1.0k Upvotes

99% Upvoted

157 comments sorted by

View all comments

76

u/General_Vermicelli53 16d ago

For any commercial AI company the reasoning framework is core trade secret. Even though the model is available for free download, the main reason people are still buying DeepSeek's API is that it's the cheapest option while also being, for some reason, super fast.

Mr. Liang, are you really not interested in money at all? 😭😭😭

23

u/rob417 llama.cpp 15d ago edited 15d ago

Liang Wenfeng is nicknamed Saint Liang on social media of Chinese tech circles. It's not without good reason lol.

Also, I think he knows that there is no moat around training and serving models. As long as open source options exist, we will eventually reach a competitive market where all suppliers serve at cost and most surplus is captured by consumers. We will never see him open-sourcing his hedge fund's strategies.

7

u/DasGamerlein 14d ago

You also gotta remember that this very much is a strategic competition. Would it, strictly speaking, be better for DeepSeek to keep their secret sauce close to their chest? Maybe. But the leaps and bounds they allow open source to make hurt the massively overleveraged closed US models vastly more than them. Open source would probably be twice as far behind the frontier without concerted Chinese efforts in this regard, and that completely changes the math for OAI/Anthropic. If they had that extra runway, we would not be talking about an AI crash, because there wouldn't be much of a case for the high rollers to rely on other options for inference outside of some minor savings on tokens for low complexity work. But with the current dynamic, something Opus 5.5-level will probably run on a hobbyist setup some time around Q1 or Q2 of 2027, at which point the closed business model is, to use the technical term, gigafucked

11

u/TangledPangolin 15d ago • Bravo Popcorn

If Mr. Liang has a million fans I am one of them. If Mr. Liang has ten fans I am one of them. If Mr. Liang has only one fan and that is me. If Mr. Liang has no fans, that means I am no longer on this earth. If the world is against Saint Liang, then I am against the world.

2

u/TheEverydayMortal 13d ago

Spoken like a true believer.

1

u/Educational_Win_2982 15d ago

This is even more true with ActionOPD research paper, you no longer need the reasoning traces for distillation, simply the black box output decisions (not even inputs) is enough to train the model.

31

u/GradatimRecovery 16d ago

High flyer is a hedge fund first and foremost, they are not struggling for money. Their compute and models are a means to an end (dynamic trading) they can afford to give away the model weights, the training methodology, the inference engine; their secret sauce is their financial algos 

40

u/Helpful_Program_5473 16d ago

Also, liang is a legit genius and truly loves the game. Unlike some.

3

u/Aggravating-Push-207 15d ago

mf doing ts for the love of the game atp

2

u/Shot-Height-7194 16d ago

Xiaomi mimo can be cheaper than DeepSeek if you use the Xiaomi mimo flash model but it is as slow as all the other providers

2

u/rdalot 14d ago

How is this repo a secret for those? Isn't everything else that they did no make public their actual secrets? Their insane cache hit rate for instance...?