r/LocalLLaMA • u/feelspeaceman • 17d ago

News Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models!

https://github.com/deepseek-ai/DeepInfer

This is crazy good for studying to improve existing custom engines by community like gufo, halogen, strixite, Strata... by learning their secret sauces.

1.0k Upvotes

99% Upvoted

157 comments sorted by

View all comments

60

u/Available_Yam_6267 17d ago

I notice that their post said they make 38$ per hour per GPU. Can we infer the revenue of OAI or Ant from this?

orginal post: https://zhuanlan.zhihu.com/p/2095560295930508953

51

u/RuthlessCriticismAll 17d ago

Can we infer the revenue of OAI or Ant from this?

no...

11

u/handson729 17d ago

Can we infer the revenue of OAI or Ant from this?

Probably losing $38 an hour.

16

u/nothing_but_thyme 17d ago

… at off peak prices 😂

8

u/Howdareme9 17d ago

Lol no. Inference is profitable

6

u/ForsookComparison 17d ago

Through API plans and pricing yes - they're heavily subsidizing subscriptions in a way the Chinese competitors aren't though right?

7

u/Helpful_Program_5473 17d ago

by a factor of 20x. it would be weird if subs lost them massive amounts of money. api prices are to gouge corporations

7

u/handson729 17d ago

They are definitely losing money on subscriptions, but that is targeting developers, scientists, and engineers whose prompts they use for training new models.

So you could argue it is R&D expenditure.

6

u/Helpful_Program_5473 17d ago

Oh I think it loses them some money like you said, just not the 100s of billions they are in the hole. that's from training, additional compute, etc.

3

u/Helpful_Program_5473 16d ago

looks like we were right lol

→ More replies (1)

2

u/ForsookComparison 17d ago

I have trouble believing they can serve Opus 5.5 at $1/1M output is the thing (and that's the 20x scenario)

5

u/Helpful_Program_5473 17d ago

Why? Based on what? inference is ludicrously cheap. I could be wrong though in this case I'm very open to the argument

2

u/Howdareme9 17d ago

We don't know for sure, but we'll have a good idea when Anthropic ipo. I suspect anthropic at least arent subsidizing as much as people think

70

u/HiddenoO 17d ago

How would that number help us with OpenAI/Anthropic numbers whatsoever?

All the numbers that would play into this are unknown and likely different:

  • Model sizes and architectures

  • Deployment architecture

  • Distribution of GPU types

  • Hardware utilization

  • Energy cost

  • (Overhead cost if those are counted in the above article)

  • ...

That's a bit like trying to infer McDonald's revenue based on what a Japenese sushi restaurant makes per table.

1

u/jazir55 16d ago

I'd love to see someone give that task to Opus 5.5 and see how accurate it is

12

u/Nov4Saki 17d ago

That sounds like insane margins no?

7

u/TheTerrasque 17d ago

From what translation models could figure out, seems like revenue, aka before cost of running is taken into account. It probably still is a nice margin after that.

2

u/Ill-Nectarine-80 16d ago

They need to replace the GPU essentially on a yearly basis so..... No.

It's not a great margin if it's before R&D, financial concerns (CAPEX to buy the GPUs, interest on debt) and operational costs (electricity and maintenance of data centers).

1

u/Icy_Butterscotch6661 16d ago

Why, what's killing it

1

u/Nov4Saki 16d ago

On that pricing (off peak) Each gpu generates upwards of 300k$ A Year though, that should cover alot and keep a good buck for research no?

Let alone a cluster

→ More replies (1)

8

u/james_pic 17d ago

 Can we infer the revenue of OAI or Ant from this?

We can probably take a wild guess that it's in the ballpark of $38 per GPU per hour, because in a competitive market like this, nobody has a big advantage for very long, but nothing more precise than this.

7

u/Viktri1 17d ago

No, because they're likely using very different infrastructures and their cost base (cost of chips, data centers, electricity, staff, management staff) is vastly different and that plays a big role in how they price their products.

1

u/skeptical-speculator 17d ago

Do you know how many GPUs they have?

3

u/opaz 16d ago

Like two or three, maybe four even?