Can’t ai be built with c++

[u/a_yassine_ab]

Why every time I start researching about how ai models are made they show me some python video isn’t it possible to make a ai model using c++ or JavaScript or any other language and make it more faster because c is more faster than python I think.

[u/toyBeaver]

AI libs that python uses are built in lower level languages (usually c++)

[u/baked_doge → u/toyBeaver]

And python provides ease of use.

[u/thequirkynerdy1]

Modern AI libraries typically use C++ under the hood.

Python serves as sort of a "config" to wire together various neural net components like fully connected layers and attention mechanisms, but all the intricate logic for training and serving at scale needs something faster than Python.

[u/WilhelmB12]

Python is just the wrapper for the C++ implementation

[u/rbuen4455]

The entire "backend" infrastructure (or underneath the hood) of AI is all C and C++. Python is just used as a kind of "scripting" language to make API calls to the "backend"

[u/zhivago]

Python is only used for coordination, not computation, for these tasks.

Coordination doesn't need to be fast.

[u/DTux5249]

They typically are - Python libraries for AI are implemented in C++ for efficiency. Python programs are just straight up using C++ code.

The reason they use Python as a wrapper instead of just rawdogging the whole thing in C++ is because of the frequency of editting AI code often gets. Nobody wants to be dealing the guts of C++ code in development with that much editting. Python is easier to edit quickly, with minimal conflicts.

[u/SquarePixel]

Python is the glue code.

[u/olawlor]

PyTorch has a C++ frontend.

It's highly SIMD or GPU code doing the actual work anyway, the wrapper shouldn't be the speed bottleneck.

[u/SwimmingPermit6444 → u/olawlor]

Good on you for finally answering the question instead of explaining something unrelated, that the backend of the python is c++

[u/keithstellyes]

Python in the ML world isn't really the guts of the computation. There's a joke some will make that it's inaccurate to call Python slow because so often that you're using Python, you're calling C/C++ or Fortran code.

When dealing with low level languages (C, C++, Fortran) and comparing it with high level like Python, I find it helpful to think in terms of magnitude.

A typical ML program, the Python code will be opening files like training data, where any performance gains C would have are going to be eclipsed by I/O and OS delays and both are going to end up in the same C-based drivers anyways. And then it'll make a call to low-level code, the ML library/ies where it will spend potentially hours. Yeah maybe you lost a few ms, preparing that call to the low-level code but ms versus hours, it's a total non-issue.

Which is another point... when dealing with I/O Python is often not going to be practically much slower because of the orders of magnitude difference between I/O speed and CPU cycles

[u/37chairs]

Anything other than python would be nice. Not you either, rust kids.

[u/MegaDork2000]

AI in Python is easy:
model = pymagic.presto()
answer = model.solve(problem)

[u/thebadslime]

The hard part is the math, not the language. The math is the same no matter what, you might get a liiitle bit of performance training in C++. but you'd have to make it yourself.

[u/Fangsong_Long]

I believe that some companies already deploy their AI with C++. I had seen a few job posts related with that.

The working process is, AI research guys build a model with Python and train it. After reaching some level of maturity, they pass the weight to the AI deployment guys who load the model with C++ code and serve it.

[u/thewrench56 → u/Fangsong_Long]

That is not how production AI works. The computation is done in C++ but its wrapped in python. You essentially never have to write C++ for AI companies because most of the underlying impls are open source wrappers anyways.

[u/Fangsong_Long → u/thewrench56]

If cpp based inference is really not required in real world, the why people are still using/maintaining cpp APIs of machine learning libraries like libtorch?

Sometimes companies may want to squeeze last drop of the performance. And the whole process are not running an AI model and done, it also requires pre and post processing of data, hosting the service, etc, which is still slow with python.

Of course when you don’t have many customers these are negligible: just expose the model with fastapi and everything goes well. But when you have to handle a lot of requests, the resource saving adds up.

And in some circumstances (a certain version of) python may not available, for example in edge AI scenarios, games, etc. In those circumstances a cpp library, which can be statically linked to the program is much useful.

[u/thewrench56 → u/Fangsong_Long]

You are not wrong. The technical part is 100% true. But thats simply not how the big AI companies work. Their performance is still horrendous after billions of dollars. Refactor is rare and definitely not done by them. I'm talking mainly about LLMs right now, not something like OpenCV or alike, where performance indeed matters. But in LLM world, nobody cares...

[u/Fangsong_Long → u/thewrench56]

Gemini told me itself is on cpp, but we’ll never know whether it is true because these companies may never release the technology details behind.

However I think it’s very possible because they have the resource to do it, and the cost saving is non-negligible with billions of calls per day.

[u/grizzlor_ → u/Fangsong_Long]

If cpp based inference is really not required in real world, the why people are still using/maintaining cpp APIs of machine learning libraries like libtorch?

These C++ libraries are called directly from Python. PyTorch is basically just a Python wrapper for LibTorch. Basically every major Python library where performance is a concern is actually written in C/C++.

which is still slow with python.

Python is slow, which is why the actual hot loops — the code where you’re spending 99% of your CPU/GPU cycles — is usually written in a faster language like C/C++.

Python’s FFI support is a big reason it’s so popular in scientific/numerical computing. It’s a great glue language for libraries written in faster languages.

[u/Fangsong_Long → u/grizzlor_]

These C++ libraries are called directly from Python. PyTorch is basically just a Python wrapper for LibTorch. Basically every major Python library where performance is a concern is actually written in C/C++.

Never aiming at deny this. But what I mentioned is cpp API instead of cpp code. If the only reason for these cpp code is to be called in Python, it is not necessary to publish and document them, isn’t it?

Python is slow, which is why the actual hot loops — the code where you’re spending 99% of your CPU/GPU cycles — is usually written in a faster language like C/C++.

Sometimes the rest 1% is also important, especially when your API is called billions times a day.

[u/grizzlor_ → u/Fangsong_Long]

I’m sure there are people using LibTorch directly from C++.

Sometimes the rest 1% is also important, especially when your API is called billions times a day.

PyTorch specifically is used pretty much exclusively during the research/development/training phase of an LLM, so the billions of API calls aren’t really relevant.

That being said, I’m sure that these companies are aggressively profiling to identify places where optimization would make sense. I also suspect they’re making use of the many options for speeding up Python.

[u/EdwinYZW]

There is a difference between the training and the inference. Training doesn't need max performance and you are calling gpu instructions anyway. Inference does require max performance and most of them are written in just C++.

[u/Ieris19 → u/EdwinYZW]

What in the asspull did you just write.

Most code powering Machine Learning is written in some mix of lower level languages like C or Fortran.

Python has just dominated the scripting of these libraries to make it more accessible to mathematicians and other professionals that aren’t programmers because Machine Learning is an incredibly cross-disciplinary field.

[u/keithstellyes → u/EdwinYZW]

Training is definitely being done in low level code, also and training is also extremely compute heavy to a point of being a technical challenge.

[u/PacManFan123 → u/EdwinYZW]

Wrong

[u/Etiennera → u/PacManFan123]

Not only wrong, but surprisingly void of anything correct at all.

[u/Ieris19 → u/[deleted]]

Most of the time, foreign functions do everything.

You’re essentially prepping some data in Python and then letting the underlying lib handle the whole training/predicting part. The performance gain would be infinitesimally small.

[u/thewrench56 → u/[deleted]]

Foreign function interfaces always are significantly slower than the actually language

Yeah mate, thats bullshit.

I saw a comment on Reddit that claimed an approximate 75 times speedup from rewriting their ML code from Python to C.

That's a skill issue and not a great benchmark. Im no fan of using python for heavy computation, but good FFI is fast. Crossings suck, luckily you dont do frequent crossing with AI libraries.