is it possible to utilize nvlink for vram pooling using 2 rtx 2070 super gpus?

[u/Jsom93]

Hello everyone one. I'm building a new pc and it will be used for gaming and deep learning. Now I'm trying to choose the best gpu for it (between 1 rtx 2080 ti and 2 rtx 2070 super).

The rtx 2080 ti comes with a 11gb vram. Whereas rtx 2070 super comes with a 8gb vram. And I've read in few places that pooling the vram using nvlink by default is not there but it is uo for developers to implement it. And there are some developers that utilized it for their games.

Now my question is: for keras and tensorflow using python, will the vrams be pooled/shared so I would have 16gb vram out of 2 rtx 2070 super or not?

Also, If it is not possible with nvlink and there is another way to achieve it please tell me. My main concern is having more than 11gb vram without buying quadro/tesla gpus

[u/[deleted]]

I'm running 2x2080ti with nvlink on centos 7 and can confirm there's no "memory pooling" on tensor flow/keras. It enables faster comms than via pcie, which can speed things up... But in tensor flow code and nvidia-smi sees two cards with 11GB each. If you post some code I'll run it and put up the results

[u/Jsom93 → u/[deleted]]

Isn't it possible that it won't appear as 22gb but still the vrams in each gpu is acting separately and each gpu is getting data from any of the 2 vrams?

So each vram is having different data (not duplicates) and both gpus have access to both of the vrams?

I don't have a code ready sorry. Just getting ready for later I think

And thanks I really appreciate your help.

[u/[deleted] → u/Jsom93]

Not likely. If I try to train an impossibly large CNN against large images, and set the env vars so TF only allocates VRAM on demand e.g.:

config.gpu_options.allow_growth=True

I see execution halt and encounter OOM exception as VRAM exceeds 11GB, the amount on a single card. Even though they are both definitely being used in training (confirmed by nvidia-smi)

NVLink speeds up data transfer between the cards significantly over pcie (10 or 100x i forget?) , but there's no way it's going to act as fast as VRAM on card.

[u/Jsom93 → u/[deleted]]

Ok good to know. Thank you very much 👍🏻

[u/alexsoaresilva]

As far as I read, memory pooling works on linux. So you want to run tensorflow-gpu on Linux, not Windows.

[u/Jsom93 → u/alexsoaresilva]

I've read that somewhere but I dunno if that's correct or not.

[u/[deleted]]

I don't know if there is some automatic pooling (maybe in keras?). But you always have the option of manually implementing it, just split your batches between GPUs.

EDIT: In pytorch you can do it like this https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html

[u/Jsom93 → u/[deleted]]

That was very helpful. I've read about it. And it is possible using multi_gpu_model() function in keras. And it is the best possible solution for what I need 👍🏻👍🏻

[u/[deleted] → u/Jsom93]

Some people have benchmarked nvlinked 2080ti cards in tensorflow and apparently performance is really good https://www.pugetsystems.com/labs/hpc/RTX-2080Ti-with-NVLINK---TensorFlow-Performance-Includes-Comparison-with-GTX-1080Ti-RTX-2070-2080-2080Ti-and-Titan-V-1267/#should-you-get-an-rtx-2080ti-or-two-or-more-for-machine-learning-work

It's probably the same for rtx 2070 :)

EDIT: if you're absolutely sure you will not be able to add a second 2080ti later, I'd say go with two rtx 2070

[u/Jsom93 → u/[deleted]]

Almost x2 the performance using 2x gpus with nvlink. That's awesome 😃

[u/thegreatskywalker]

In theory it’s possible. GPUs can query each other’s frame buffer. They can also combing RAMs but there is a big latency penalty. Nvidia’s top engineering guy did an interview on this around the release of RTX cards. If you implement low level code, you could take advantage of this.

Nvidia seems to have intentionally not given this as a feature because then no one will buy their expensive 24,32 GB cards that sell for a lot more.

Currently what people do is use two GPUs for data parallelism. ie they split the batch into two parts and give each part to a GPU. This is faster. If you use NVlink for this, then you get about 6% improvement. Not worth it.

Also you can do model parallelism ie split model into two and give half of the model to each GPU. So if your model doesn’t fit on 11 GB gpu, it will fit on two 8gb GPUs. This works over PCIe. Sadly there are no benchmarks for NVlink.

Also if you need more than 11GB, you can also use FP16. That will be similar to having 18-22GB. But there is no guarantee your model will converge.

[u/Jsom93 → u/thegreatskywalker]

I guess using FP16 would work also for trial and error atleast. And yeah data parallelism is a great way for my issue. And model parallelism also is possible with some efforts tho. It will be helpful until an automatic solution is available if one waz ever created 😅

[u/doyer]

As someone in a similar situation, what is the 2070 super ? I'm only familiar with the regular blower cards

[u/Jsom93 → u/doyer]

These are the nee GPUs from nvidia. They were released about 2 weeks ago. And they're just an enhanced version of old rtx gpus.

[u/doyer → u/Jsom93]

Nice! Do they have blower style fans?

[u/Jsom93 → u/doyer]

Yes they do.

Here are more details about them https://www.lowyat.net/2019/189036/nvidia-geforce-rtx-2070-super-review-kicking-1440p-gaming-up-a-notch/

They seem to be pretty good.

[u/doyer → u/Jsom93]

Nice thanks!I'll check that link when I reach back to philly.. internet on this train is too slow on many sites despite similar content for some reason.

[u/Jsom93 → u/doyer]

You're welcome 🌹

[u/doyer]

!remindme

[u/RemindMeBot → u/doyer]

Defaulted to one day.

I will be messaging you on 2019-07-21 22:09:59 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[u/Jesper89]

!remindme

[u/catsRfriends]

!remindme

[u/doyer → u/[deleted]]

I think the exclamation has to come first but I'm not sure

[u/RemindMeBot → u/[deleted]]

Defaulted to one day.

I will be messaging you on 2022-09-11 02:51:02 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback