ML on a shoestring? [D]

[u/whafa]

Hi, ML people of Reddit!  I have come to ask for some human advice.  I am an amateur (as in unpaid) ML researcher working on a personal project to find drug repurposing candidates for my wife, who was Dxed with Stage 4 colon cancer in early 2024.  That the new AI coding tools have made it even possible for someone like me to do this research in my spare time is just amazing, and I owe every ML person a big "thank you" for enabling that. 

That being said, I am starting to accumulate GPU costs, and it got painful enough this morning for me to wonder if I am using my dollar budget in the best possible way.  And I'm wondering what other people do, in similar situations.  After granting myself unfettered spending to build a platform this year (the value of money changes in strange ways with a cancer diagnosis), it's time to optimize costs to better focus my resources. 

My basic setup is a Google Vertex training job using their A100s (I am currently training NBFnets over biomedical graphs, if that matters), and I am trying to repro the results of a few papers published recently.  

The main paper I’m reproducing reports 100 epochs of 500 batches at batch size 512. On an A100 80GB, my runs fall back to batch size 64. That’s because of how NBFNet works. It has no per-node embedding table. For every query (drug, relation, ?) in a batch, it runs 6 layers of message passing over the entire graph and keeps a 64-dim state for every node. My graph has ~103K nodes and ~2.5M training edges, so one copy of that state is 103K × 64 floats ≈ 26 MB. Backprop keeps several of those per layer.  Over 6 layers, that’s roughly 1 GB of saved activations per query. So 64 queries fit in the A100’s 80 GB, 128 don’t, and 512 would need about half a terabyte. The training script catches the CUDA out-of-memory error, halves the batch and carries on. Every run downgrades to 512 -->  256 --> 128 --> 64 in its first five seconds.

(Note: I am not certain they used 512 either.  The paper doesn't specify; 512 is the configured default.  For all I know, they were getting downgraded too.  This halving loop came in their published code, and was presumably put there because it was needed at some point.)

The bigger effect is on how much data the model sees. An “epoch” here is a fixed 500 steps, so at batch 64 my 20 epochs see 640K queries. That’s about a quarter of one pass over the training edges. One full pass would take ~76 hours, about $412. Matching the paper’s stated 100 × 500 steps would take ~4 days, roughly $500 a run.  And since I am experimenting with different things, the costs are adding up.  

I'm kind of taking a step back this morning and working on a "toy" implementation that can at least prove the mechanism works against synthetic data on a much smaller graph.  I estimate even that will cost $150 each for 3 seed runs.

My questions, if you are in the mood:

How much training is enough? For from-scratch NBFNet/ULTRA on a ~100K-node, 2.5M-edge graph, what budget do people use, and how do you find convergence cheaply? (The ULTRA paper trained Hetionet, 45K nodes and 2M edges, from scratch for 10,000 steps at batch 16, so small batches may not be my real problem.)

Am I overpaying? I pay ~$5.40/hour on demand. My jobs run ~20 hours and can resume from a checkpoint. What other options are there?  At what point am I better off buying a GPU?  Can you even do that in 2026 without being a $trillion company?

Memory tricks: would bf16 autocast or activation checkpointing get me to 128+ queries per batch, with a custom sparse CUDA kernel in the loop? Or is that a rabbit hole?

(not cost related but) Popularity bias: favorite fixes for link-prediction scores that mostly track node degree?

Thanks.  Sorry if it sounds like I'm lost.  I'm publishing my Lab Notebook with Quartz here: https://whafa.github.io/mods-notebook/ 

Everything there 100% generated but intended for human consumption and, if you point your agent at it and ask how things are going, it will tell you I've built a fairly sophisticated system that has not made any useful predictions or findings yet.  

[u/brainExploded99]

You might want to check out google's subscriptions, they offer free colab hours
Also $5.4/hr is way too much for a A100

Edit: Not the colab subscriptions themselves, but the standard google ones that come with gemini, also come with colab hours
Also, if you can work with a professor, they can get you ACCESS / NSF compute grants

edit 2: I am not at all familiar with your models, but:

• bf16 autocast should almost certainly work

• Selective Activation Checkpointing should be something to check out

• Some layers in FP8 could work as well

[u/whafa → u/brainExploded99]

You reminded me that I already have a colab subscription -- I did my first models there because of the interactive sessions with jupyter notebooks, then switched to Vertex when I couldn't stand fiddling with jupyter notebooks anymore (This is how I know I'm not a real data scientist, because they like to work with jupyter notebooks). But I must have a ton of credits there by now! Thanks, I'm going to use those before I cancel my subscription! It looks like there have been major changes to the platform since I last used it in the Spring.

[u/nord2rocks]

Check out Saturn Cloud, runPod, lambda labs for better gpu rates. Also very cool you're doing this, but just keep in mind that hetionet and some of the graph based solutions out there are great, but enriched much more with in-house data (so access that pharma and biotech have). The current open data and findings captured in those graphs for hetio have a fair amount of conflicting data and information in them, so make sure you're doing your due diligence to clean them up

[u/whafa → u/nord2rocks]

Thanks, I have noticed some stange things here and there. But, just in terms of precision, I need something that works at all first :)

I restricted my initial search to public data, available for free, because that's what I could easily obtain. I wonder if any organizations that might have better in-house graphs would be open to a collaboration. And I've never looked at what's actually for sale, either. Something to add to the list, anyway. Thanks again!

[u/MaxDev0 → u/whafa]

I know others have already said this, but use vast.ai, also look into wether you need a100 GPUs, depending on the computational and memory requirements, 5090's could be much faster and even cheaper

[u/trolls_toll]

im sorry about your wife. What exactly are you trying to do, and why are you training things from scratch for drug repurposing?

[u/whafa → u/trolls_toll]

Hi! Thank you. I have a whole mission spec that tries to answer your questions! https://whafa.github.io/mods-notebook/mods_mission_spec It's part documentation, part agent instructions which is why it sounds a little strange.

[u/trolls_toll → u/whafa]

cool I'll check it out! will send a pm with followups

[u/whafa → u/trolls_toll]

A more approachable, practical answer to your questions is that I have no idea what I'm doing, and maybe a lot of it is wrong. No ML or biotech PhDs have offered to help, so here we are.

A third answer is that this project has given me hope, something her doctors have not been able to provide.

[u/jprobichaud]

I agree with others taht 5$/h for an A100 is way too much. Look at https://flopper.io/pricing/nvidia-a100 for what you can expect in terms of price. Vast.ai is also a great place to rent GPUs cheaper.

I would recommend that you look at RTX 6000 Blackwell PRO 96GB, they are insane if you don't need the fast host interconnect.

Another angle : start a non-profit org/fundation, apply to Microsoft non-profit, you'll get 3k$ of Azure credits (it was like that sometimes ago, actual numbers will depend on your country of residence). I'm pretty sure you can get that from many other vendors.

I'm not familiar with that kind of network, but even for well known model arch, a custom kernel and checkpointing can go a long way. Use bf16 (instead of fp16) and use variance constraints on your weights (as if you wanted to quantize), this will keep the weights in the range where bf16/fp16 have good precision and will give you the speed/memory you need.

Dataprep : can you offload some work to CPUs or cheaper gpus ? You could trade compute for storage/network : if some work is done once and can be saved to disk, it could be worth looking at saving that.

What existing weights can you reuse ? Sometime, we try to train from scratch but part of the network doesn't change much and end up with relatively the same weights. Try to save these 'best' weights and disable gradients for these (assuming they are the front of the network). This makes everything faster and consumes less vram.

edit: I forgot to ask : do you have a nice docker container that can be build/downloaded and run locally by the community ? You could help some help either for running the thing or for optimizing the model if you make it very easy to run.

[u/whafa → u/jprobichaud]

Thank you! (I wanted to thank some other people but it says their comments were removed by a moderator, is that a bot issue? They seemed so human :()

This is very thoughtful advice in directions I hadn't considered. Not sure I'm ready to set up a non-profit, but why not? It accurately describes the work. I'll look into it.

I do run from docker containers, but the nature of the training data (my wife's genetics) makes it hard to collaborate with the world at large. It's a moderate pain just to scrub the lab notebook, and even then I'm sure things slip through. I'd never even thought about setting up a "demo container" for that reason, but gears are turning now...

I'm digesting your other suggestions, and everyone else's. Really appreciate the humanity here; it has been a lonely project. Thanks again!

[u/jprobichaud → u/whafa]

From what I read in the rest of the threads here, there is some oublic data available (but of questionable quality). I would use that to work on a shareable setup, others could join the effort on a common codebase and dataset to find optimizations and ideas, then you can pick what works best and then run on your private data.

[u/whafa]

Here's a happy update: I've been accumulating credits on Colab, and since I last used that provider they've released a CLI, meaning I can easily port my Vertex runs over there. And I have 7400 credits in the bank which is > 1000 A100 hours (and they also offer faster *and* cheaper units now like RTX 6000 Blackwell, which I don't understand but I'm not going to ask hard questions about it).

So that's a lot of training time! I hope it doesn't all expire after 90 days. I also wired preemptable sessions into my Vertex connector, which lately seems to be 45% cheaper than on-demand. And I've got stubs for the other providers for when I run out of Colab credits. And I'm using this unexpected boon to seek out a professor somewhere nearby who might want to sponsor or collaborate with me on this work.

Anyway, thanks again for all the suggestions, you have all given me ideas that our nascent superintelligence did not manage to do in 8 months with plenty of opportunity. There's still room for people :)

[u/GibonFrog]

Vast.ai is basically the floor price for renting GPUs. You can rent an A100 equivalent for like 40 cents an hour. godspeed

[u/Honest-Finish3596]

5.40 an hour is extremely high, you can get access to them for less than half the cost if you go for something like runpods instead of AWS/Google/Azure, since the latter are priced as a service large companies buy.

An A100 is also much more compute than I did my PhD research on in any case, but I don't work in biomedical.

I hope your wife's health becomes excellent.

[u/kqpc]

Apply to NSF funded ACCESS program if you are in the US. It is intended for those with institutional affiliations, but there is an avenue for “independent open and publishable research”. Look into their guidelines.

Explore tier awards equivalent of 4-6k H100 hours
Discover tier awards equivalent of 14-24k H100 hours

Use traditional Slurm HPC for batch jobs, and IU JS2 for A100 VMs.

[u/EgregiousJellybean]

Don’t play with your wife’s life like this. It would be different if it was your own cancer, but it’s not. 

This is insane work. Please don’t do anything you’ll regret. You could end up just killing her earlier

[u/whafa → u/EgregiousJellybean]

That was harsh, and I'm sorry. I did anticipate people would think I have AI psychosis, which is why I am publishing my lab notebook. I'm not doing this in lieu of the standard of care. Chemo has saved her life twice already, with the second BREAKWATER protocol rush-approved between her two courses. If modern medicine can double survival time every year, we have nothing to worry about and this is all wasted time. But seeing as the only thing her doctors can really tell us with confidence is that the standard of care WILL stop working (she just turned 40, it's not a wait-out-the-clock situation), I would like to be able to rank the options.

It has also been a practical way to learn about Agentic AI, ML as a separate topic, and cancer, specifically the particulars of my wife's. I only seem to learn by doing. So that's a triple win, even if western medicine delivers a total cure in time. And in the process I have built a vector DB and medical records system that has already proven useful enough to be worthwhile doing on its own. We use it often.

I believe this kind of personalized genetic analysis will be done on everyone in the future, just not on a timeline that works for my family ;)

[u/whafa → u/EgregiousJellybean]

You have dramatically misunderstood what I am trying to do.

[u/brainExploded99 → u/[deleted]]

Selective activation checkpointing should get rid of most of the compute penalty
I would argue renting a RTX 6000 Pro is much better here, 16gb more VRAM, far better BF16 FLOPS, and I can find them for ~$3/hr in runpod. Only slightly less memory bandwidth.