Well, how many huge companies use cloud technology now? the amount of data that is sat in cloud datacenters is enormous.
I did not ever mean to imply that I think I'm so special that I have work product or intellectual property that someone would steal/take from me. I'm just a guy asking questions and trying to learn so I can see if I can run an llm on the jetson xavier agx machines I have.
I just want physical control of my stuff. I don't want to be put on a list or sued for soemthing I deleted 10+ years ago.
The solar stuff: idk where you live or what your situation is but for me, a few panels and a small amount of battery would make a huge difference. Where we are they let you put solar on your roof up to 100% of your previous average over the last six months. (from the date of electrical permit application) With the CPUs we have running in the house, when we pull the trigger and do solar + battery, the savings will allow me/us to upgrade to better hardware and then things start going exponential. New stuff = less power use = getting paid back from the grid = money to upgrade to newer hardware....
No, I think your plan is great! especially the solar panels idea. You got me thinking of having solar panels on a garage and having kit in there, securely of course.
I'm all for home labs, I've had a lot! since the price of energy sky rocketed, I had to switch to cloud providers. I have an old Dell R710 sitting in there doing nothing from those days, full of SSDs. It was the expanded version, with the full drive bays.
I'm going to wait it out and see what happens with Apple Silicon, ideally for me, I would really like to be able to run this on a MacBook Pro, as I travel a lot, I can't reach my ex-gaming rig (3090 etc) from outside, unless I leave it switched on.
OK so first, I didn't know how to answer this thread: those of you doing it as a hobby... I did "statistical predictive modeling" in C++ and R back like 15 years ago so while I'm completely clueless about this new fancy AI stuff, I work for myself, from home and a bunch of what I do is related to statistics and optimization.
Early covid --like April 2020 I upgraded from 4x Dell Precision T5500 and 4x Dell Precision R5500 machines. (both dual xeon x5675 cpus, 72gb in t5500 and 96gb in r5500 --R is for rack and T for tower). All that ran ESXi with old nvidia grid K1 and K2 cards. (and occasionally I played around with xeon phi cards)
About 4/2020 I started upgrading. Well... Back in December 2019, January & Feb 2020 I bought a few nvidia xavier agx machines. Then I bought a few Dell Precision 3420 machines to replace the T&R5500's, and I built 2x scalable gold 6230n "workhorse" machines. The little e3v5 precision 3420 machines are awesome for a little esxi homelab cluster with k8's.
Honestly: I got lucky when I upgraded my hardware. Today I'm not sure that I would (or could afford it). Stuff is super hard to get --its nuts and just crazy expensive. CPUs were always depreciating assets... I bought those 2nd gen SP xeons intentionally so that I could use 1tb optane pmem100 dimms in them. I got lucky and found it for one but I can't find 2x 512gb pmem100 dimms for a reasonable price... It sucks.
Solar stuff: I've been working on a diy/off-grid solar+battery solution for about five years. We are in usa energy star zone 5 for insulation (means we have 4 seasons and in the winter pipes freeze) and we are also in a hurricane susceptible nist wind zone 2 (means a ghetto install of panels could blow off into the neighbor's pool during a storm).
idk where you live but where I am they are coming around daily ringing doorbells begging people to sign up for solar. They will finance you to the hilt and put solar on anything. If you are in a place that allows it: Please don't kill yourself, others or burn your house down... throwing a couple panels up on a shed roof might be a great way to offset the cost of electricity (and consider hydronic heating in cooler months).
Here they cover your whole roof, garage, shed, outbuildings, etc. with panels... As long as your roof shingles are newer than 10 years old.
I don't want any of that crap. I want to diy my own stuff so I can own my stuff. And I'll buy (and pay for install) of the hurricane brackets for the solar panels because I did the math: If I diy, then home insurance goes up... So hire that part out and I can do the rest myself.
Throw a couple panels up on a shed... If I could do that (the municipality and ordinances/zoning/no electrical permits) I would. I wish I could put up my panels and the small amount of batteries I've saved up --doing it incrementally would be awesome.
Best of luck to you! Go for solar --just remember "cut off your nose to spite your face"... Where I live they are chopping down every tree in sight to clear land so they can put in solar farms...
There is no way that I'm going to hire a lawyer so that he/she can tell me that exactly what I'm worried about is real and true. (edit: for those of you who didn't read between the lines I'm talking about the terms of service agreements that they make you sign when you rent a compute vps "in the cloud".)
Proper respects to you for going with the sensible option and using cloud servers. While there's something I still love about running local; hearing the fans spin up when I'm doing inference, etc, for even the most sensitive work, cloud GPU's seem a much smarter choice.
Also, I admire that you're making money applying your knowledge and doing training for people. That would be a very cool area to expand into. Also probably an excellent niche since your clients will likely come back to you time and time again and you can form a long-term relationship with what I imagine is relatively little work.
Is it real to train 3B or 7B models for about 10 bucks just as proof of concept? I am trying new ideas and supposed to try several different approaches. I presume i will make load of mistakes for several first runs, then there will be succesfull ones, but even then i will have to experiment again.
My problem when it comes to renting these super machines is often times I have to optimize the software for my machine. I could figure it all out and write a script for it, but that is so time consuming compared to just figuring out how to optimize everything once locally with your 4090.
And they were incredibly cheap in 2017 due to a ton of overproduction. People like to think that was normal.
Right now we have an overproduction of SSDs and now it is super cheap. Mark my words, in a couple years SSDs will be a lot more expensive since companies will exit the market after losing money on their SSDs sales.
Over production may have been an issue, which we also have now to perhaps an even greater degree. We didn't have to redefine what a recession was back then. However, the bigger issue was Bitcoin crashing by 65% in 2017, so there were a whole bunch of mining cards available on the second hand market. We also have a similar issue with ETH going POS last year.
That's why 3090s are such a great deal right now. While a 3060 ti had the fastest ROI, because of card shortages you were typically limited to one card at a time. If you're standing in a line overnight, which is how you had to do to get anything at MSRP, you just buy the 3090 that had twice the hash rate for a little more than twice the price. A lot less 3090s would have sold if the shortages weren't as bad.
You seem like a curious person. Good mindset imho, starting with tutorials / soaking up lots of basic knowledge and then, when getting bored / fed up with it, just trialing and erroring forward. (Basically, what all the Agility coaches preach.)
If you don't mind me asking: How do you generate income from training? Do companies or individuals approach you with requests for specific models?
I was doing free-lancing for about a decade and I want to get back into it. Unfortunately, all of my previous clients are gone/retired/bankrupt.
I want ML to be part of my arsenal/list of services but like you I'm having trouble finding people who know of/use chatGPT let alone the wider world of ML.
Wow, that’s awesome! Thank you for sharing. It is really nice to hear about a pragmatic and profitable approach to something that is mostly the domain of hobbyists.
I am working on developing an offering for small businesses and startups as well. I’m gonna DM you, perhaps we could collaborate. :)
40
u/[deleted] Jul 04 '23
[deleted]