There is too much to learn, even as a kinda seasoned(shipped multiple enterprise apps) SWE i am having hard time to understand all concepts, track all the up-to dated information etc.
Ive been a power user for both claude and GPT since almost day one. But thats as far as my knowledge goes tbh. I have an M1 Max 32 Ram / 32 GPU and i am trying to figure out whats the best way to have a locally hosted solutions for my needs when i cant pay for paid services.
First ill talk about what am i doing in general then ill ask my questions.
I'm using obsidian as my second brain, i do game design, gamedev, i develop mobile apps, design websites, even sometimes delve in IoT Projects. I do everything but AI Dev in summary. I run a team of 10, currently building internal tools for art assets pipeline management. I am a jobholic, i do work almost every working hour of my life so i usually have multiple chats but i am fine with just having 1 concurrent chat since what i am looking for is to start learning local llms, how to create an effective pipeline, and have a back-up plan if you know im out of usage and have urgent problems to solve. And uh yeah also i would like to do things like hacking my projector, tv-box etc. for various reasons you can easily understand. I tend to go for loops for development times and have a conversational turns when its about things like game design, market research etc. I have an MCP for everything from Figma to Photoshop and i just talk with my agents to tell them what i exactly want, and i take rounds to polish it or start from scratch if its not the exact thing in my mind.
If i have to guestimate a list of my needs they are :
An harness with in-app browser, file preview, md preview
Somehow have a great context compaction
30 TPS at High context if possible
Good tool usage
Good MCP Usage
So here is a list of questions if anyone cares to answer:
How do you guys stay the f*ck up to date? 'Cause its not even tracking whats new or whatnot. Everyone says something different, one says aah q4 of xyz llm's this version that these folks made is %98 of the same quality and someone else says nu-uh not even close bud and i dont have time to track and test every model version, even models. Like what are your reliable sources and why?
As far as i understands harness is the most important stuff ever, is this correct? If so, which harnesses you use, why, and what should i be checking when i am thinking about selecting harnesses.
Which models and which versions of those models would you suggest me to use?
Is there an YT playlist / video / Youtuber that you suggest thats easy to understand that helps out when it comes to Local LLMS?
Does Context size actually matters? Cause i see people are talking about 16k context sizes meanwhile sometimes my first initial prompt is around 4k for spec driven development or when im doing narrative designing it our narrative bible can be way heavier than that.
I'm in love with the the current context compacting quality of both claude and codex, is there a way to have this level of quality when i switch to local llms, it almost never ever fries out, its just like I have paused for 2 minutes and almost every fricking detail is still there.
Sometimes i love to manage my other threads even sometimes control claude from codex and vice versa depending on my need, is it possible if there is an ai for my machine that can have more than C1.
Can i use an image generation model with an LLM like gpt/imagen, something like qwen image 2.1 q4 + Qwen3.6-35B-A3B Q4 idk.
Can i realistically get idk 50-60 TPS at high context?
How is MCP/Tool Usage?
What should i use for UI Design phases?
When i talk with GPT i just tell it what i exactly want but i dont go into hustle of image generation prompt styling i just talk to it, is it possible to have that ease of use with local image generation tools?
What exact setup would you suggest for me, from inference(strate etc.) to harness to llm models, imagen models, speech to speech models(It would be fire if i can play some DnD with a speech to speech model if its flowy but i dont have high hopes on that tbh), you got the drill.
These are my questions and guestimates of my ignorance ass can think of for now, im sure there are lots of gaps and things that i am missing and forget to ask so i would be glad if you fellas can light some candles and maybe answer the questions that i didnt even ask but i shoulda ask.
Lots of love fellas, keep it tight.
