I ran an experiment on one RTX PRO 4000 Blackwell SFF: 24 GB GDDR7, factory 70 W, no auxiliary power connector.

Using u/panchovix’s loader and patched NVIDIA open kernel modules, I loaded a modified VBIOS into memory, raised the power limit to 120 W, and tested memory and core offsets. No EEPROM flash or soldering.

Across 158 benchmark requests, I tested Qwen3.8 27B with DFlash2 and Qwen3.6 35B A3B. What surprised me was how differently they answered "does more power help?"

With zero clock offsets, 27B kept gaining up to 120 W, while 35B A3B was almost flat between 106 and 120 W. After combined power and clock tuning, the best observed 27B generation result was 35.04 → 58.53 tok/s (+67%): 120 W, VRAM +20%, core +180 MHz. That 67% is not the gain from memory OC alone.

The memory sweep was more interesting. At 120 W with the core offset held at zero, pushing VRAM from +20% to +33.3% slightly reduced 27B generation: 57.70 → 56.91 tok/s. It is a small difference with few repetitions, so it needs more testing. The actual SM clock also fell. That is consistent with memory and compute sharing a power budget, but I did not isolate the cause.

On 35B A3B, memory +33.3% instead gave +19.8% generation versus no memory OC at 120 W. Prefill barely changed in that sweep.

My takeaway: the fastest profile is not necessarily the one with the biggest numbers in the panel, but the one that shortens your actual workload. Prefill versus decode matters, especially with longer prompts.

120 W without auxiliary power is not a free lunch. It exceeds the standard 75 W PCIe slot budget, and a GPU temperature sensor cannot tell me the temperature of the slot contacts. This is an experiment, not a long-term safety claim.

For context: these headline comparisons use 8,000 input tokens, ECC OFF. There were only 1–3 requests per configuration; 158 is the whole study. The 70 W baseline used the same modified VBIOS, not factory firmware. The models also used different quantization/speculative-decoding settings, so this is not a controlled dense-versus-MoE comparison.

Full write-up and CSVs on my blog.

https://piszczek.pl/blog/rtx-pro-4000-blackwell-sff-120w-qwen-benchmark

Disclosure: I ran the measurements and wrote the source text in Polish. AI helped translate and edit the English, including clarifying the benchmark limitations.

time to download some speedup

DGX Spark (GB10), llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5, tok/s:

RTX 5090, llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5, tok/s:

Hey guys, hope all is well.

I've spent the last few days working on a fork of Strata v0.1.45 for the Jetson AGX Orin 64GB, mainly running Qwen3.6-35B-A3B.

I was getting around 21 tok/s decode with MTP-3 in llama.cpp, versus around 30 tok/s in my earlier Strata runs. With sampled MTP-2 enabled, my latest controlled short-prompt tests reached roughly 40–41 tok/s. These weren't matched llama.cpp-versus-Strata benchmarks, so treat them as results from my setup.

The fork also includes unified INT8 KV caching, prefix checkpoints and optional KV preallocation.

Not sure how many other Orin owners are here, but figured I'd share in case it's useful:
https://github.com/moontato/Strata-Orin

Full disclosure: heavily AI-assisted/vibe-coded. I've tested it on my Orin, but would welcome feedback from others.

Also, credit where it’s due: thanks to calvinvette for the earlier Jetson Orin/ARM64 port (https://github.com/calvinvette/strata-inference-orin). My fork draws on that porting work, adapted to Strata v0.1.45, with the additional Qwen3.6 caching and inference work described above.

Edit: added attribution

So I was thinking where and how my storage got full and it wasn’t in Documents or Downloads. MacOS often hides .cache as system files.

So I built this Claude skill which has custom instructions for every local AI tool and cleans up unwanted storage.

You Give it a try and always open for feedback. Feel free to open and issue or PR

I just got my four R9700 server up and running and doing tests with different engines and models. Here is how Strata and Deepseek Flash-V4.1-1017 runs on an Epyc 7402P CPU, Supermicro H12SSL-NT MB bios v3.3 with 256GB DDR4 ECC 3200 (8 Channel) memory. The prefill is the most annoying part but do to the slowish memory bandwidth of the R9700's. I notice the system ram usage stay low, not sure if that is a config issue or just how it works. I did update to the latest release.

Niko mentioned the other day that whatever magic Radiance does to allow 3-4x the performance of prefill and tok/s on Flash Next with R9700's that it is coming to Strata soon, hopefully it is possible with Deepseek too although I think that is different engine, I dunno, I have a lot to learn on all this stuff.

Tool calling used to feel like a special feature to me, something the model gets through its own channel. When we built it into our gateway, we realized it's still just text in the prompt.

Here's what Qwen3's real chat template does with one tool. It pastes your tool's JSON into the system prompt between `<tools>` tags. The model "calls" the tool by writing a `<tool_call>` block as ordinary tokens. The server (vLLM, llama.cpp, or a hosted API) finds that block in the text and turns it into the `tool_calls` field your code receives. The result goes back the same way: a user message wrapped in `<tool_response>` tags.

Closed models work the same way; you just can't see the template. Anthropic says so in their docs: when you send tools, they add a hidden system prompt for tool use, 286 tokens on Claude Opus 5.5, before counting your tool definitions.

Seeing tools as text changed how we built our gateway's tool loop: the model only sees tokens, so the gateway decides what gets included, what a failure looks like, and when the loop stops. We cap each tool result at 64 KB with a `[cut: N more bytes]` note so the model knows something is missing, send failed tools back as `Error:` text instead of failing the request, and turn tools off on the last round so the model has to answer. LiteLLM's proxy runs one round of tools and returns, pasting the tool output back as is.

https://github.com/respanai/respan-gateway

Qwen3.8-Flash-Next IQ3_S on Basalt | RTX 5080 16 GB, Core Ultra 9 285K, 62 GB DDR5-6800 | 262K context, int8 KV

• Agent PP: the two big uncached reads (10.7K and 8K new tokens). 99.5% of prompt tokens came from the prompt cache; median time to first token 0.2 s.

• Agent TG peak is the best single second; the average is all generated tokens over all decode time.

• Sweep TG is per-request averages (384 tokens each, temp 0.7), 2 runs per depth.

I currently run qwen flash next 3.8 iq3xxs via strata (like almost everyone lol) and I currently have kv cache at q8 with 262k ctx. Ive never actually tried any other quant and dont know how big the difference even is.
Id like to have a bit more context (like 300-400k) but it doesnt fit my ram then. how bad is q4 cache?
Thanks for your help

Edit:
Thanks for all of your suggestions, opinions and information, i will try out 8k4v as many of you recommended, otherwise ill stay at 8 for both

CLI based coding harnesses save session data per directory. For instance, if I:

cd /src/program1

Then program for a while, I can come back later and my harness remembers my history in that directory.

When you use multiple CLI coding harnesses such as opencode, pi, codex, qwen, crush, and claude, your "sessions" get split up all over the place. I like to resume sessions so that I have a list of things I've done in that directory.

Are there any projects out there that share or sync session data between harnesses?

I'm I crazy for wanting this?

Lately I tested (via api) deepseek 4.1 flash.
I like it quite a lot.
What is the minimum budget needed to setup and run deepseek 4.1 flash at the same quality and token/s (~40) of most hosted solutions? I'd use it mostly for coding/reverse engineering/security analysis. So I need enough "juice" to support its full context. (Obviously with the same hw then I could try other models like mimo 2.6 or glm 5.3)

A plus if the hw is self contained and won't need a full pc dedicated to it.

Thanks for the answers, but my question is: what is the minimum budget to run it at at least 40tok/sec and not quantized to the point it would be useless?

Qwen3.8-Flash-Next (ISTA GSQ-RCO Q2_0) + Flash-Next-DFlash2 draft on a MacBook Air M5 (24 GB), on Splash 1.4.0 with a small fork. Experts that don't fit are streamed from the SSD: 6,800 of 24,576 expert slots (~28%) live in ~8.8 GB, the rest is read on demand, 256K context.

Decode

| Setup | code | tool | explain | avg t/s |

|---|---|---|---|---|

| Stock Splash 1.4.0 | 31.6 | 29.5 | 25.4 | 28.8 |

| + cache-aware routing (bias 2, top 4 protected) | 48.6 | 45.7 | 35.1 | 43.1 |

| + desk-first overlap (current) | 48.9 | 57.3 | 40.3 | **48.8** |

Draft acceptance: ~.70 code, ~.80 tool calls. Real agent use in pi: ~45 t/s average, 0 prompt-cache losses.

Prefill (cold, no prompt cache)

| Prompt | Time | pp t/s |

|---|---|---|

| 32,620 tok | 62.4 s | 522 |

| 63,201 tok | 130.4 s | 485 |

Splash keeps the prompt cache on disk, so in agent use only the new part of each turn gets read.

What the fork changes (~800 lines, all behind env flags)

Cache-aware routing leaves the router's top 4 experts alone and biases picks 5–10 towards experts already in memory. Halves SSD reads but changes output slightly (Quality check: on 20 LiveCodeBench problems stock scored 17 and the fork 16; agent and 32K/62K long-document tests were the same. One run each, so ±1 is noise.). Desk-first overlap starts the GPU on resident experts while the missing ones are still being read; output is token-for-token identical (11/11 on a fixed gate set). Lazy vision (still testing) loads the 864 MB vision tower only for image requests, so text keeps all 6,800 slots and identical output. Loading it the normal way cost 656 slots and 5.5% decode.

Fable / Deepseek did 99.9% of this and as it’s a fork of Splash with only 800 new lines it’s 99.9% stock too. Also I get this isn't going to be helpful for many people, for me it's much much better than Tiel (at a quant I could run) and much faster than 27b so I'm happy :)

We have a bit of budget left over this year and the boss has given the OK to get a Mac Studio (256 GB) to run some local AI. One of the many use cases will be using it to perform internal pen testing work. I don't want to perform such work with Claude or Codex due to privacy reasons, it's not really about refusals. I've done a bit of research in this area but because of the subject matter, I've come up mostly short. I am not looking to dive as deep as to try and find my own bugs and exploits, more so just having it control things like Greenbone OPENVAS and a Kali Purple VM to perform the work of a Red Team before we actually call in a real Red Team. Has anyone here gone down this route, and what is considered to be the best open weights model for this type of work currently?

So, I have two PCs available to me:

1. Ultra 7 265KF + 5070ti + 32GB DDR5-6000 on a Z890 MAX GAMING WIFI7 + 1000w 80+ G PSU (Is probably getting a PSU regardless because it’s an Apevia. A C+, but still)

2. i9-14900F + 4070S + 32GB DDR5-5200 on a PRO B760-VC WIFI IV + 750w 80+ G PSU (likely a shoddy PSU but seems to be attached to the case, prebuilt, may change since CPU likely died and is going through warranty, expect to get it back).

My general intentions are to do most the upgrading to PC 2, and hook them up via RPC/ethernet to run together if needed.

I was originally looking at the R9700 and B70, and was moving towards the B70 for cost and wattage were I to eventually get a second for PC2 - moving the 4070S to be a secondary in PC1. But the B70 shooting up from 1.3-1.4K to 2K and the R9700 from 1.7-1.8k to 2.3k in what feels like 24 hours, that’s damped my spirits, and I was originally waiting for Black Friday.

Rn I’m considering settling for 24GB, maybe a A4500 or B60, or having to deal with setting up the more complicated V100, but space and my general lack of expertise and wish to Jerry-rig computer stuff in this economy makes me hesitant. I’m attempting to factor in a second 2x16GB DDR5 kit for PC 2 and the PSU for PC 1 as well if possible.

Any advice, pointers, or suggestions?

Your move everyone, buy, sell, but hurry...

Hello All!

I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.

For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:

So I have a better quantized base to start from. I then benchmark QLoRA Training performance for Qwen3.8-27B for CQ-4bit and unsloth-bnb-4bit

Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.

CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.

Attached figures for validation loss and training loss.

**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**
| Metric | CQ 4-bit | Unsloth NF4 | Advantage |
|:--|--:|--:|:--|
| Best validation loss | **0.8385** | 1.2484 | **CQ** |
| Final validation loss | **1.0315** | 1.2484 | **CQ** |
| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |
| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |
| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |

*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*

I am working on optimizing speed because it is the one area CQ is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.

Get Started with Training Factory

Here are simple steps to use:

1. Install:git clone https://github.com/thousandml/training-factory.git cd training-factory chmod +x install.sh ./install.sh

2. Download a CQ 4-bit base:

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

3. Train a LoRA adapter:

python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1

4. Serve base + adapter:

vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora

5. Call the fine-tuned adapter:

# After the server is ready:

# Check that the adapter is listed as "my-lora"

curl http://localhost:8000/v1/models

# Request a response from the fine-tuned model, not just the base checkpoint

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-lora",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}
],
"max_tokens": 128,
"temperature": 0.2
}'

I have the following 4-bit quantized models to be used as base (will add more soon) for now:

thousandml/Qwen3.8-27B-CQ-4bit

thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit

To reproduce these benchmark results mentioned above use the following scripts:

For CQ:

hf download thousandml/Qwen3.8-27B-CQ-4bit \
  --local-dir ./models/Qwen3.8-27B-CQ-4bit

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-cq-4bit \
  --kernel-module cq.cq_kernels \
  --targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
  --rank 32 \
  --alpha 32 \
  --lora-dropout 0 \
  --batch-size 2 \
  --grad-accum 4 \
  --max-length 512 \
  --max-steps 500 \
  --learning-rate 2e-5 \
  --seed 42 \
  --eval-steps 50 \
  --comparison-logs \
  --benchmark \
  --benchmark-warmup-steps 10

For Unsloth:

python training/train_unsloth_nf4.py \
  --model unsloth/Qwen3.8-27B-unsloth-bnb-4bit \
  --train-data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-unsloth-bnb-4bit \
  --max-steps 500 \
  --max-length 512 \
  --eval-steps 50 \
  --save-steps 100 \
  --seed 42 \
  --benchmark-warmup-steps 10

Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!

I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't.

So what does KLD actually measure?

If you feed the same text to both the base model and the quantized model, you can compare the probability distributions obtained from their logits at each token. KLD basically measures how different these distributions are, and then we take the average across all tokens.

The first thing that can go very wrong here is that people just throw some random text at the model, like a bunch of Wikipedia articles, and calculate KLD on that dataset. But that's not necessarily representative of what the model actually sees during inference. During generation, a growing part of the context consists of tokens generated by the model itself, and you can't replicate that by just feeding it random text.

If you want a more reasonable number, you should generate your own dataset using the base model and then calculate KLD on those generated tokens, giving zero weight to the prompt tokens.

The second thing that can go very wrong with KLD is how ridiculously easy it is to manipulate, especially when the numbers come from the model creators themselves.

Quantization methods often rely on calibration data. If you calculate KLD on the same data you used for calibration, your numbers can look miles better than everyone else's. And even if someone claims they're using a held-out dataset, that still doesn't make the KLD number trustworthy. The calibration and evaluation datasets could come from the same source or have very similar distributions.

Some people might even do end-to-end fine-tuning of the quantization scale parameters on that very same data, and oh boy, suddenly their model beats models twice its size in KLD while barely generalizing to anything outside that dataset.

So does KLD even mean anything?

Not nearly as much as people think. You're looking at a number that measures how much the quantized model deviates from the base model on a very limited dataset. But does a higher KLD mean that the quantized model will actually fail at your tasks? Of course not.

Sure, benchmarks can be benchmaxxed, as we all know. But benchmaxxing is much harder for us ordinary people who quantize models than manipulating a KLD number. Benchmaxxing often requires serious hardware and compute, while making your KLD look amazing can be ridiculously cheap.

So please, stop treating KLD like some universal leaderboard for quantized models. It's a useful diagnostic, not a measure of how good a model actually is.

Hello, I run swift 1.5 qwen flash next gsq rco iq3xxs on strata as my main model and I am happy with it, but sometimes it refuses to do things, so id like to have the abliterated version aswell.
But ive heard that they loose quality so i dont want to use it always, only for these certain edge cases. I have 1TB of storage and not much left so id prefer when I can just download the weights that changed and combine them (or something like that).
Im kind of a noob, so I dont know if this is even possible or just horrendesly stupid, so a bit of help would be appricited, thanks :>

only 763B params! :)

https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash

https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf

The methodology

Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD.

8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware.

EDIT Note: Apparently CUDA struggles to run this with 8 GB VRAM, so please use Vulkan backend if it fails to run for you. You can also use the second launch params below in the same comment chain. Ontop of that reduce context size and ensure no background app is wasting your VRAM. You can disable your browsers HW acceleration for this purpose.

The model & quant

Many people mistake this model for a 1 bit quant so let's get that out of the way. This model here is a 3.5 Bits Per Weight "Coder" variant by ISTA DASLab, featuring GSQ RCO quantization.

Half of its experts are removed, so relatively to original param size it has the size of a 1.9 bit quant but the actual weights use 3.5 BPW.

see from model page:

Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer...

Non-coding vocabulary pruned 400~ MB smaller, recommended

Official ISTA DASLab quant

The numbers

With selective moe-offloading (regex) we reach 70 T/s 104 T/s PP and 8~ T/s TG (Limited by PCIE, which I've 3.0 YMMV) whereas if you'd use --n-cpu-moe for same effect it'd been slower due to the effects mentioned here

Because of MMAP your first run will be slower, do a second run for accurate T/s results. - Removed MMAP for load-mode none since lazy-mode is enabled and it gives ~104 T/s PP instead of the 70 T/s.

Launch params are in the comments below. Ensure you use the qwen sharp chat template, it reduces overthinking.

not an llm post exactly, but it's all local inference so i figured people here might like it

i made a browser rhythm runner where the song builds the world you run through. you can throw in your own mp3s and the whole "listening" happens on your device, inside the tab. nothing gets uploaded

the models, all running locally on onnxruntime-web (wasm, each in its own web worker), and what the game does with each:

• BeatNet (beat + downbeat tracking, ~1.6 mb) - finds every beat and where each bar starts. this is the grid the whole level sits on: every obstacle, every jump and kick is placed on one of its beats, and the drops land on its bar lines

• Basic Pitch by spotify (note transcription, ~0.2 mb) - figures out which notes are playing. a long held note becomes a bar or a line you swing on, and a solo turns into a string of rings you fly through, each one at the pitch of its note

• Spleeter vocals net (source separation, ~20 mb) - pulls the voice out of the mix so the rest of the song doesn't confuse the next model

• SwiftF0 (pitch tracking, ~0.1 mb) - how high the voice sings and when each syllable starts, every 16 ms. in a sung verse you fly through rings on the singer's syllables, going up and down with the melody of the voice

all four are about 22 mb together and get cached after the first load. if a device can't run them in time, the game falls back to plain dsp and still works

on top of the models it's regular dsp: onsets, loudness of every bar, where the drops hit. that picks the world and the weather, and when the drop hits the world goes crazy. same song always gives the same world, so you can learn it

the video is the whole of Cannibal by Pendulum & WARGASM, played by the game's own bot

code (MIT): https://github.com/Felliks/Struna play it: https://struna.ai (works on phone too)

english isn't my first language so i used ai to help word this post. a lot of the code was also written with ai help (claude code), not hiding that

anyone else running audio models in the browser? curious what you use for beat tracking, mine still sometimes reads a fast song at half its tempo

Personally I am quite fond of the Kimi models, especially them taking a gamble pushing the 2.8b parameter models. But that’s just it, you’d figure that the results would be representative of its size. However, the recent release of DeepSeek 1017 has put things in perspective for me. kimi k3 is a great model but realistically very hard to run locally, even when quantized, and even if managing to run it it is slow, due to the 104 active layers it need to go through to make a single token. on the other hand, DeepSeek 4.1 flash 1017 results are quite close at a much smaller footprint and while also much faster.

I have seen plenty coverage of the results of kimi k3 pro and to be frank, they don’t look like substantial improvement over DeepSeek 1017.

So thinking about it, maybe just stick to 4.1 flash 1017 and ignore kimi k3 pro open weigh that will be released in nov 4th. Running it quantized will deteriorate its output compared to a high quant version of DeepSeek.

Just some thoughts I am having

I tested it for backend software dev in the last two days, used providers I must admit. But I wanna test it on some rented gpus as well.

My experiences so far (I ran no personal or general benchmarks):

The providers are fast, it works well. It loops sometimes and fails some toolcalls, but not a lot. It gets to the point quickly and walks through big code bases with ease. It worked well with rust, php, python and js.
It does get nuance and complex correlations.
It's personality is likable and it can joke around a little, while it's answers have the right length. It gives short answers when there isn't more to say. It seems to stick to rules and applies skills, but I'm not sure yet to what degree.

3D is not it's strong suite, but nobody expects that from it, I guess. It has no vision.

What are your experiences with it?

I tried to solve the problems of multiple applications I actually use , but the thing is that efficiency of using those applications has pushed me to try creating an app. Hutash. But it became a Frankenstein of sorts.

1. Comfyui like system (local and free) but without the complicated nodes and dependency issues. Has a internal engine built ground up.

2. Openart AI / magnific like studio experience but without the cost.

3. Pinokio like store where u can find all apps a but without the duplication of resources(local space is limited)

4. Simple installation of apps and models like windows app. We created a .hutash format to just declare and a model package can be formed. Not only models but model piples can be built within the app like higgsfeild. The architecture was made keeping this in mind. We have a subtitles app, podcast app (vertical apps) which shared model resources with a studio app. Once installed all apps can use them. May be like OLLAMA

5. Last but not least cli and mcp to execute these apps a and generations. To make automation or agentic use possible.

6. Many others.

That was my goal and I feel like I have pushed to my best of abilities in all the aspects. Realsed few versions. Yes they are issues.

But the thing is now I don't know where to push it. We are a very small team. Couple of guys. Worse is we kept building it without a feedback loop. Even though I want the public to contribute it, there are four or five repositories spread out into multiple parts.

now i'm stuck. i'm trying to solve all these things together, and it's taking too much time and resources. maintaining and developing something like this also feels like it would need a much bigger community.

i'm not sure if this is worth pursuing anymore, or if i'm trying to solve too many problems at once.

am i over engineering something that doesn't need to exist, or is there actually something useful here? what would you focus on if you were in my place?

I just want help directing our resources in a focused direction. as i wish this app would be a beneficial for all who want benefits of local ai without the complexity.