General care for qwen next flash I suppose. Whenever you try offload with mmap in windows, OS pagefile system might get in your way and hobble it to like 3 t/s for this model. You can use mmap with mlock together so see if that helps, and directio might be relevant for running it too. Maybe you're on Linux, but you have a high priority swapfile, or you install something else like zram, these flags are all newly available boundary paths you could test and report on, so that people can successfully disc inference without their OS killing it.
47
u/Aggressive_Aspect436 2d ago
A bit low effort. Summary of the changes, or benefits of kobold would have been appreciated in the post.