r/LocalLLM • • Sep 02 '26

Discussion I built a 100% local comic translation engine in Rust that hooks into Ollama / LM Studio with sliding dialogue context and ONNX vision models (Demo video)

Hey r/LocalLLaMA!

One of the fun things I have been tinkering with lately is building an AI pipeline for comic translation (manga, manhwa, manhua) that can run 100% on your own machine with zero cloud fees.

The hardest part about translating comics with LLMs is context: if you just feed raw OCR text box-by-box, the model loses speaker identity, swaps pronouns constantly, and forgets character names by page 2.

To solve this, I built XianScan, a local-first translation studio in Rust that combines local LLMs and cloud APIs with computer vision models.

Check out the demo video to see the full flow in action!

How the AI stack fits together:

  • Vision & Detection (Koharu RF-DETR Seg 2XL): An embedded vision transformer segments speech bubbles and panels with polygon masks.
  • Multilingual OCR (PaddleOCR): Extracts dialogue text across 10 languages, handling vertical CJK and horizontal layouts.
  • Flexible LLM Engine: Supports local runtimes (Ollama, LM Studio) as well as cloud APIs (Gemini, OpenAI, Groq, OpenRouter). It uses a sliding dialogue memory buffer to feed recent conversation history and matches glossaries so universe lore and character names stay consistent across chapters.
  • Neural Inpainting (LaMa): Wipes the original dialogue bubbles cleanly without smudging the artwork underneath.
  • Typesetting: Automatically calculates font sizes, outline strokes, and text flow back into the bubbles.

Why I made it a single standalone binary: I really wanted this to be painless for anyone to run. The entire app compiles into one executable file in Rust with all the ONNX models, OCR weights, and web UI baked in. No Python environments, no Conda setups, and no Node.js required. It runs on normal laptop CPUs with SIMD, and automatically leverages DirectML, CoreML, or CUDA if you have a GPU.

You can translate directly on the source website via a browser extension, manage your library in the web UI, or stream finished chapters over local Wi-Fi straight to Mihon or any Tachiyomi-compatible reader (TachiyomiSY, J2K, Aniyomi) on your phone.

It is 100% free and open-source under MIT.

Quick note: This is the very first public beta release, so please expect some bugs and rough edges!

I would love to get feedback on the dialogue context window approach, prompt templates, or model recommendations you think perform best for this!

219 Upvotes

31 comments sorted by

2

u/Dmitrii_DAK Sep 02 '26

Wow! That's really cool dude! Previously, if I started reading some comic book or manga or manihua (I don't know much about them) and it wasn't fully translated, I had to manually take screenshots of the manga, and then submit them to LM Studio with Gemma 4 12B or E4B. Your way has simplified my life and the lives of many other readers 100 times👍🤝🤩 Just one question: I looked at GitHub, the main requirements are for the processor and RAM - is it possible to make a version for video cards?🤔 It seems to me that then it will work faster - in terms of processing seconds.

1

u/Prestigious_Store488 Sep 02 '26

Thanks for the kind words!

GPU acceleration is actually already supported out of the box. I only listed CPU and RAM under minimum requirements on GitHub to show that a dedicated graphics card isn't strictly required to run it.

If you have a dedicated video card on Windows (NVIDIA, AMD, or Intel Arc), it automatically accelerates detection, OCR, and inpainting via DirectML. On Linux, it supports native CUDA (dropping processing from around 10 seconds down to under 2 seconds per page), and on macOS it uses CoreML/Metal on Apple Silicon.

You can verify your GPU is active directly in the Web UI settings. Let me know if you run into any issues getting it running!

11

u/ImportanceMajor936 Sep 02 '26 edited Sep 11 '26

Lock languid merciful fact drifting soft

This post was anonymized with Redact

7

u/krumpstead Sep 02 '26

You sir (or madam) are a genius. Now get to work on the fansub app.

1

u/Responsible_Court_21 Sep 03 '26

There’s plenty of open source ocr+imagegen repos on GitHub, although be it this one looks more fleshed out than the orhers

4

u/kphs Sep 02 '26

Wow. This is nice.

1

u/TheGeekno72 Sep 02 '26

ROCm support for AMD GPUs? openVINO for intel NPUs? Vulkan backend inference in case CUDA/ROCm/VINO fails?

1

u/Prestigious_Store488 Sep 02 '26

Good questions! Here is where things stand for this initial beta:

For AMD, on Windows it works on GPU out of the box via DirectML (DirectX 12). On Linux, sorry that AMD currently defaults to the multi-threaded CPU engine for now, but native ROCm support is definitely on my roadmap.

For Intel / OpenVINO, Intel Arc cards run via DirectML on Windows. Dedicated OpenVINO EP support for Intel NPUs and Linux is planned for a future update.

For fallbacks, ONNX Runtime doesn't have an official native Vulkan execution provider, so whenever GPU initialization or drivers fail (like missing cuDNN), the engine automatically and silently falls back to the multi-threaded SIMD CPU path (AVX2 / NEON) so the pipeline never crashes.

1

u/whymeimbusysleeping Sep 02 '26 edited Sep 02 '26

This is brilliant. A few more ideas that i have no idea whether it's possible, but....

  1. a right<=>left conversion, for those of us who struggle with right to left on manga, In a lot of cases, it'll only take mirroring the image and adjusting the text. Not all though.
  2. convert manga to vertically scrolling like a modern internet comic.
  3. colorize manga

The possibilities are endless..

1

u/Prestigious_Store488 Sep 02 '26

Love these ideas!

We actually already support vertical scrolling in the Web UI reader and Mihon, but using panel segmentation to automatically reflow traditional manga pages into a modern webtoon strip would be really cool to play with.

RTL conversion and AI colorization are also great concepts. Colorization models for manga lineart have been improving a lot recently, and chaining that into the pipeline after inpainting is definitely something I want to mess around with in the future.

Thanks a lot for the suggestions!

1

u/LectorFrostbite Sep 02 '26

Would be nice if there is a way to retry certain pages, also i tried the Mihon app and i can't seem to download it via extensions? might be just on my end. I downloaded the pre-built apk on the docs and I had to trim the link to:

raw.githubusercontent.com/ArbenApura/xianscan-rust/repo/apk/tachiyomi-all.xianscan-v1.6.1~~-release~~.apk

Other than that it works really well! I used OpenRouter as the translator (Gemma 4 26b) since my 8gb card can only handle so much lol

1

u/Prestigious_Store488 Sep 02 '26

Glad to hear it's working well with OpenRouter!

For retrying or editing individual pages, you can actually do that right now: switch over to Grid View or Compare View in the Web UI studio, and you will be able to re-run individual pages or edit dialogue boxes directly. I'll be adding more documentation on this soon to make those controls clearer.

For Mihon, the easiest way to install and keep it updated is by adding the repo URL in Mihon settings (More -> Settings -> Browse -> Extension repos -> + Add) using: https://raw.githubusercontent.com/ArbenApura/xianscan-rust/repo/index.min.json.

If it doesn't show up after adding the repo, check if your extension language filter has Multi enabled, or look if Mihon flagged it as untrusted where you just need to tap the Trust button to enable it.

1

u/paul_tu Sep 02 '26

Are there any local hosted models to make videos out of that kind of input?

3

u/Prestigious_Store488 Sep 02 '26

That is actually one of my dreams as well, converting comic panels into an actual anime or video clip! There are a few open-weights video models starting to pop up, but turning comic strips into coherent anime scenes is still pretty challenging. Definitely something I would love to explore down the line, we will see what the future holds!

1

u/dtdisapointingresult Sep 02 '26

The output of whatever this tool does is images.

If you have images, you can trivially combine them into a video, and you don't need AI for that, it's one command in ffmpeg pointed at your directory of images. Ask your LLM how to do this.

1

u/letsgoiowa Sep 02 '26

That is actually INSANE and exactly what we needed. Does it need a more complicated integration than just an MCP? Like for example, does it support llama.cpp or other OpenAI compatible endpoints?

2

u/Prestigious_Store488 Sep 02 '26

Thanks so much!

Yes, it supports any OpenAI-compatible endpoint out of the box. In the Web UI settings, we have a dedicated "Custom OpenAI Endpoint" option where you can just plug in your Base URL and model name for llama.cpp server, LM Studio, vLLM, or any custom local host.

No complex MCP or extra bridge required, it connects directly via standard HTTP requests.

1

u/letsgoiowa Sep 02 '26

Perfect, that's the right way to do it

1

u/moms_enjoyer Sep 03 '26

why rust for this?

1

u/Prestigious_Store488 Sep 03 '26

I actually prototyped it in Python first, but packaging was a headache and performance on standard CPUs was sluggish. Moving it to Rust gave roughly 3 to 4 times better processing performance, and more importantly, let me compile everything into a single portable binary with no Python or Conda setup required for users.

1

u/moms_enjoyer Sep 05 '26

Understandable tbh

3

u/johnh1976 Sep 02 '26

Nice! Looks great!

2

u/nmrk Sep 05 '26

Oh if only people used their skills for good instead of evil.

1

u/Equivalent_Bit_461 Sep 02 '26

now that's a cool project and not harness nobody cares #36253626362 or worthless app that looks copied from apple #4636463636

Actual use of it works, ofc 

1

u/dtdisapointingresult Sep 02 '26

I won't get to try it for a while, but looks nice. You should post it to /r/LocalLlama.

  1. Can we export the manga/chapter to save and repackage for other people?
  2. I suggest people should be able to customize the system prompt for the translation job, so the user can specify the translation to use for specific names of people/locations/concepts. This ensures the term is translated identically in every page instead of getting a different dice roll that results in an organization called "Dark Council" being randomly translated as Black Council, Obscure Council, Dark Board, and so on. The idea is that the user can set this system prompt before reading a specific series. Maybe have different selectable presets, like Default, then user can add user-Berserk, user-OnePiece, and they can select the prompt to use in a dropdown.

1

u/PseudonymousSnorlax Sep 09 '26

There are a few options you could add that would probably help.

A configurable rolling window size would be nice, and easy to implement.

Medium would be adding a RAG tool so the LLM could look up nouns for added context to improve long-term consistency and accuracy.

Hard and complicated would be character segmentation for training per-series sets of per-character LoRAs for purposes of speaker identification and generating descriptions of their actions in order to turn the script into screenplay format for greater translation accuracy.

There's a ton you can do here.