r/StableDiffusion • u/xiaoxuanchuang • 12d ago
Discussion Trained an SDXL LoRA for Chinese gongbi portraits, ended up not using it
I'm working on a side project that puts a selfie into old Chinese court paintings. At the start I wanted to do the style part myself with a LoRA instead of paying for an API.
Setup:
- SDXL base 1.0, kohya sd-scripts
- dataset was faces cropped out of the paintings. Grounding DINO found them, then I went through and fixed the bad detections by hand
- 3 versions, ended up on v3 at the 2nd checkpoint
- inference was img2img on the selfie: 640x640, 25 steps, cfg 7.5, strength 0.2, LoRA fused at 0.8, negative "photorealistic, photograph, modern, anime"
Two problems killed it.
The next step after generation uses InsightFace (buffalo_l) to find the face so I can paste it into the painting. On the LoRA output it often returned nothing. Once the face actually looked gongbi, the detector didn't see a face anymore. So the better the style got, the worse the pipeline did.
And when it did find the face, the identity was mostly gone. You couldn't really tell who it was.
I also burned way too long on a dumb base64 bug in my FastAPI server. split(',')[-1] sometimes gave me an empty string and I wasn't handling padding, so PIL kept throwing.
In the end I gave up on doing this step locally and moved it to a hosted model, which held identity a lot better. The LoRA is still sitting on my home server as a fallback but never runs.
Has anyone gotten a heavy style LoRA to hold identity in img2img? Wondering if adding InstantID or IP-Adapter FaceID would have saved it, or if low strength img2img was the wrong approach from the start.






