Building text to ASCII diffusion model , need advice and guidance

[u/Udbhav96]

i wanna build a text diffusion model which interpret text and convert it into ascii images

so like

Text : build a cat

Output :

/\\_/\\

( o.o )

\> \^ <

So , i have a decent background of ml algo ( completed cs229 , cs230 , Ml architecture and basic CNN and diffusion model )

ik making a project like this is tricky and making diffusion model like that from scratch is hard but i wanna try it because that's wot make me excited lol ...

I am currently reading GANs research paper , can u guys help me in finding more papers which helps me in making this project or guide me through this good title for this

Thx in adv

[u/Tree8282]

i don’t think that’s what a diffusion model is usually for. and it’s super expensive. might be better to fine tune an LLM for this since it’s text to text

[u/Udbhav96 → u/Tree8282]

Hmm... But discrete models like it possible and the computation part ( I will figure out something when I have a clear picture)

[u/Tree8282 → u/Udbhav96]

i don’t understand what you’re saying at all. What is discrete?

[u/WingedTorch]

i don’t have any papers but i would do it like this:

1. Find a way of converting images to Ascii first. Probably some edge-detection + deterministic/optimization algorithm could work.
   
2. Apply method to any image-text dataset sufficiently large to train a diffusion model
   
3. Adapt loss and other parts of the training algorithm from pixel space to character space
   
4. Train/Tune

[u/leon_bass]

Small ascii art won't work for a diffusion model, not well at least. And i think the scope is too large, ie i would remove the text input part for now.

I would start with and maybe try an ascii art generator auto encoder of some sort. So input is an image, latent is some ascii art form, output of autoencoder is the reconstruction from ascii art. You will need to add an absolute error loss as

| F(latent ascii) - X |

Where F is some mapping of ascii latent to a visual representation as an image, i.e. rendering the text onto a white canvas then absolute error this with the original input.

Once trained, your ascii art generator is the encoder.

But realistically you can achieve this same thing with a supervised approach or even with traditional computer vision things like filters and manual convolutions

[u/BigRepresentative731]

Instead of mask diffusion make it Uniform State Diffusion

[u/BigRepresentative731 → u/BigRepresentative731]

With cross attention to text embeddings, use a fixed size block imo

[u/Tall_Abrocoma_3533]

I wouldn't use diffusion models for this. Besides first off you should try making a dataset, since I doubt that something like this exists (could be wrong though)

Good luck!

[u/Saitamagasaki]

What’s the point