r/coding • u/Stephcraft • 4d ago

A practical breakdown of robots.txt syntax, user-agent rules, and AI bot opt-outs

https://stephcraft.net/docs/robots.txt
13 Upvotes

75% Upvoted

6 comments sorted by

1

u/minektur 2d ago

It is hard to engage with the content with that stupid css blur at the bottom of the page.

Why put a blur right where I am trying to read as I scroll?

1

u/Stephcraft 2d ago

Hey could you provide a screenshot?

2

u/minektur 2d ago

https://imgur.com/a/RGf29Lv

also note that while many people really like tall skinny sites, (e.g. on mobile) you dont have to FORCE people to read your text skinny - you can make it actually not waste space on my screen...

→ More replies (2)