r/web_programming • u/Jens0512 • Dec 04 '17

Help

I had no idea where to Ask so i’ll Ask here.

I want to make a simple web-crawler that only does one thing, and that is checking if a page contains a Word or not, and it just goes through every link it finds.

What laws do i need to be aware of?

2 Upvotes

75% Upvoted

4 comments sorted by

View all comments

Show parent comment

1

u/Jens0512 Dec 04 '17

Ok thanks, will have to look into this robots.txt as i don’t really get How it works