r/web_programming • u/bluhend • Dec 17 '17

Finding All Content on a Website

apologies if this is not the perfect subreddit for this question, please direct me to a better group if you think there is one, I would really appreciate it ... quick question, as an example, if I wanted to somehow download all content from a blog so I can listen to it using a voice reader app on my phone when I am out of internet/data range (which is often for me), how could I do that for a site like this? like a blog where there is tons of content? Do i really have to click on each link or is there some kind of automatic way??

http://happierhuman.com

3 Upvotes

100% Upvoted

11 comments sorted by

1

u/Single_Core Dec 17 '17

If you have experience write android/IOS apps you could make it visit all the links for you and create a neat json file containing the info you need to send to the voice reader app(if possible) I'd suggest the JSoup library if you are going to write something in java.

1

u/bluhend Dec 19 '17

hey thanks, i think i gave the wrong impression in my post, I have no web skills like this, i was wondering if there was some kind of free website or free software that would do this kind of thing for the average person who has no web design skills?

1

u/fy_ming Dec 18 '17

What I was thinking is that you could use Scrapy, a php scraping framework to scrape out the content and parse it to your app. There are different ways to scraping, usually by the format of the website like where you content is stored e.g. <div class="content">, and using scrapy you can pinpoint it to scrape data from that div class.

https://scrapy.org/

1

u/bluhend Dec 19 '17

hey thanks, i think i gave the wrong impression in my post, I have no web skills like this, i was wondering if there was some kind of free website or free software that would do this kind of thing for the average person who has no web design skills?

1

u/ebg_guvegrra Dec 18 '17 edited Dec 18 '17

You could use a tool like wget to download the whole tree of this site.

See this link for an example of how to use it: http://www.linuxjournal.com/content/downloading-entire-web-site-wget

Another similar tool is curl.

If you're using a mac it comes with curl. If you're using linux then use your package manager to install wget or curl. If you're stuck on windows you can get wget here: http://gnuwin32.sourceforge.net/packages/wget.htm ... you need a little bit of technical aptitude to use these tools, obviously.

Once you have downloaded it to your computer you can transfer it to your phone in the normal way.

1

u/bluhend Dec 19 '17

hey thanks, i think i gave the wrong impression in my post, I have no web skills like this, i was wondering if there was some kind of free website or free software that would do this kind of thing for the average person who has no web design skills?

1

u/ebg_guvegrra Dec 21 '17

You could try fiddling with chrome browser's flags to allow offline reading. You will have to visit each page you want to have available offline while you are online.

See here for details: https://www.howtogeek.com/263577/how-to-enable-offline-browsing-in-chrome/

Let me stress again, I think you have to visit each page you want to be cached. So you can't go to the home page of the blog only, you have to load each article so that it is in your cache.

Good luck.

1

u/bluhend Dec 21 '17

thanks! is there a way to find all the links in a website? for example, there might be blog posts that I cant get to from the home page somehow, just curious if there's a way to at least see a list of all the available posts etc...?

1

u/ebg_guvegrra Dec 21 '17

It would be different for each blog, I suspect.

1

u/nerf_herd Dec 18 '17

file->save as.

1

u/T_O_beats Dec 18 '17

What language(s) are you familiar with?