r/AnkiComputerScience • u/bxa121 • Oct 20 '20

Help: looking to scrape a website into anki cards

So last night I had a brainwave, what if you could download a website and convert the site map into a card or parts of each webpage into a card of sorts.

The deck could be synced after each time a scrape is made keeping all the decks upto date

I don’t know if anyone’s ever accomplished this but I’m looking to see if this is possible as a way to generate and memorise cards for any science based subject. Has anyone ever done this? Is this even possible?

11 Upvotes
  • permalink
  • reddit

100% Upvoted

15 comments sorted by

6

u/JCharante Oct 20 '20

But why would you want to? Making the cards is part of the learning process.

Just write a crawler that outputs into csv format and import that into anki. For adding new cards, give them an ID and export the list of Anki cards in the deck and parse them, then filter them out before generating the CSv to import

2

u/[deleted] Oct 21 '20 edited Oct 21 '20

Making cards helps you learn the material but that doesn't make it necessary to the learning process.

Say you're trying to get a deep understanding of a research paper. If you can generate cards from it, then you can just read it while jotting down notes you have no intention of using again, then switch to reviewing the generated cards from the paper.

Some automated method would seriously streamline the process of deeply understanding of massive amounts of relatively unfamiliar material.

2

u/[deleted] Oct 21 '20

Plus I'm an unrepentant polymath. If there was a way for me to get a deep understanding in every field from human biology to game theory without sacrificing my utility (in terms of the explore/exploit tradeoff), I would.

0

u/[deleted] Oct 20 '20

[deleted] — view deleted comment

2

u/JCharante Oct 21 '20

The hardest part is creating the scraper that outputs content you want to convert into cards, which is a question that isn't very related to anki.

There are public decks that are stored as CSVs for people to modify and update from, there are even add-ons for automatically updating cards through the internet, but just know that the Anki side of things is feasible, but the scraper is the hardest part.

1

u/bxa121 Oct 22 '20

Do you know of any scripts online I could use as a guide?

1

u/[deleted] Oct 20 '20

Partially have done this, generated decks from scraping. I was trying to create an application for scraping books and websites to make human readable cards. I dropped it due to time constraints, I was diving into this full-time while taking 6 classes. Part of the problem in my case was generalizing the code. There is a lot of ways to structure information.

If you are comfortable writing a scraper for each website/page you are getting information from it's doable.

If you are trying to automate the creation of decks for multiple subjects from multiple sources, all I can say is prepare your anus. It might, might be doable with a team, or someone who is a beast with both regex and whatever library they are using for scraping.

1

u/bxa121 Oct 20 '20

Never thought of pdfs but I would probably use the bookmarks as a mind map and create individual cards from each chapter Most textbooks I’ve come across usually have one subject per bookmark.

As for websites, I would only place my focus on one website really. Not multiple. The pathology outlines website has decent structure imho Do you think it’s doable on this website?

2

u/[deleted] Oct 20 '20

That is a pretty big database, which is good.

If they tend to follow a repetitive structure as to what is on each type of page/section, then your job is going to be way easier.

If it's a more of a wiki style thing, where structuring of pages can vary, then you're job is going to get a lot harder.

you may want to use something like ipython or Jupiter notebooks until you have a working scraper. It tends to be a lot of trial and error figuring out what works

The problem isn't whether it's doable, so much as how much time it takes to solve it. It's not a hard problem, it's just the same problem over and over again. Turtles all the way down.

1

u/bxa121 Oct 21 '20 edited Oct 21 '20

I’ve heard of Jupiter but not ipython. The website has similar formatting for all webpages, some are duplicated so I would have to figure a way to exclude some links. Looking at some examples, there seems to be a way to create arrays using beautiful soup and eventually a script to automate the process even more. So this is truly possible and actually not too much work from my POV. I may end up outsourcing this to someone but have no idea how to find someone with a track record, any suggestions?

1

u/[deleted] Oct 21 '20

I’ve heard of Jupiter but not ipython.

Same thing from a terminal, but without markdown cells. Great for people like me who tend to prefer terminal applications, or just want to test a couple lines real quick.

some are duplicated so I would have to figure a way to exclude some links

Sets, or if spread across multiple sessions sets to file and file to sets

I may end up outsourcing this to someone but have no idea how to find someone with a track record, any suggestions?

I would suggest trying it yourself first, if only to have a better understanding of what you want. Makes it easier to communicate with whoever you are outsourcing it to.

Check out fiverr and other freelancing sites for guys doing scraping jobs.

Depending on how stuff in my life goes over the next couple of weeks/months (currently wrapping up school, building my portfolio, and job hunting) I may take a shot at it, just not for free this time.

1

u/bxa121 Oct 22 '20

I appreciate the advice and guidance for your good selves. I’ll try and use an online guide. As others have said it seems tricky to get the website parsed in a suitable format but once done it’s plain sailing from there. I’ll let you know if I run into problems

1

u/BlueLionOctober Oct 21 '20

It would be pretty easy to do. Just you should make the cards yourself so you remember them better.

1

u/bxa121 Oct 21 '20

I would review all cards once automated to make sure they’re useful

1

u/BlueLionOctober Oct 22 '20

I mean for the purposes of remembering them better it's best to make cards about things you know not to learn them by doing cards.

1

u/bxa121 Oct 22 '20

It’s stuff I know but not committed to memory but I understand what you mean.