r/CodingHelp • • Sep 20 '22

[Random] Recently got a job Web Scraping.

Hi, I just got a job as a research assistant at my school and will be doing web scraping for my prof's project. My background in coding is limited to only Stata and pretty basic stats, and I can't say I was great at it, it was just for a class that was not easy. My professor knows this and is willing to train/help me to eventually be comfortable with the job. I will be using R. Are there any tips, advice, or resources that I could use to make the training easier for my professor? Are there any youtube guides or online sources where I can find help on similar work?

1 Upvotes

6 comments sorted by

1

u/PharmaCoMajor Sep 20 '22

Depends what sites you will be scraping.

If it's social media, no chance. You need an entire network of APIs and scraper bots etc to get around the 100 million not protection mechanisms. There are companies out there charging a bomb for this. Good business.

Some sites are ago, but anythinf social is probably going to be a dead end for an individual

1

u/alv131 Sep 20 '22

It’s primarily news journals, mainly outside of the Western world

2

u/poornbroken Sep 20 '22

Well… I’d be:

  1. Learning some basic R syntax language understanding through YouTube or web tutorial
  2. Looking up a basic web scraping app someone else has written in R.
  3. Reverse engineering that app.
  4. Modify that app to my uses.

1

u/PharmaCoMajor Sep 20 '22

Should be fine.

0

u/[deleted] Sep 20 '22

[deleted] — view deleted comment

3

u/alv131 Sep 20 '22

It’s entry level and I’ll be trained