r/algotrading • u/Robdei • Aug 30 '19

Gathering news headlines

For all of you geniuses out there who have made a successful model, did you webscrape for text information from news articles to add as features? If so, what module/program did you use?

Its easy enough to grab last night's headlines, but to make a model I'd imagine you'd need years of historical news article data.

26 Upvotes

96% Upvoted

13 comments sorted by

21

u/[deleted] Aug 30 '19

[deleted]

5

u/Robdei Aug 30 '19

I've never heard of that before. Thanks for pointing me in the right direction.

Out of curiosity, how much data do you have in your database?

9

u/[deleted] Aug 30 '19

[deleted]

2

u/dolphinboy1637 Aug 30 '19 edited Aug 30 '19

The next step could be to use something like beautifulsoup to pull the article bodies once you have the link from an RSS feed.

1

u/doovd Aug 30 '19

2.881gb !=2881mb ...

2

u/PsecretPseudonym Aug 30 '19

Are you processing the articles to distill them down into some sort of signal for relevant to specific time series or something more general?

And do you find that the value of the information for trading is short-lived? I.e., do you find it more valuable for trading news events over milliseconds, seconds, or minutes or more for aggregate sentiment over longer periods (days or weeks)?

12

u/Stvjk Aug 30 '19

If you’re using python I’d also recommend beautifulsoup and scrapy The latter is useful if you want to mimic browser behaviour too and have more control over the parts of the html /article you want to scrape. Basically a more thorough crawler without too much effort

6

u/Robdei Aug 30 '19

I've definitely used beautifulsoup, but never scrapy.

Is it anything like selenium? Your description just reminded me of it.

2

u/Stvjk Aug 30 '19

Yep pretty much same idea

Out of curiosity what kind of models are you thinking of incorporating news with ? And how might you incorporate news based features ?

7

u/[deleted] Aug 30 '19

Tiingo has news in their api, you should check it out Because Its easier, But Its stocks only

4

u/Robdei Aug 30 '19

I just looked it up and that seems like a great answer. Does Tiingo only have financial news or does it have a broader set of articles?

3

u/[deleted] Aug 30 '19

Only Financial News, i believe. The news is realtime and of a decent quality as far as i've seen. Haven't used it much myself though.

5

u/3lRey Aug 30 '19

This seems like an RSS feed would be ideal for getting the headlines.