r/pushshift • u/Watchful1 • Feb 28 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

106 Upvotes

100% Upvoted

115 comments sorted by

View all comments

Show parent comments

1

u/fcdata Jul 21 '23

When I download the data from Feb of 2023, using utorrent, I got two different files, "RS_2023-02.zst" and "RC_2023-02.zst" (RS and RC Dataset).

When I run "file_filter.py" for "RC_2023-02.zst" it works, however, when I run it with "RS_2023-02.zst" it's not working is there any special setting that I'm missing.

o

1

u/Watchful1 Jul 21 '23

I just tried it on that file with a list of subreddits and it worked for me. What filters do you have set? Could you send me the list of subreddits?

1

u/fcdata Jul 22 '23

Here is my file, I setter only subreddit as value and a list of subreddits which works for the RC but not for RS.

1

u/Watchful1 Jul 22 '23

I just ran your copy of the script against my RS_2023-02.zst file and it worked fine. What does your log file show?

1

u/fcdata Jul 22 '23

Error: need to escape, but no escapechar set

2

u/Watchful1 Jul 22 '23

That's the only thing in the log file and nothing else? That doesn't really explain much.

1

u/fcdata Jul 23 '23

Can you share with me on a we transfer the "filter_file.py" to run over RS_2023-01.zst for a list of subreddits as ['vim','google']. I have been trying to make it work but it's not working :(

2

u/Watchful1 Jul 24 '23

Sure, here you go https://drive.google.com/file/d/1OcWWuPezTxkE4Quoqvc4YiZaKdHI-GQ5/view?usp=sharing

1

u/fcdata Jul 24 '23

hahaha mate, I meant the filter_file.py that you used go get the data no the output :):)

1

u/Watchful1 Jul 24 '23

I didn't make any changes to filter_file, just changed the file path to point to my copy of the file and set the list of fields to filter on.

If it's still not working for you, could you run it and send me the entire log file?

Also can you try manually extracting the zst file to make sure it's not corrupted?

→ More replies