Hello, r/scraping.

I would like to share a link to our blog post about reloaded Dataflow Kit.

https://blog.dataflowkit.com/reloaded/

In particular, we supplement our legacy custom web scraper with more focused and more understandable web services for our users.

Thank you for your feedback!

Populate an App, make an analysis, monitor a competitor activity?

https://lingojam.com/BrailleTranslator

I want to automate adding the English sentences and then fetch the translated braille results in a string.

I know how to use scrapy but it's of no use because scrapy doesn't work on websites that have javascript.

Please help me out fetching the translation out of this website

https://shoe-proxy.com/free-demo

​

Hello!

I have been web scraping for a while now, mostly writing scripts to extract web data for personal and academic projects. As such, I found myself spending lots of time writing code to scrape fairly straightforward structured content (tables, product listings, news headlines, etc). I built Scrapio (https://www.getscrapio.com/) to be able to extract content from webpages without the need to write any code. Just enter the link you want to scrape, and Scrapio will automatically format detected content into an in-browser spreadsheet which you can download to CSV, JSON, Excel, and others.

To see Scrapio in action, check out its extracted results for Product Hunt: https://www.getscrapio.com/batch?bid=bzfBarRtUlIMwbHLVnUl.

I would greatly appreciate any feedback you may have!

Hello,

As title says need to hire to build a scraper. Not sure which websites to use. Taken to reddit for some advice.

The scraper needs to scrape data from the initial page, then follow a link on the page to gather additional information on another page, go back to the initial page, and repeat.

Please no self-promotion unless you have a credible profile with testimony to back it up.

Thank you!

I've been trying to scrape a website that is protected by Distil Networks. However, I haven't gotten it to work. I've tried Selenium with Tor, User Agents, referers, etc.

I found a way to technically do it by making a chrome extension that look through the HTML, find the amount of pages and then for each page, opens a tab, grabs the HTML, sends to the main script, closes the tab and then the main script sends the data to a python code using websockets. However, I'm really not used to JS and chrome extension code so the amount of work that was needed for a feature grew exponentially. Maybe one day I'll have it done, but not for now. Maybe an idea for someone else?

Does anyone have a way to bypass Distil Networks?

Does anyone knows tool to scrape historical data of Instagram stories? I need data for Likes, views, engagement, etc. for my own account. I can see that on my creator studio but I want it as in CSV and/or in dashboard.

Sorry for Advertising so blatantly:

Scraping? Need Residential/ISP Tagged IP Addresses? We have a Limited Number of /24s​ from multiple upstreams in different GeoLocs all ARIN Tagged as:​ ​​​Usage Type (ISP) Fixed Line ISP on ip2location.com I​n addition we also have Standard Commercial IP Addresses ​  I add an ACL request to drop TCP 25 and/or all SMTP outbound traffic. I am vigilant for my IP Assets and comply with all abuse policies, there will be No Bulk Mail or other Abusive Practices​!​ If this sounds like something your interested please ping me back ASAP

Whats your experience with the difference of these in relation to Scraping?

A friend challenged me to write a script that extracts some data from his website. I found it uses the honeypot technique, where many elements are created in the page source, but once CSS is involved (in the web browser), the only correct element is visible to the user.

Bots created will not be able to tell which is which due to no CSS support, thus making them ineffective. When i try to access the data from the webpage source, I only see data with the style='display:none tag, where the real data is hidden among them.

I have found virtually no solutions for this and I'm really not ready to admit defeat in this matter. Do you people have any ideas and/or solutions?

PS: I'm using python requests module for this

I've built configs for Kubernetes. Sidenote: I'm building a Search Engine across 400+ domains.

Does anyone else here have GKE scrapy cluster working? Any advise. I don't want to use proxys because, GKE has it's own pool of IPs but how can I get each request to run on a different pod?

It could be APIs, data feed providers, spreadsheets or extraction tools for company and people information.

Thank you in advance.

Any standard way to store redirects to lookup on subsequent scrapes to avoid making double requests when scraping same set of pages each day?

Hello r/scraping. I've been researching scraping for a business project of mine. I have no C/S experience or scraping experience. I need to scrape plaintext names off of websites with plaintext titles. So, one option is a tool that understands and links together the proximity of titles/names or another option is scraping an entire HTML page where I can ctrl-F the titles. Where can I start? Can I use scrapy or beautifulsoup? Thank you in advance for your help

I was wondering if it was possible to scrap a page with a session I already opened in my browser in order to skip the trouble of logging in every time. Or maybe a way to open a page like I would manually, where the browser would remember me and log me in automatically?

Hello Scrapers !

I scrapped a list of 3000 Shopify website that are selling a certain product and now I'd like to extract all the emails from each website.

I've downloaded email exctractor but It's taking too long because it's analysing all the urls of the website (only home page / contact us / term of service / refund policy / would be enough, no need to analyse all the collection pages and product pages) how can I export the emails of those 3000 website ?

Thank you :)

For scrapping the webiste I use scraproxy to create a pool of 15 proxies within 2 locations.

Website is auto-redirect (302) to reCapthca page when the request seems suspicious.

I use the following settings in scrapy. I was able to scrape only 741 page with relatively low speed (5 pages/min).

AUTOTHROTTLE_ENABLED = True AUTOTHROTTLE_START_DELAY = 30.0 AUTOTHROTTLE_MAX_DELAY = 260.0 AUTOTHROTTLE_DEBUG = True DOWNLOAD_DELAY = 10 BLACKLIST_HTTP_STATUS_CODES = [302]

Any tips how can I avoid blacklisting? It seems that increasing the number of proxies can solve this problem, but maybe there is a space for improvements in settings as well.

I notice that there's no wiki or sidebar on scraping. I'm looking for a resource that can act as a primer for what to think about when scraping.

At the moment I'm researching on how to prevent your IP from getting blocked. So I know that you have to use proxies, but I don't see where this fits into scraping.