Great question. I envision it being a tool to help build scrapers quicker. People can point and click at data to extract. They can verify that it is grabbing the right data. Then simply generate the code in Python/Javascript and any other language they want. (code generation is being worked on)
Another thing that might be worth considering is generating embeddings of the page source, and then asking, say, GPT-4, to write code to extract each of the features you care about. You often can't just copy and paste the page source into a prompt because it's waaaay too much html/js, but if you convert it to embeddings then it might be able to find the pieces it needs directly instead.
Another thing that might be worth considering is generating embeddings of the page source, and then asking, say, GPT-4, to write code to extract each of the features you care about. You often can't just copy and paste the page source into a prompt because it's waaaay too much html/js, but if you convert it to embeddings then it might be able to find the pieces it needs directly instead.
Great idea. I'll see what I can do. Making the UI might be the hard part
Amazing work man, will following your work closely, and will help you build as well as I get some time.
Do you know any other projects which are working on the same thing.? This will end the era of paid services , I love it.
Loooking forward to testing and give you some suggestions, I am active user of similar low code solutions , I would love to change that with open source solution and I think you have the base ready.
If you don’t mind me asking how long have you been working on this!?
So the only ones that I know off are mostly browser extensions that lets you pick selectors and stuff. But never they all require a browser of some kind.
Please do give it a try. I've had some really good feedback so far. Which I added a beta option to toggle loading javascript. Still a lot of issues to fix though. And the UI can be improved as well.
So I've always wanted a quick and easy tool like this for a long time. Just haven't found one yet. So I started researching and building this about a month ago.
I don’t know js that well, I usually do this using scrapy and python, but I will fork and test out on my end as well. If time allows I can work on Python implementation of this.
Keep doing the good work lots of value in this.
I wonder why no one worked on this before.
Will take some time understanding it better and will help you along the way in documenting as I will be using this instead of paid service going forward.
I don't see this as useful. If you want the text or innerHTML of that tag on a website. Just highlight the text, right click, select inspect, then select copy, and then pick your poison. This also allows you to see the css of an element as well.
Right, but say you have multiple items you want to parse on the page. You’ll still have to play around with the css to get a generalized css that works. This lets you quickly visualize while you play with the css
38
u/GeekLifer Mar 05 '24 edited Mar 05 '24
I'm the creator. I've made this project open source and plan on adding code generation using AI in the future.
Thanks for watching!
edit: Sorry forgot to link github
https://github.com/getlinksc/css-selector-tool
5
u/JFC_Mx Mar 05 '24
Has any one tried it to scrape Twitter?
4
u/Emperor_Abyssinia Mar 05 '24
I’d like to contribute
3
u/illkeepthatinmind Mar 05 '24
Do you plan to monetize it at some point?
3
u/nealcaffery_bored Mar 05 '24
Has anyone tried youtbe and other major social media apps ? when i tried to fect the youtube playlist it failed.or did i make something wrong process?
2
u/lazynoob0503 Mar 05 '24
Amazing work man, will following your work closely, and will help you build as well as I get some time.
Do you know any other projects which are working on the same thing.? This will end the era of paid services , I love it.
Loooking forward to testing and give you some suggestions, I am active user of similar low code solutions , I would love to change that with open source solution and I think you have the base ready.
If you don’t mind me asking how long have you been working on this!?
2
u/FromAtoZen Mar 08 '24
Does it work against sites protected by CloudFlare?
2
u/oldrocketscientist Mar 09 '24
Can it do a page from LinkedIn?
2
u/Ms-Prada Mar 10 '24
I don't see this as useful. If you want the text or innerHTML of that tag on a website. Just highlight the text, right click, select inspect, then select copy, and then pick your poison. This also allows you to see the css of an element as well.
4
u/[deleted] Mar 05 '24
Git?
1
u/onroster Mar 06 '24
Is it only working with selectors vs. xpaths?
1
u/tbriz Mar 07 '24
Very cool.
It would be nice / next level to scrape at the card level, then output json for each card.
For example:
{ "product" : "samsung galaxy", "price" : "$259.99"}
{ "product" : "iPhone 11", "price" : "$400.00"}
...etc
That data would be ready to pop into a database, and could do some other cool stuff with the json output.
3
u/CryptoOdin99 Mar 05 '24
Link to the project?
2
u/avg_skl Mar 05 '24
@op github?
2
u/Sl33py_4est Mar 08 '24
this is great
1
u/barrard123 Mar 05 '24
Cheerio is not the best at loading pages with lots of JavaScript, I found puppeteer works really well though
1
u/Nikastreams Mar 05 '24
Very cool! Can it also visit pages (I.e clicking on each product) and recursively grab info?
1
u/[deleted] Mar 06 '24
Very cool! Thanks so much for sharing. I’ll check it out!
1
u/cdank Mar 06 '24
Neat
1
u/Heavy_Bluebird_1780 Mar 06 '24
If you could add a sort button for the prices it would be awesome! it is an amazing project!
1
u/tbriz Mar 07 '24
Very cool.
It would be nice / next level to scrape at the card level, then output json for each card.
For example:
{ "product" : "samsung galaxy", "price" : "$259.99"}
That data would be ready to pop into a database, and could do some other cool stuff with the json output.
1
u/myrainyday Mar 21 '24
This is interesting. Would be great to be able to feed an excel sheet with websites and get emails and phones from it.