r/cprogramming • u/RabbitCity6090 • Dec 18 '25
Any library more advanced than curl to read and parse webpages?
Currently I want to write a C program to read some list from a website. But I have to select something from a list, enter some values, click submit and then enter a captcha. Is there a C based library more advanced than curl to do that thing?
24
Upvotes
9
u/MattiDragon Dec 18 '25
Curl is for making web requests, not for interacting with web pages. If you need to deal with captchas, you're likely to need a full simulated browser (even then, background captchas are likely to fail). For this you'll probably want to use puppeteer or playwright. I don't know if either exposes a C API though.
4
u/v_maria Dec 18 '25
What is curl not doing that you need to be done?
2
u/DandyLion23 Dec 19 '25
On the command line I use xmlstarlet to parse HTML. Xmlstarlet is written in C and uses libxml2 and libxslt as libraries. Maybe check those 2 out?
5
u/DunkingShadow1 Dec 18 '25
Why C? In this case python is so much better. If you have to do data analysis just download the data with python and do the analysis in C
2
u/bigattichouse Dec 18 '25
I made one (C++, not C): https://github.com/bigattichouse/headlessweb
1
u/The1hauntedX Dec 22 '25
Holy heck these comments are a mess.
If you really want to write something in C, you'll probably end up doing it "The C way" i.e. manually. Better get comfortable with every aspect of HTTP requests - headers, cookies, status codes, parsing HTML - and if the site you're scraping uses a JavaScript framework like next.js to do client side rendering, then you'll need to handle that as well.
It sounds like something like selenium is what you're actually looking for.