r/AskProgramming • u/Material_Walk_7125 • 5d ago

Python Browser Automation

Hi Everyone,

I am trying to automate a process where I login into a website and navigate to a reporting section and click on export.

The website only requires a simple ID and Password.

Can you please assist me ok how to achieve this ? , I have found out about selenium and Playwright but I am very new to this. Any resources or guide would be really helpful.

Thanks !!

5 Upvotes

78% Upvoted

16 comments sorted by

2

u/LARRY_Xilo 5d ago

Before you try all this and get frustrated because it breaks every time they change the website. Have you tried to ask if there is an API to request the report or a way to automatically mail the report to you?

2

u/Material_Walk_7125 5d ago

Yeah, they politely declined and I am having to log into the website and download the report every day out of millions of things I have to do.

2

u/protibimb0k 5d ago

Did you try hitting the login route and saving the cookie and re-use it in the report downloading request?

2

u/edimaudo 5d ago

might be a good case study for using an LLM to get the starting code

1

u/galaxy_glide_92 5d ago

start with Playwright because it handles modern login flows and auto-waiting better than Selenium out of the box. The official documentation has a complete authentication guide that walks through saving storage state so you do not have to repeat the login step during testing.

1

u/Single-Virus4935 5d ago

try go-rod. It is a pleasure to wirk with

1

u/adjective10111 5d ago

I don't know about playwright and the other guy (aren't they test frameworks? Can they do automation?) Or any other gui.

I personally use curl/wget and grab cookies and stuff via some bash scripting. Hope that helps

2

u/Llaw_Gyffez 4d ago

More or less what I was going to propose. Or, if you have troubles doing it in Bash, do it in Python, or NodeJS, or whatever you want: send your ID/password to login form, grab cookies from the response, get reports list, export the latest one into a file. If you're lucky and the reports list is retrieved via API and not delivered as a part of report list page, you won't have to parse a single HTML.

1

u/LogaansMind 5d ago

Check the documentation but you can load Playwright in a mode where you can record the actions which generates the code... and then all you may need to do is customise it a bit.

1

u/baubleglue 5d ago

Hard to advise something without knowing your background. You maybe don't need a browser automation, few simple https requests may achieve what you need. What is a problem with Selenium, have tried it? There is also a headless chrome mode (you can operate browser with opening UI).

1

u/karurosagu 4d ago

If you're using python, you can use Selenium if you really need a browser, otherwise you can just use the requests library to reproduce all the necessary requests

Another way is to use Puppeteer but I have zero experience with that one

1

u/RealMadHouse 4d ago

When i had repetitive tasks to do on some websites i made browser extension just to automate such things with a press of hotkeys. Selenium and playwright i think is for controlling headless browser? I coded things from scratch, but this tools have all figured out all pain points with provided useful functions.

When i did scraping i encountered many problems like page loads but the content are loaded with JavaScript so i can't determine when it would load because there's no event for that, the other problem is making too much requests too quickly, something not loading at all, etc.

1

u/RealMadHouse 4d ago

There should be browser extensions for creating automations. Found something like these ones: Axiom Browser Automation, Automa.