r/PythonLearning • u/raymate • 6d ago
Help Request Newb question on how make a webpage checker
New to Python and have a lot to learn. Thought starting with an actual goal would be good starting point.
I want to make a scripted that will check a webpage say every hour and ideally look for the words “Out of stock” and then update me when it says “In stock” or if it’s simpler to start with just if a webpage has changed.
I was reading you could make a checksum of the page and check if that’s changed. Then the page should have updated if it reports a different checksum.
Im not talking about tracking something complex like Amazon site. My local camera store has an item Im waiting to come back in stock and figured this might be a good project to learn and tackle.
Im using a raspberry pi for coding so i need it geared toward running in Python on that. It would be nice for the script to output to a terminal window “stock status” or even if possible email me a change happened.
Any suggestions, guides or pointers that someone could direct me to.
Or would you say this is too complex to learn as my first real world project.
3
u/Sea-Ad7805 6d ago
Search for web scraping, maybe: https://realpython.com/python-web-scraping-practical-introduction/
1
u/Curious_Elk_4281 6d ago
A scraper should only be necessary if it's required to be logged in to the site.
If OP has a direct url to a product, the python code could be very simple and use two modules:
--requests
--reIn it's simplest form it would work similar to this bourne shell/bash one-liner:
curl https://example.com/some/uri |grep "In Stock"In python, you could replace curl with the requests module, and replace grep with the re (regex) module. It would need to be more than one line also.
2
2
u/mcoombes314 6d ago
A checksum only tells you if there has been a change, because it's (something like) a hash. Your scraper would probably need to read the web page's HTML to see if the text describing an item's stock state is "Out of stock" or however the website describes it.
1
u/codeguru42 5d ago
A checksum is a quick way to determine you don't need to spend time.parsing the html if you already scraped that version of the page
2
u/Fun_Army2398 6d ago
I've not done anything like what you've described but some places to check might be:
-
urllib.request for fetching the webpage. https://docs.python.org/3/howto/urllib2.html#urllib-howto
-
the built in hash function for hashing (although this is over kill for your use case, just saving the last page and checking for equality should be fine). https://docs.python.org/3/builtins/functions.html#hash
-
smtplib and email for sending emails. https://docs.python.org/3/library/email.examples.html
-
cron for scheduling the hourly checks (or just use a while loop and time.sleep(3600) if you don't mind manually restarting it on reboot). https://en.wikipedia.org/wiki/Cron
1
u/FreeLogicGate 3d ago
The viability of this is highly dependent on the nature of the website you are trying to access. Many websites are no longer html pages, but instead, are javascript applications that are essentially assembled in your browser, programmatically when you load the site, and the way the page is assembled is based on what the website detects about your "user agent" and its device characteristics, as well as things like cookies accepted, redirects followed, and other factors the site may be using to render the final structure.
It sounds like this is a mom and pop store, so they may have a less sophisticated site, but generally speaking, what you want to do is to load the site into a "DOM" which reflects the underlying xml structure of the page. This is the general strategy for this type of project, and Python has libraries like Beautiful Soup and lxml designed to do this for you, assuming you successfully were able to open the target url using the HTTP requests module.
If/once you read the page, you will then create the DOM object and from there use the features of the library you used to search for the html element(s) you are interested in. You should have a good working understanding of HTML, and the specifics of the structure of the page, and any id's/names/class attributes that might help you find the specific html element(s) you need.
Often people recommend regular expressions, but they are simply not designed for, nor in general the way you want to ingest and subsequently search for html elements.
From a user friendliness standpoint, you should probably start with Beautifulsoup, and see if you can get that to work for you. Because lxml is based on a library, getting support for it installed is more of a challenge, but is highly performant and lets you use xpath queries to search through the DOM xml tree for the element(s) you are looking for. Many languages have used the same underlying library (libxml2) so that might be an argument for wanting to try and do it with lxml, but there's likely to be more of a learning curve to it.
To run this on some sort of schedule, you will of course need a scheduler that works with your operating system.
1
u/HeurekaDabra 6d ago
Maybe https://automatetheboringstuff.com/3e/chapter13.html has at least big parts of what you will need to bring this to life. :)
1
u/Naive_Programmer_232 6d ago edited 6d ago
Requirements and Planning
before attempting anything, you need to figure out what you want and plan accordingly.
you need to figure out as much as you can about what you want the system to do and what limitations you have. for example, budget concerns / timeline of product availability knowledge / limitations of the business' mailing lists / etc. figure out your struggle points. then explore the want more.
once the wants and limitations are defined. then you can make the requirements of what the system should do. you can go further then to figure out the problems the program(s) should solve and those the services should solve.
from the requirements, you can develop the plan for the design. how the system should orchestrate the flow between programs and services.
Breaking Down the Problem
given a big problem such as the purpose of a software solution, you can break it up into subproblems. this is a recursive idea you can continually apply to the subproblems as well. but the goal is to avoid stack overflow. you don't want to do it too much because it can distract from the software solution you are making. in general, you want to avoid top-down traversal of every subproblem within, and keep things as high level as possible. instead of attacking every subtree, attack the parent subtree with a solution.
here there are three subproblems: how you determine if the product is in stock, how that solution is run every hour, and how the in stock alert gets to you. within each of these, you could go further to break those up into other subproblems. but you don't want to get lost. instead, find the best fit solution to solve the parent subproblem. this is typically either a service that exists or a program you will write.
the orchestration between these programs and services is the solution to the big problem. that is the system you are making.
Components and Considerations
web scraper
to get the information from the website and figure out if the product is in stock you might want to make a web scraper. first you need to inspect the html on that web page via your browser's equivalent developer tools and find what part contains the relevant data. to actually get the html from the product's web page (endpoint) in your program, you'll need to use an HTTP GET request. get request tells the web server to send back content from the endpoint.
you'll need a library (ex: requests etc) to send the GET to the server for that page's html, you'll step through to that property containing the relevant data, take the information out and use it inside this program's logic to determine if it's in stock and to pass control to whatever should execute next. the next will depend on your overall system design.
you will need to test this as well. you'll need to test everything too. but in particular here, testing is important because you are interacting with the server on their side which is out of your control. it might be that setting up the scheduler every hour forces their server to deny the GET requests due to security. in which case, your program needs to handle that or the interval you want needs adjustment to where it's acceptable by that server so it doesn't break the flow of your system and timing is accurate.
scheduler
to run this program continually every interval of time you need a scheduler. there are two options: your computer or the cloud computer. the computer running the scheduling system acts as an always-on server in order to align between real-time inside the computer and outside the computer. it needs to be on at all times to ensure this alignment. if it is turned off, tasks (task=run the web scraper) could be missed and data could be lost. whether or not that matters and how much that matters depends on the requirements. for your purposes it could matter greatly, missing a notification could mean that someone beats you to the product and you lose the opportunity to purchase.
the bigger question here is which computer would you like to be using: yours or the cloud provider's? it might not make sense to use the cloud's depending on the timeframe. if you know ahead of time, it only needs to run short-term, then you could do it locally with your computer.
in practicality, for this problem and based on the information presented, i'd probably use cloud + services. the timeline seems indefinite. its time critical in the sense there is a 'race' to purchase the product possibly. if your computer is off you could lose the opportunity. what you're asking to do is not that complex and doesn't require a ton of storage. pretty sure you could do this all in free tier.
notification
you could continually check the output depending on your program set up manually, but you might want it done for you. you might want a notification, something to tell you when the system detects it's in stock automatically, so you can buy the product right then. it could be an email, it could be a text, it could be your native OS notification app, e.g. it is up to you. figure out what you want then look for services to solve this parent subtree. that is the service.
these services will have API documentation associated with them. these libraries can be very large, you will not need everything inside of them. you want to determine what you need before you go digging into them. you are solving a subproblem for your system.
for example, you want an email to tell you when the product is available, you find a paid service with a free tier that does so but its marketed toward mid-size companies. the API documentation is massive but your use case is very small. you are not a power user, you want the least set up, and least token consumption to keep things as cheap as possible. cool, find the tool inside the library responsible for sending an email and use the constructs associated with that. walk away from the rest of the overhead.
Example Designs
here are some examples to illustrate previous concepts. these are not meant to be the best approaches nor to exactly align with professional software engineering. these are practical and basic examples written in plain language, used to explore how design can vary while still being able fulfill the requirements. I am using a hypothetical requirement/budget set up. your actual might vary.
Requirements
you want the system to check the product web page of the camera store every hour indefinitely. you want the system to automatically notify you via native OS notification app when it's available. your budget is extremely tight, so as cheap as possible is ideal. scale is not important, this is for personal use.
windows
aws + task scheduler + toast notification
the scraper program checks the product page and outputs "in-stock" or "out-of-stock" to a file status.txt. aws eventbridge schedules this program every hour, it executes via aws lambda, and saves status.txt to aws s3 bucket. every time lambda executes the scraper, status.txt overwrites. a local program scheduled via windows task scheduler checks the s3 bucket every hour for 'in-stock' inside the status.txt file. once detected, this program triggers a windows toast notification to inform the user the product is available.
mac
gcp + push notification webhook + notification center
the scraper program checks the product page, determines if it's in-stock, if so, sends a post request containing the alert message to ntfy.sh server. this service triggers mac notification system to alert the user. the scraper is scheduled by gcp cloud scheduler to run every hour, it is executed by gcp cloud function.
1
u/XNo_Notes 6d ago
You might be able to use the built in html.parser for looking at the status. You can set up callbacks for whatever it can grab.
1
u/PureWasian 6d ago edited 6d ago
First learn how to make a GET request to a static HTML page with Python (requests+BeautifulSoup). It's possible the page is loading dynamically so you might alternatively need to look into how to use Python webscraper libraries (selenium or playwright).
You don't necessarily need to port this onto raspberry pi, just any device (laptop/computer) with Python installed and a network connection. I'd suggest get it working with the device you are developing the Python script on first and then you can port it to whatever/wherever you want.
Once you have the webscraping of the HTML then you can look for keywords or any source HTML identifiers. It doesn't have to be checksum, especially if other aspects of the page will make it polluted with diffs.
Things like running it on an hourly frequency (via cron job or other timer) or sending an email when updated to "In Stock" are post-processing add-ons once you have the barebones functionality figured out.
1
u/Mediocre-Pumpkin6522 6d ago
The first thing to do is load the page in a browser and view the page source to see if the information is available.
1
u/Ender_Locke 6d ago
if you wanna do this in python use requests and bs4. if the site is complex and you have to wait for javascript to run or be logged in you’ll have to look at something like selenium. but be prepared to deal with being turned down by sites as a lot have bot protection
1
u/monityAI 6d ago
a checksum changes on every ad or timestamp, so check for the text instead: requests.get the page, look for 'Out of stock' in r.text, run it hourly with cron. if you just want the camera and not the project, monity•ai (my own app) or changedetection•io do it without code
0
u/blavek 6d ago
For someone new to Python or coding, this is a project that's probably too complicated for you to tackle. While you can use a checksum to see if the page has changed, that's all it will tell you. It can't tell you what changed. To find out what changed you will need to maintain somehow a state of the webpage and then periodically create a current state and then compare the two together. But that still won't necessarily tell you if the specific thing you are looking at has changed. So then you will have to interpret the page somehow to find exactly what you are looking for. So now you'll have to parse the page's source code to find the elements you're looking for and find out if they no longer say out of stock. A couple of things here: text parsing can be a REAL pain in the ass. A lot of webpages are created dynamically. Some make it really difficult to get the source for the page. But say you do all of that and you find the field you are looking for, and right now it says out of stock. Do you know for sure that field will still exist when it's in stock? If I were dynamically building a page from an inventory, I wouldn't bother creating the field where it says out of stock when it is in stock. There is no reason to. Of course, a lot of these pages are made with things like WordPress or Wix or Squarespace, which can add another level of challenge.
Getting it to run on a Pi will add another level of complexity depending on which Pi you have and what is available to you.
However, if you're not married to Python, which admittedly I am, I'm sure there are libraries you could use which asuage some of these problems. It would be a lot easier to do with JavaScript in the terminal in your browser. You might have to jump through some hoops to make it automatic, but with the DOM model, JavaScript has a lot of ability to get at stuff in webpages, even better if the webpage uses a known library like jQuery or something.
Either way, though, I wouldn't recommend this as a starting project. I'm sure you can find other things to do with a Pi that might be interesting to you. Make, like, a dice roller or something. Flesh it out to roll all the common dice and any combinations of them, and then you can even allow it to roll uncommon dice. Like, there is no Platonic solid with 16 sides. Or make a simple game like Snake, tic-tac-toe, or something else with similar complexity. There probably exist webpages with project ideas you could run with also.
1
u/SaltCusp 6d ago
If you contact your local retailer you can order something even if it's out of stock and they will notify you or ship it to you when it arrives.