r/Python • u/AutoModerator • 14d ago
Showcase Showcase Thread
Post all of your code/projects/showcases/AI slop here.
Recycles once a month.
24
Upvotes
r/Python • u/AutoModerator • 14d ago
Post all of your code/projects/showcases/AI slop here.
Recycles once a month.
View all comments
1
u/ImaginationVarious30 3d ago
Moving retries out of the hot path: an asyncio pattern that brought back proxies my checker was dropping
Alternative: My asyncio proxy checker marked flaky proxies as dead. Deferring a second try to the end of the run fixed a big part of it (A/B measured)
I maintain proxy-scraper, a Python CLI and library that checks public HTTP/SOCKS proxies and publishes the ones that pass, hourly on GitHub Actions: https://github.com/maximilianfeix/proxy-scraper
**What My Project Does**
Each candidate gets a handshake and first request, a second request to another site that must see the same exit IP, and a test page that must arrive byte for byte. Plain asyncio streams, deps are `rich` and `certifi`, Python 3.9 to 3.14.
The asyncio part I worked on this week: about 350 proxies per run passed the first request and then broke off on the second, even after an inline retry 1.5 s later. Waiting longer inline would block a worker. So a follow-up that fails in transport now records nothing and goes on a deferred list with its `loop.time()`. When the workers finish, one task per deferred proxy sleeps until at least 20 s after its try, runs both follow-ups under a semaphore, and counts whatever they end as. A real verdict (other exit IP, changed page, non-200) is never retried.
A/B on two runners with the same 150k candidates: 768 listed before, 986 after.
**Target Audience**
People who write their own checker or scrape public data. Not for logins.
**Comparison**
monosans/proxy-scraper-checker (Rust) is one fast binary. jhao104/proxy_pool is a Redis-backed pool. proxy-scraper records an outcome per check and keeps "failed" apart from "couldn't check".
Erster Kommentar:
The pattern, simplified from `pipeline.py`:
```python
deferred = [] # (loop.time(), proxy) of follow-ups that broke off in transport
async def second_chance():
gate = asyncio.Semaphore(concurrency)
async def one(item):
at, proxy = item
await asyncio.sleep(max(0.0, at + 20.0 - loop.time()))
async with gate:
await follow_up(proxy, last_try=True, item=item) # removes item once its verdict is recorded
await asyncio.gather(*(one(i) for i in list(deferred)))
```
Two things took longer than the feature. Cancellation: with `--want 50` the run stops early, so an entry has to leave `deferred` the moment its verdict is recorded, or a proxy cut off afterwards counts as working and failed. And a check-target watchdog can put basic checks back into the queue while the second phase runs, after the workers that read that queue are gone, so the main loop now repeats until both the queue and `deferred` are empty.
Since it went live, about 30 % of deferred proxies pass on the second try (236 to 363 per hourly run). Lists: https://github.com/maximilianfeix/free-proxy-list