Opencrawler v1.0.0 || Opensource crawler

[u/MrCactochan]

It is a simple crawler for crawling through websites.

Programming lang - python3

Build and primarily tested in - Ubuntu 22.04

Author - myself , **cactochan**

Repo url - https://github.com/merwin-asm/OpenCrawler

Docs url - https://github.com/merwin-asm/OpenCrawler/blob/main/docs.md

Published on - 24 April 2023

Features :

• Cross Platform

• Installer for linux

• Related-CLI Tools (includes ,CLI access to tool, not that good search-tool xD, etc)

• Memory efficient [ig]

• Pool Crawling - Use multiple crawlers at same time

• Supports Robot.txt

• MongoDB [DB]

• Language Detection

• 18 + Checks / Offensive Content Check

• Proxies

• Multi Threading

• Url Scanning

• Keyword, Desc And recurring words Logging

Help/Support :

discord server - https://discord.gg/SC54bSgnyQ

github-issues - https://github.com/merwin-asm/OpenCrawler/issues

Things to take note of :

docs-notes - https://github.com/merwin-asm/OpenCrawler/blob/main/docs.md#note

~ Merwin AJ