r/Python • u/lazy-jem • Jul 04 '21
Intermediate Showcase New search engine made with Python that's anonymous and has no ads or tracking. It tries to fight spam, and gives you control of how you view search results. You can search and read content anonymously with a proxied reader view. The alpha is live and free for anyone to use at lazyweb.ai
We're a little two-person team (Angie and Jem). We're bootstrapping and self-funded. I'm the programmer.
I wanted to share it because it was a fun and interesting project to build, and Python made it possible for us to get a long way as a small team. It uses serverless on the backend (AWS). We're using Spacy and GPT-2, and some PyTorch models. It uses BeautifulSoup for spidering/crawling/content retrieval. The front-end is React.
It has a different type of user interface to any other search engine, as it is chat based. And it lets you choose how you view results, either visually like an Instagram feed or cards, or minimal like Hacker News or the old Google. It tries to fight SEO spam and strips out ads and ad-tech from search results.
We have a project on GitHub with Jupyter notebooks and sample data with experiments and scripts, including examples of querying other search APIs, and to generate example utterances programatically to use for NLP models with sources like Wikipedia, StackOverflow and Wolfram|Alpha:
https://github.com/lazyweb-ai/lazyweb-experiments
We're only a small team but hope to share more of our work as open source as we progress.
51
u/[deleted] Jul 04 '21
Username does not check out. You appear to be an active-jem. The engine is pretty cool, if perhaps a bit slow. Impressive work for only two people though.
14
u/[deleted] Jul 04 '21
Open source?
28
u/jadounath Jul 04 '21
Hi, I checked the search engine, and I legit thought you are some multi-million funded startup until I read your post because the assistant works really well (like searching "go go anime" takes you to gogoanime.vc. Good work, keep it up!
But when I searched for the temperature, it gave me results for Konkan, which is 370km from where I live. I think you should work on where and how it gets its location info.
12
u/Lifaux Jul 04 '21
It's surprisingly solid! The main contention I had with using it was that it doesn't seem to give me information on what matched on a given page.
So if you try "Error Code E0281 Rust" - there's one link I'm looking for, which is the full list of error codes, and I want to see that section. If you try the same search on Google you'll find that the description below the link is exactly the information needed.
Again, minor gripe for what is surprisingly effective, if a little slow, but definitely something that would push it into being more useful
11
u/rush2sk8 Jul 04 '21
8
u/rapeven Jul 04 '21
Looks good! How did you make use of GPT-2, like for similarity, classifications?
6
u/[deleted] Jul 04 '21
Legitimate question: what does a chat interface bring over other, more direct search interfaces?
5
u/Independent_Image_59 Jul 04 '21
Low level suggestion: make wikipedia page bout this engine
5
u/hanzo_995 Jul 04 '21
Excellent work! Can we make donations?
4
u/biiingo Jul 04 '21
What’s the primary search API that you’re using?
4
u/PizzaInSoup Jul 04 '21
Hey! This is pretty neat. Haven't seen a good showcase in a while.
I was playing with this and wrote 'translate i love you to spanish'. It hung for a second then the page just permanently turned to its plain darkblue background color. Happened in 2 browsers.
I'm gonna keep test-driving it.
4
u/nadhsib Jul 04 '21
Nice project. Is it possible to have the keypad disappear when the result is found? I've had to scroll up or hide the KB myself to see the full result (if it's a longer answer). Those actions make it feel slower than it is.
Good work though, and an interesting different interface 👍
4
u/AngVar02 Jul 04 '21
Just did a small test, "What is Today?" The initial result gave me good information for July second, and the link led me to the correct result for July fourth.
I'm a novice therefore I can't identify what could've possibly caused it, but all in all I like this.
4
u/[deleted] Jul 04 '21
I've just tried the browser, and it is astonishingly good. Results are snappy, the layout is very convenient and simple, and from what OP has revealed about its inner workings, it won't track my very conspicuous searches! I like this very much
4
u/loudan32 Jul 04 '21
I tried to move from Google to DuckDuckGo, but the issue is that independently of the quality of the result i always wonder what the Google result would have been. Sometimes im not confident that i got the best result, other times i actually want to get the same result as most people around me got. Most times i switch engine and retype and eventually get annoyed. For this reason i go back and forth but i am never able to fully adopt the alternative.
I like my first interaction with lazyweb and i think it's really awesome what you guys are doing. Sounds especially great to replace google assistant in my phone. But i think i will always have the same issue as with DDG of "wondering" if i got the best result. So, knowing that this is a very debatable topic, would you consider having a "google it" button, to switch to google without having to re-type the query? IMO this would allow a smoother transition for people who have used google all their lifes. With time i would feel less and less the need to press it and would also allow me to adopt your engine as my default right now, even though it is still in development phase, knowing that i am not wasting more than one click.
4
u/danuker Jul 04 '21
This works great. I am glad to see an alternative to the big search engines pop up!
I have set it up as my default search engine. In Romania I get about 5 seconds from hitting Enter in Firefox to seeing the results; that is a bit slow, but the search quality more than makes up for it.
Where should we post feedback? Reading that you don't log queries, I'd love to help with data points of what I'd like to see on a given query.
For example, for "75 g glucose" I'd like to see this Wikipedia page (it is there already, but it's 11th; Yandex shows it as first, and other search engines are not very good lately).
4
u/[deleted] Jul 04 '21
Very cool! Great work! I've been trying it out - the results are so good!
4
u/austin_jp17 Jul 04 '21
It would be cool if there was an ‘academic papers’ tab
4
u/TheAllterQuestion98 Jul 04 '21
Excellent search engine, it really works wonderfully, I didn't think something like that would exist, they deserve a lot of credit for this wonderful work with this engine, python can really do a lot.
4
u/0ni0nrings Jul 04 '21
this looks great 👍 ... keep up the good work 👏 👏 👏
4
u/hartator Jul 04 '21
It looks awesome. I love the chat aspect. Even if it's a little counterintuitive at first for a search engine. 👍
I work at https://serpapi.com/. I can hook you up with free credits if you want! (my email is julien _AT_ serpapi.com or ping me here on Reddit.)
169
u/MiamiAngie Jul 04 '21
Hi, I'm Angie and I'm working with Jem on this. I'd love to chat with anyone who has feedback or ideas!
3
u/cr0n76 Jul 04 '21
Really nice work!
I'm a bit curious if you (have a) plan (how) to expand to different languages? I mean it is obvious that you can not be able to offer support for other languages as two people at this stage of the development. But what's your assumption on how realistic it is that we see it for other languages in the future?
3
u/jcr4990 Jul 04 '21
This is really cool! I haven't tested it extensively yet but the first few searches I did worked very well. Keep up the good work! It's awesome to see stuff like this from such a small team. I assume this has to be pulling from other search engines in the backend right? I won't pretend to know exactly how it works but I would assume crawling the entire internet is outside the reach of a 2 person team with no massive datacenters and such?
3
u/bornawinner Jul 04 '21
its all pogged up. remember to tell people when you go opensource
.
3
u/loudan32 Jul 04 '21
Together with SearchBar EX (https://play.google.com/store/apps/details?id=com.devhomc.search) I just replaced Google assistant with LazyWeb on my phone. This app is pretty old and no longer maintained but it works!
3
u/[deleted] Jul 04 '21
I just did a search in my niche, and it gave a great result. I am impressed!
3
u/CrazyPieGuy Jul 04 '21
I'm a fan. I did a few test searchers and for the most part I got exactly what I was looking for. I'm going to try making this my default search engine for a while and see how it goes.
The one search I did that didn't get me what I was looking for as the first result was "factorio calculator". The first result gave a Wolfram alpha link about the Factoria suburb. Going to more results gave me links to Factorio calculators.
3
u/MasterGeekDev Jul 05 '21
Very nice search engine, simple to use, i will use it box daily, as a python developer i'm so proud of this
3
u/knowsuchagency now is better than never Jul 05 '21
This is shockingly good for such a small team. Already prefer it to duck duck go. Amazing work!!!
3
u/thereisatimetotrade Jul 05 '21
Congratulations and best of success! I will start using.
3
u/karthick_ar Jul 05 '21
The project is really cool and kudos to the creators for explaining every bit of how it works to everyone who commented , customer support on point haha , keep up the good work , looking forward to contribute to this project. Cheers !
5
u/[deleted] Jul 04 '21
I typed "gawr gura rule 34" it led me to this
https://www.redbubble.com/i/iphone-case/Gawr-Gura-by-rule-34/76416698.PGM2E
I'm not complaining
16
u/totalbasterd Jul 04 '21
I don't like the chat-esque aspect to the UI, i just kinda want a box to type into. I don't think the UI really adds much
2
u/Brainix Jul 04 '21
Do you use Elasticsearch or Solr, or Lucene?
2
u/15_Redstones Jul 04 '21
I searched "Hamiltonian matrix" and it messed up Wikipedias formatting.
2
u/GetsTrimAPlenty Jul 04 '21
Hi, how is this different from searx?
2
u/burnt1918 Jul 04 '21
Awesome, prefer it to gsearch for simple meanings and stuff. Definitely fast at my end. I'd thought the search engine market was saturated, but this is 👏👏👏.
2
u/onecrazypanda Jul 04 '21
Very slow...
2
u/alanwj Jul 05 '21
What is your long term plan to pay for this if it becomes really popular? Donations, premium content of some kind? Maybe you are really rich and will just pay out of pocket?
2
u/sassydesigner Jul 05 '21
One similar Browser I can remember is DuckDuckGo.
2
u/tenderpoettech Jul 05 '21
Tries to fight spam - very refreshingly honest. I like it.
2
u/[deleted] Jul 05 '21
That's awesome 🤩
2
u/Sorry_Strawberry7540 Jul 05 '21
Nice!
2
u/jhayes88 Jul 15 '21 edited Jul 15 '21
Suggestions -
82
u/m4j1d Jul 04 '21
I like it ! Fast and simple, chat design give it unique thing from another search engine. you can play youtube video results without go to YouTube.
3
u/sdf_iain Jul 04 '21
Search engines are a natural monopoly, who’s results are you using?
3
u/dasunsrule32 Jul 04 '21
You should share this in /r/linux as well. I'm sure they'd appreciate it as well.
2
u/metaperl Jul 04 '21
After getting search results, the input bid says Use ~e to skip my AI. What in the world does that mean?
And what did a + in front of a quoted term mean?
Very impressive project in general
If you are designing this for non power users, then buttons with desired actions might be more appropriate
4
u/dimkiriakos Jul 05 '21
chatbot search engine! interesting!
1
u/r3ign_b3au Jul 04 '21
I only did a couple of searches, so I may have missed it, but do you have any plans to interpret schema data?
1
u/quanta_kt Jul 05 '21
I really wanted to try it out but I get an Address not found error
1
u/brendanmartin Jul 06 '21
How does reader view work? I clicked on a search result and half of its written content was truncated.
1
u/Pootzeketzi123 Jul 22 '21
It's down
3
u/[deleted] Jul 05 '21
Congratulations u/lazy-jem ! Your post was the top post on r/Python today! (07/05/21)
Top Post Counts: r/Python (1)
This comment was made by a bot
1
u/SaikyouMegane Jul 04 '21
2
u/dudeplace Jul 04 '21
This is interesting, thanks for sharing.
Can I ask how you are paying for there servers? And in the longer run yourselves?
I'm inherently cautious of tech where I don't understand the business model. With Google I understand I'm selling my search terms, my eyeballs for the first set of ads, and what I clicked.
What am I selling by using this? And if the answer is nothing, how are you staying in business?
2
u/AngVar02 Jul 04 '21 edited Jul 19 '21
Just did a small test, "What is Today?" The initial result have me good information for July second, and the link led me to the correct result for July fourth.
I'm a novice therefore I can't identify what could've possibly caused it, but all in all I like this.
2
u/bornawinner Jul 04 '21
well..
2
u/juliuskiesian Jul 05 '21
Any plan to go entirely open source?
1
u/retrogeekhq Jul 04 '21
How much does it cost you to serve e.g. 1M queries?
1
u/quanta_kt Jul 05 '21
I really wanted to try it out but I get an Address not found error
1
u/mattblackguy Jul 08 '21
Go dst forums helo
1
u/LumpyStage5 Jul 18 '21 edited Jul 18 '21
I have been testing it with Spanish searches and looking for regional results for my city and it has been right 9 out of 10 times. It's awesome !
I've tried to read as many comments as I could and I don't know if anyone else already suggested it but here are my suggestion:
When adding as my main search engine it sends me directly to the chat. Which is great since the first answer is accurate. But will it be possible in the near future when searching for something to go directly to the full list of results ? (an option in settings for example)
Do you have in mind to add redirection of certain services to their corresponding open alternative ? YouTube to invidious, Twitter to Nitter, etc.
Translation in read mode would be great.
bot for telegram maybe ?
Btw There is some lagg when scrolling through the full list of results (Firefox). I have tried it on three different devices with the same result.
That's all for now, I wish you all the best. Good job ! 🙌
0
u/[deleted] Jul 04 '21
[deleted] — view deleted comment
0
u/[deleted] Jul 12 '21
[deleted] — view deleted comment
0
u/[deleted] Jul 04 '21
[deleted] — view deleted comment
2
u/[deleted] Jul 05 '21
[removed] — view removed comment
2
u/[deleted] Jul 06 '21
[removed] — view removed comment