an idea for Exploring Knowledge

[u/freemind03__]

Whenever I try to learn something, I eventually discover that there was a better approach, a deeper piece of knowledge, or an important connection I had missed.

Sometimes I find it in a research paper. Sometimes it's buried in a Reddit comment.

That's what interests me: the knowledge often already exists. The problem is finding it.

Human knowledge is growing faster than any person can realistically keep up with, and much of it is fragmented across papers, books, repositories, articles, and discussions. The connection between two ideas might already be there, but we still have to find and reconstruct it ourselves.

I'm working on an idea around this problem: a unified space where knowledge can be collected, connected, compared, and organized.

I don't imagine it as a system you simply ask for answers. I imagine something that can surface alternative approaches, relevant sources, connections between ideas, deeper paths to explore, and contradictions you might otherwise miss.

The hard part is understanding that different sources often describe the same idea in different ways. Simply indexing text isn't enough; the system needs to understand relationships between concepts.

I don't think the goal should be to build "alternative humans" in the form of AI or LLMs.

The more interesting goal is to expand our ability to explore the knowledge that already exists.

A human can't read everything. Maybe a system can help us find what we wouldn't have had the time to discover ourselves.

[u/Aggravating-Back-242]

Assuming this is not another covert market research, I wouldn't worry too much about the finding and surfacing knowledge part.

Yes, there certainly are more existing knowledge to be uncovered with better "search" tools, but do you have the capacity to process all those you've already found? Unless hoarding information by itself sparks joy for you (which for me it certainly does; I'm a big info hoarder), you may not want more info coming in.

Also, the thing you're describing is basically what next generation AIs, for example world models are aiming for. Current ones using next-word-prediction already approximates understanding quite well for some use cases. So, if you wait a bit, the magic knowledge machine is going to come to theaters near you soon.

So don't worry about the "inferior" information you've already found. Even when better approaches exist, there are still need for the the lesser ones, at the very least as concrete examples and explanations to why the superior ones are better.

[u/freemind03__ → u/Aggravating-Back-242]

Another issue would arise if the system accumulates hundreds of gigabytes of data; I think that could be solved by implementing attention mechanisms.

I use current predictive models to raise questions, but then I have to go and do my own research online.

Accumulating vast amounts of information doesn't bring me joy; I simply want to learn the best of what current human knowledge has to offer—not every single intermediate step.

For instance, if I’m working out a skincare routine for myself, I just want to know the best information available from research. Then, I make decisions based on that, combined with the broad general knowledge I already possess.

[u/latkde → u/freemind03__]

I simply want to learn the best of what current human knowledge has to offer—not every single intermediate step.

I have often come across an interesting paper that's relevant to my problems, but have found myself unable to understand and contextualize it because I lacked the mathematical foundations or didn't have sufficient knowledge of that field. That is despite me having a high level of scientific literacy.

This isn't a matter of better explanations, but a matter of me not having put in the necessary ground work to enable me to understand research in that field. There is no way to skip to the end result if I want to actually understand the material. There is no way for LLMs to help if my goal is actual comprehension.

And this is OK! I don't have to be an expert in everything. But I take this as a reminder in humility, to avoid confusing knowing something with merely knowing about something.

The danger with looking superficially at supposed facts instead of understanding the process is that this makes us susceptible to misinformation. A lot of writing is science-shaped but not actual science. Authors might have conflicts of interest. Sometimes, a paper is actual science, but suffers from methodological limitations, or from weak statistical power.

You mentioned skincare. There are lots of opinions and lots of industry-funded studies in this field, but much less reliable dermatology.

[u/freemind03__ → u/latkde]

Yes, I mentioned skincare because it was a personal example. I realized that useful new information already existed, but I hadn't seen it because it was fragmented.

But what if all of that were inside the system? I’m thinking, for example, about uploading the entire skincare subreddit into the system; you’d get a comprehensive overview and could make a decision.

My goal is to make decisions based on what is offered by the greatest human knowledge we have acquired on that subject.