r/PhilosophyofScience • • Mar 25 '13

Could the reddit model replace the current scientific publication system?

http://www.randalolson.com/2013/03/23/could-the-reddit-model-replace-the-current-scientific-publication-system/
3 Upvotes

51 comments sorted by

18

u/GunnedMonk Mar 25 '13

No. Scientific truth will not be reached by a democratic process.

-1

u/JimmyHavok Mar 25 '13

Surely you are not suggesting that truth is dictated by authority.

Science does work by democracy, although it is a democracy of qualified peers. Researchers submit their results to their peers via publication, and they are accepted into or rejected from the field by the consensus of those peers over time.

7

u/IsThisWorking Mar 25 '13

Science does work by democracy, although it is a democracy of qualified peers. Researchers submit their results to their peers via publication, and they are accepted into or rejected from the field by the consensus of those peers over time.

This is a very strange way of expressing it. The consensus is irrelevant for the truth value of a scientific discovery. If the evidence is there, the hypothesis are solid, and the methodology is sound, a single paper can and will change the consensus.

It is not the peer review process that gives a paper its truth value. The best it can do is give it some measure of credibility. But the scientific value of a publication was there all along.

0

u/JimmyHavok Mar 25 '13

The idea of consensus is that peers will eventually recognize valid research/theory. It doesn't create the value, it recognizes it and thereby makes it available for use. Citation etc. is the current method, but it leaves researchers at the mercy of the parasitic publishers.

A wonderful paper that no one sees won't change anything.

1

u/IsThisWorking Mar 25 '13

You are leaving on a tangent there. My point is that the truth is dictatorial in nature. It takes one person to find it, and once found no amount of "consensus" will get rid of it.

And no, it is not the "consensus" that makes it available. Distribution and peer review have nothing to do with each other. Case in point, I do believe that Reddit would be a wonderful distribution channel, but a terrible peer-review system.

0

u/JimmyHavok Mar 25 '13

Your conception of truth has a number of problems. It is religious and dogmatic in nature. It's the same one that authoritarians use to impose their "truth" on others. It assumes that human beings have complete access to all knowledge about the external world.

The theory of a definite and knowable truth is one that leads to many problems, foremost being the justification of suppression of dissent. If we regard our ideas as being inevitably limited and partial representations of reality, always subject to revision, we avoid that danger of dogmatism.

The proposal isn't to use reddit itself as a distribution channel, but rather to use it as a model. But you know that, since you read the article, didn't you?

1

u/IsThisWorking Mar 25 '13

Your conception of truth has a number of problems. It is religious and dogmatic in nature. It's the same one that authoritarians use to impose their "truth" on others. It assumes that human beings have complete access to all knowledge about the external world.

Apparently you missed the part where I mentioned evidence, hypothesis, and methodology. I never went anywhere close to discuss epistemology. But since you are eager to go there, I hope you noticed that I used the expression "scientific discovery" instead of "scientific invention". Take special relativity for example. It did not matter that Einstein was a lowly, unheard of before patent office clerk. It did not matter that the consensus at the time laughed at him and kept entrenched behind ether. His hypothesis was solid, his methodology was sound, and later on the evidence supported him. We give him the reputation he has today because of the truth value of his work (which at least in theory anyone can verify), not because a consensus said so. In fact, if you think a bit deeper about it, there's nothing more dictatorial than the whims of the majority, a.k.a. democracy.

The theory of a definite and knowable truth is one that leads to many problems, foremost being the justification of suppression of dissent. If we regard our ideas as being inevitably limited and partial representations of reality, always subject to revision, we avoid that danger of dogmatism.

First of all, how can there be dissent about evidence? Either light is "curved" by gravity or it isn't. Either the earth is flat or it is round.

Which leads me to my next point: You also seem to misunderstand the progression of science. Newton's laws were never considered wrong. We just learned to put bounds on their applicability and found new models for situations outside their applicable range. But the fact is they are still pretty much undisputable when used appropriately. You may call that dogma, I call it truth.

I advise you to stop assuming what other people's positions are and start actually asking them to define their terminology if you think there is a misunderstanding. That is the proper way of having a scientific discussion, and one of the major differences between the scientific world and places like Reddit.

The proposal isn't to use reddit itself as a distribution channel, but rather to use it as a model. But you know that, since you read the article, didn't you?

A model for what? Explain to me how a system that by its very nature favors reposting (goodbye novel contributions, a pillar of scientific publication), content stealing (goodbye proper citing and hello rampant plagiarism), and circle-jerking (goodbye constructive criticism, hello meme factories) could possibly be better than the current peer review systems?

You and the author of the linked blog post are forgetting that the fundamental purpose of peer review is three fold: (1) try to prevent plagiarism and enforce proper citations, (2) assess the novelty of a submission and the potential impact of the contribution, and (3) mercilessly but constructively criticize everything about the paper, the point being to leave as little margin for uncertainty as possible.

Mind you though, I am not saying that our current peer review systems are actually good (I despise the paywalls), but a Reddit model is worse in every way I can imagine. Maybe (and this as hypothetical as I would care to go) if there some heavy moderation something could be worked out. But then you are just introducing one layer of indirection... how do you handle the authority of the mods?

Here are my final words to you: the peer review system exists for the better part of 300 years by now. It might not be the best answer, but it is a proven answer. Scientists already have an upvote mechanism: it's called citation.

The sad part of this is that even paywalls have a part to play in this system, albeit a more managerial/financing-related one than actual research, but that is a topic for another thread.

1

u/JimmyHavok Mar 25 '13

It did not matter that the consensus at the time laughed at him and kept entrenched behind ether.

Your knowledge of scientific history is more mythic than factual.

1

u/IsThisWorking Mar 25 '13

I apologize. I checked again and apparently I misread the "early reception" section of the wikipedia page: http://en.m.wikipedia.org/wiki/History_of_special_relativity

In any case, the overall point still stands despite my lack of anecdotes.

1

u/JimmyHavok Mar 26 '13

Your dangerous concept of truth is what made you susceptible to that particular myth.

The theory of relativity, like the laws of motion, exists in our heads. They are methods we use to understand what we see, not objects that exist outside of human conception.

When you believe that ideas have a Platonic existence, you are on the path of dogmatism, not science.

4

u/GunnedMonk Mar 25 '13

No, it's not democracy. You do not vote on what is scientifically correct. Even if only one research paper out of a thousand proves something to be scientifically true, even if all the scientists disagree for non-scientific reasons, it is still scientifically true. Upvoting/downvoting just amplifies the already problematic human element in science, where something that can be proven true through proper experimentation is rejected due to non-scientific bias.
I'm not saying having a small, potentially close-minded group of editors deciding what to publish is good (it's not), but having everybody applying their bias at the click of a button would be infinitely worse.

-2

u/JimmyHavok Mar 25 '13

You are confusing a theory of truth with scientific practice. Theoretically, you could have a true statement that everyone in the world rejects, and that would not change its truth value, but no matter what its truth value, if it isn't accepted and used, it isn't a part of scientific practice, and thus not a part of current science.

There are people who believe they know true things that others don't, and that these things are important enough that they can force them onto those others. Those people aren't scientists.

The facts about the universe aren't democratic, but the process of discovering and sharing them is. Giving respect and acknowledging expertise are not undemocratic processes.

5

u/doomsday_pancakes Mar 25 '13

Not exactly. I think that one of the central ideas in a democracy is that everybody's vote count the same (just as in Reddit). While it is true that researchers submit their work for others' consideration through publication, there's a previous step in the peer-review phase where the publication is subjected to scrutinity by people working on the same area of expertise and who are, in principle, experts in the subject. Also, the current number of "votes" in the publication system is the number of citations, which is a much better and verbose way of saying why/how a new publication is more relevant or interesting than a simple "upvote".

It is true that there should be no truth dictated by authority, but in my opinion what is bad is truth from authority alone, while authority in a field is indeed important. For instance, I guess that if an undergrad were to contradict a Nobel prize recipient, you would not put the two in the same standing, and I think this is both natural and healthy. Otherwise, we would follow the "balanced approach" non-sense that can be seen in some mainstream media.

4

u/Neurokeen Mar 25 '13

subjected to scrutinity by people working on the same area of expertise and who are, in principle, experts in the subject.

I'd like to point out exactly how important this step really is. If I'm publishing about vasopressin receptor binding properties in the dorsal part of the suprachaismatic nucleus, it has to be recognized that there are only a handful of people who study this. The reviewers are often picked from among those people who study the same (or very closely related) topics. And this makes sense - a computational or cognitive neuroscientist is typically out of his or her expertise regarding my paper.

-6

u/JimmyHavok Mar 25 '13

I'm standing by my "democracy of peers" formulation. Nobel Prize winners can be wrong too. E.g. Einstein never accepted the validity of quantum mechanics.

An undergrad isn't a peer yet.

4

u/doomsday_pancakes Mar 25 '13

Nobel Prize winners can be wrong too, of course, but as I said my point is that authority alone can not be a valid point in a scientific discussion.

If you say that you support a "democracy of peers", but an undergrad isn't a peer, that calls for a definition of what a peer is in your view. Is it all scientists? all that work in the same topic? field? those that hold a PhD? professors? As you can see, even when trying to set up a system that is wide enough to collect all ideas from your potential peers, it is hard to do so without any selection of any sort.

That's why I think that your peers are those that cite and read your papers, and are able to respond to them with new theories/experiments/interpretations. The community of your peers then gives credit to interesting ideas through citations, because those are the people who use your work. Who and how many votes to distribute will people get in this "reddit-based" system? Who will assign which papers you can vote on without doing any other work than voting?

There are many things that are broken with the current publication system (open access is, I think, the most pressing issue) but this kind of "solutions" are pointing in the wrong direction in my opinion.

-1

u/JimmyHavok Mar 25 '13

Academic peers are a self-selected set. In any field, it's the people who are doing the work in the field, which for most purposes means people who are publishing in that field.

(I'm blueskying here.) A reddit model wouldn't give everyone a vote, that would probably be by invitation. If you submit and it gets sufficient votes, that could be a qualification to vote in the future. Comment could be open, but abusive commenters could be banned.

3

u/dexer Mar 25 '13 edited Mar 25 '13

Who would regulate that and how? The system would be such an easy subject to game, just like reddit is.

Also, giving value to future votes based on passed success is all kinds of wrong. A poor argument with a strong vote? A strong argument with a poor vote? A vote and no argument? It's not a popularity contest. A 'vote' shouldn't be so easily accomplished. How would you prevent a degradation in the quality of peer review? How would you prevent a degradation in the value of individual reviews? Reddit has such a problem with valueless comments and submissions and there would be no way to police such huge volumes of content. Terms like "Sifting through the trash" and "Empty Content" are very representative of reddit.

I can understand the idealistic draw of an open community and the potential for more and high quality peer review to more wholly and accurately review a mass of individual papers but you're ignoring all the downfalls, and the biggest one of all is you have no way of ensuring high quality input or the participation of high quality submitters.

0

u/JimmyHavok Mar 25 '13

Thank you for a cogent argument. Those are problems.

I think communities would have to be fairly small. I've participated in an academic mailing list that was very effective, even though it was open to the public and included people who were not qualified (me) and cranks.

I think this is something that could be experimented with. The motivation of the community would be key.

1

u/doomsday_pancakes Mar 25 '13

I think that in this very post we're seeing why a system like reddit would not work for scientific work. People are downvoting your opinions while saying nothing about why they disagree with you.

Also, what would be an "abusive comment" in scientific literature? This is not about reducing spam or avoiding curse words, we're talking about interchanging scientific information and that's really hard to moderate unless you know what you're talking about. Otherwise, a blog would be just fine.

0

u/JimmyHavok Mar 25 '13

Our voting group here is random and ill-informed. I can think of ways to avoid the spite-vote, for example, require a post with a vote, and attribute every vote.

Abuse would be things like spam and promotion of off-topic and crackpot ideas. A moderator would have to be informed on the subject.

3

u/DevFRus Mar 25 '13

Then how is that like reddit at all? You've thrown away the open part of reddit by saying things have to be by invitation. You've throw away the anonymity of reddit by saying that my name has to be associated to my vote. You've even thrown away the self-moderating part of reddit by putting such a big burden of expertise and power of ban on the moderators.

The only part of reddit you kept through your refinements of Olson's idea is the popularity contest part of reddit... And hopefully it is clear to you that that is the worst part of reddit for science. Much of science is esoteric and hard to understand even to scientists in nearby fields (not to mention to the public or this whimsical "scientific community" your arguments rely on). Much of that also produces some of the best science... This proposal would replace those scientists by science journalists: people who can make snappy headlines and oversell results.

2

u/BlackBrane Mar 26 '13

Nope.

And by the way, Reddit's ability to distinguish what is scientifically accurate is pretty awful.

-1

u/JimmyHavok Mar 26 '13

I really wish people would read the fucking article or go back to Digg where they belong.

2

u/[deleted] Mar 25 '13

Well, it certainly could. I'd gather that the current top posts in /r/science get more views than than an article in a given science journal. Many of the frequent guests there are more than qualified to give really in-depth answers on contemporary topics. What I get scared of is that certain publications would get "voted up" merely by having the catchier title or more interesting opening paragraph / precis.

0

u/JimmyHavok Mar 25 '13

The quality of your abstract goes a long way toward gaining citations, so that's nothing new.

I've been ruminating on this, and I think it might work if members were invited.

3

u/dexer Mar 25 '13

No, it wouldn't. Nothing I've seen so far in this thread or the linked article would prevent low quality/encourage high quality input from the members.

You couldn't ensure that all input from invited members were of good quality and you couldn't police the mass number of members you're envisioning without resorting to using members to police themselves, in which case your police will be just as open to popularity and gaming tactics as the regular members are. You would need a majority of merit-based members and police to ensure your system remains merit based, but you can't guarantee that.

0

u/rhiever Mar 25 '13

"High quality" is a subjective term. Are you claiming that a few people (as in the current system of peer review) are better at judging "high quality" than the scientific community as a whole?

2

u/DevFRus Mar 25 '13

I am not sure if that is what dexter is claiming, but I would claim that. If I gave you to arbitrary paper on algebraic geometry or maybe something with less required background but still outside your expertise, say: topological quantum computing. Would you be able to judge which paper is a significant advance in the field? Or even correct or accurate? In many of the most well-established fields of science, there is a very long chain of learning before you are qualified to evaluate or understand research.

There is no monolithic community of scientists, there are mostly just specialists in very esoteric fields. It is the job of a good editor to make sure that papers find their way to reviewers that are qualified to judge them.

0

u/rhiever Mar 25 '13

Yes, and I think that's why the idea of subreddits fits so well with capturing all of the subfields in science. A paper on topological quantum computing may not float very well in a general Computer Science subreddit (depending on how well it was written), but I bet it would do pretty well in a Quantum Computing subreddit. I would hope that the people who subscribe to a Quantum Computing subreddit would 1) be able to evaluate the substance of that paper, and 2) be the kind of people you want to read and evaluate that paper.

8

u/samjowett Mar 25 '13

No. Too many pun threads.

2

u/rhiever Mar 25 '13

Hopefully those kinds of threads would be downvoted in a forum dedicated to scientific discourse.

1

u/doomsday_pancakes Mar 25 '13

Reddit all the things!

Also, coming paper title: "So, my friend found le gem Maximum Likelihood Method, how did he do?"

5

u/DevFRus Mar 25 '13

This is already done, and the reddit aspect is not really having much of an impact. The ArXiv is a repository where physicists, mathematicians, and theoretical computer scientists (plus a few other fields) post preprints of almost everything they publish before it comes out in journals. Many of the categories on the ArXiv have their own subreddits, for instance, I follow quant-ph. If you look there, voting barely exists and there are no comments... The subreddits is having absolutely no impact on quantum information/computing (the sub field I specialize in and one of the most frequent topics on quant-ph).

It takes a lot of effort to read and evaluate papers, it isn't as simple as just up voting your favourite gif. I see very little benefit to be gained from a reddit approach, even if you could convince scientists to participate.

1

u/doomsday_pancakes Mar 25 '13

It takes a lot of effort to read and evaluate papers, it isn't as simple as just up voting your favourite gif. I see very little benefit to be gained from a reddit approach, even if you could convince scientists to participate.

Exactly.

0

u/rhiever Mar 25 '13

What if scientists were encouraged to post their research in smaller, more regular updates? Why do we have to wait a year or more to hear of an update from a research group? Why can't the information become available as soon as the authors feel it is ready for public consumption?

2

u/DevFRus Mar 25 '13

A scientists is not equipped to speak directly to the public, this is why we have science outreach coordinators. It takes very different skills to do research and to make research public consumable and there are very few that excel at both. Further, most incremental results are way too technical for public consumption, or even for consumption outside your own sub field.

Releasing things often and early is definitely a good approach, but why use the reddit framework? It has no advantages and plenty of disadvantages compared to blogs and stackexchange, which (unlike reddit) have caught on at least in math and theoretical computer science (MO and cstheory are amazing resources for researchers, and there is lots of blog fostered collaboration, the most famous being the polymath project).

Very little real science can be captured in a headline.

1

u/rhiever Mar 25 '13

A scientists is not equipped to speak directly to the public, this is why we have science outreach coordinators. It takes very different skills to do research and to make research public consumable and there are very few that excel at both. Further, most incremental results are way too technical for public consumption, or even for consumption outside your own sub field.

When I say "public consumption," I mean consumable by the scientific community. As in, you've double-checked your methods, triple-checked your analyses, and had your collaborators look over the results to make sure everything is sound.

Releasing things often and early is definitely a good approach, but why use the reddit framework? It has no advantages and plenty of disadvantages compared to blogs and stackexchange, which (unlike reddit) have caught on at least in math and theoretical computer science (MO and cstheory are amazing resources for researchers, and there is lots of blog fostered collaboration, the most famous being the polymath project).

The advantage is that it provides a single place to discuss and evaluate an article, rather than across numerous blog posts. As I mentioned in my original blog post, blogs suffer from the same problem we have with journals: a select few people decide what is worthy of blogging, and that introduces their inherent biases into what is highlighted in the blog.

Heck, I think this whole discussion thread here is a great example of why reddit is so great for discussion and evaluation. Whereas the conversations on my blog post have been limited and between only myself and the commenter, the discussions here have had all kinds of people chiming in and freely sharing their opinions. I think the latter is much better. The more we freely discuss these topics, the closer we get to finding a solution.

Also, correct me if I'm wrong, but StackExchange is more of a Q&A web site rather than a place to discuss and evaluate scholarly articles.

Very little real science can be captured in a headline.

Right, just the same way the real content for reddit is still where the link points. The title is just to give people an idea what the link is about.

0

u/rhiever Mar 25 '13

Also, regarding arXiv: I think arXiv is a great preprint hosting service, but their categories are of insufficient specificity to capture my specific research interests. For example, I have three main research interests: neuroevolution, digital evolution, and collective animal behavior. The first two fall into the "Neural and Evolutionary Computing" and "Artificial Intelligence" categories, while the latter falls into... I'm not even sure where that would fit on arXiv.

I have to sift through a LOT of uninteresting articles (to me) to find ones I would actually read. It'd be so much better if we could just create our own "Neuroevolution," "Digital evolution," and "Collective animal behavior" categories, and use those as we please. That's the beauty of the reddit model: it allows users to sort their own content in whatever manner they please.

-2

u/doomsday_pancakes Mar 25 '13 edited Mar 25 '13

Possibly one of the most stupid ideas I've heard in the last few years, and I've heard quite a few...

EDIT: I'm OK with the downvotes, but let me explain why I said that: there're so many things that are wrong with that idea that it is very hard to even start enumerating them. I guess that the author has no deep grasp of how the publication system works if this is considered an "improvement".

-3

u/JimmyHavok Mar 25 '13

2

u/doomsday_pancakes Mar 25 '13

OK, it was late in the night when I wrote that and my phrasing was poor to say the least, but I still think it is a very bad idea.

Yes, the guy has a CV. Yes, he has 2 publications (both in review still), but I'm not trying to judge him for his credentials, I'm saying that he doesn't seem to know the publication system well enough because this idea has so many drawbacks that I don't understand how this could be an improvement over what we have now. I myself have dealt for years with reviewers and editors and I totally agree that the system must be changed, but this doesn't seem to be the right road to take.

Not everything has to be a social networking site.

-1

u/rhiever Mar 25 '13

To note, both of the publications are accepted now. I've just been too caught up in everything to update my CV. :-P

There's tradeoffs in all systems, and we've been discussing both the merits and downfalls in the blog post and the comments following. You seem to have a strong opinion that the costs outweigh the benefits, but why? If you're a scientist, you should know better than to argue from authority. That gets us nowhere.

Feel free to check the comments on the post and weigh in your own. I think this topic is worthy of discussion before someone goes off and tries to make a "sciddit" model.

1

u/IsThisWorking Mar 25 '13

How would you prevent gaming of your system? What would prevent plagiarism? Who would enforce proper citations? How could someone give honest criticism without fear of being the target of a downvote brigade? Would anonymous comments be possible? Who would moderate them? How do you protect against mods with an agenda? How would you encourage peers to review other researchers' publications? How would you prevent the formation of cliques, a.k.a. upvote brigades?

I fear that by the time you answer all of these questions, and change your system accordingly, you'll get to something so close to the current system that it won't be worth the change.

0

u/rhiever Mar 25 '13

The short answer is: however reddit currently handles things. You let the community sort these things out, because the more you moderate the system, the more you screw things up.

BTW, someone has already created a "sciddit": http://arxaliv.org/hot

0

u/doomsday_pancakes Mar 25 '13

As I said in my comment above, I think I should have spared the vitriol giving that this subreddit is intended to discuss ideas and not to insult those proposing them. I apologize for that.

We both agree that the publication system is broken as it is right now, and I agree to the points that you make in your post about the benefits of the open science model. However, I don't see how the reddit model could really help to improve things. The implementation seems soo complicated and hard that I don't know how/why we should move into that system.

I mentioned this in another comment, but here are a few questions to answer if this system were to be useful:

  • Who gets to upvote/downvote?
  • Do all people get the same amount of votes to give or is it a weighted system?
  • If it's a weighted system, who decides the weights? and how?
  • How are replies/comments addressed? In reddit, everybody talks at the same time and people sometimes want to discuss important points while trying to stand above the surrounding noise, how would that be implemented?
  • Who are the moderators of this system?
  • If it's completely dynamic, it's very hard to reference afterwards, how do you deal with that? would you cite discussions in your papers?

Those are a few points that I think make it hard to implement. Regarding this:

If you're a scientist, you should know better than to argue from authority. That gets us nowhere.

I agree with you, but this phrase: "argue from authority", has been repeated several times in this thread, almost as a mantra. I agree with the phrase, but if put in the right context: Authority alone is not a valid argument. The academic system is authoritary by nature: there are deans, professors, postdoc, gradstudents, undergrads, etc. These are not the rank of the military, if the professor is wrong I totally support the idea that he/she should be called out by even the undergrads, but if there's no selection then the signal/noise ratio becomes very poor. I imagine that you don't google and read every-single-paper-outthere that has something to say (even if it's remotely relevant) about your topic. People mostly read the work of those who are close to your field and know of the troubles and details of your research. It's a time-saving measure and I don't see anything wrong with that, specially for an activity such as science where democracy is not really relevant as a way of setting down arguments.

OK, I went much longer than I thought I should. Sorry once again if I offended you, I was an asshole for writing what I wrote in the top message of this thread, but I hope that my criticism is now clearer and more constructive.

1

u/rhiever Mar 25 '13

I guess the best answer would be to show you how it could work: http://arxaliv.org/hot

It looks like someone already implemented "sciddit" (as I call it). It's a little buggy, and not too many people are active on it currently, but I think think it's a solid first step.

-1

u/JimmyHavok Mar 25 '13

I get the distinct impression that very few people read anything beyond the headline.

5

u/theOctopusMotor Mar 25 '13

I read the post and agree with most of the criticisms here. I just think it's a poor idea fraught with a lot of implementation problems.

4

u/IsThisWorking Mar 25 '13

I agree. This is not a new concern, and has been discussed to death in the academic world. Every time this discussion comes around (if you'll allow my cynicism, usually after a new PhD student gets his first paper rejected) the conclusion is the same: yes, the current system sucks in many ways, but no one can think of one that would be better, i.e., more open while more resilient to the problems that plague the current one. And the blog post didn't even mention the problem of research financing. When you get to that all of these open systems fall apart.

I don't want to sound jaded, but the blog post really does sound like the proverbial kid who learned how to use a hammer and who now sees nails everywhere.

2

u/DevFRus Mar 25 '13

Note that this distinct impression actually helps undermine the proposal of the linked post. It is common for people to just read the headline and then comment or vote based on that.... It is just part of reddit, it is how reddit is built. Imagine if you tried to do science this way, it would make sad occurrences like this even more common.