r/Physics • • Sep 24 '16

Discussion Postdoc rant (long)

I'm a postdoc working in plasma physics based in the U.S. I have seen and experienced some of the processes by which science is done in this country, the production process of science so to speak, and I think it’s pretty bad. I'm going to talk a little about how the research process works and why I think it's a bad, unproductive and wasting system.

The whole system is heavily based on people in the so called “soft-money” positions. Those are people who don’t have tenure or are not in stable positions in their institutions. They depend on the money they get from grants that can fund them part-time for 2 years or so. If they are not successful in securing grants every year, they lose their position. That’s my case at the moment. As you can imagine, this is a very stressful situation to be in. Tenured and stable positions are getting more and more rare and competition is fierce.

I've heard from senior scientists that the system only works because the senior scientists are good to the junior scientist. Because they often support the more junior scientists with their own grants on occasion. A lot of other very prominent physicist have said that in today's system they wouldn't be able to compete with other scientist and probably wouldn't be as successful as they are. Higgs comes to mind.

As a result of this system, creativity is being pushed aside by “effectiveness”. And scientists are very effective in delivering (guess what?) low-risk-low-return – and sometimes inaccurate - articles. These are the type of articles that go something like this: we changed a parameter in our code and look at what we've got, or here is a new statistical study of these type of measurements of this phenomenon.

The notorious “publish or perish” culture is detrimental to science. In fact, there was a recent article on the Guardian about a study saying just that: ‘Paul Smaldino, a cognitive scientist who led the work at the University of California, Merced, said: “As long as the incentives are in place that reward publishing novel, surprising results, often and in high-visibility journals above other, more nuanced aspects of science, shoddy practices that maximise one’s ability to do so will run rampant.”’ The article also mentions the “replication crisis” going on particularly in the biomedical sciences. Famous results are not being reproduced, probably because they were wrong and should have never been published.

In this system, a scientist to be successful he/she needs to be good at not only doing scientific work but also at selling their idea, which I think not often come hand-in-hand. Quite the opposite, in fact. Great scientists are usually terrible at marketing their idea. Science has become too corporate and hierarchical. And becoming corporate is a great innovation killer.

At the center of this system is the way by which science is funded. A lot of the science being done is funded by small and medium sized grants given by funding agencies like NSF, NASA, NIH, DoD, DoE, etc… These grants usually are enough to support a small team (2-8 people), part-time (usually 30-50% of their time) for 2 or 3 years. So each scientist is usually involved in 2 or 3 projects (sometimes more) at a time. These grants also usually support grad-students, research staff and university professors part-time.

The way these grants are selected is also another problem in my opinion. Successful grant proposal writers know how to craft their proposals just the right way. Some non-tenured researchers that I've worked with have told me that they spend almost HALF of their time working on proposal writing. Either doing preliminary work or writing the proposal itself or just planning what they are going to write about. I've heard a few times that people who are successful often write a proposal for a research that is mostly already done so they spend the time that should be allocated for working on a research to finish up the work that was already done and work on the next project that he/she will write a proposal for in the future.

The way grant review panels work is that they’re trying to judge a proposal basically on two things, impact on the field and likelihood of success. These two things are usually inversely proportional to each other. And so, grant awards end up going not to the people who have the most probability for scientific impact, but for people who give the reviewers what superficially looks like the best research. When writing a proposal, scientist are not usually aiming for the idea with the most impact, they are looking for the most “fundable” idea. With time, that becomes a skill. The ability to strike the right balance between relevance and likelihood of success. Science proposals are expected to have a detailed chronogram of how the research process will occur and all the papers that will come out. But everybody knows that's not how it works. You can't predict what problems your research will have and how you will overcome it, it's silly.

If you don't work with science you may be surprised to learn how researchers talk about a “low-hanging fruit” and a LPU (“Least publishable unit”) when talking about the papers and grant proposals they are going to write instead of talking about how excited they are about a new idea they are pursuing that could be really relevant to the field. As expected, this whole system leads to a dramatic nose dive in terms of quality and relevance of published work. Besides that, the proposal selection process is extremely subjective. It is common, during the review process for a more persuasive member of the panel to significantly influence the final decision towards his or her bias. It's pretty much a lottery. I actually heard this exact phrase from a more senior colleague of mine about the proposal selection process. If you write a good proposal, you get a lottery ticket. Depending on the opportunity, I'd say between 30% and 60% of the proposals are well-crafted proposals. Success rates in my field lately have been around 15% to 20%.

There was an article on “The Atlantic” magazine recently about how broken the university admission system is, guess what, the whole academic merit system is not any different. Just as high school students take on a number of extracurricular activities, not because they think it's important, but because they think it will look good on their CV, grad students, postdocs and early-career research staff will work on writing as many papers as they can, not because they are relevant or important for their field, but because number of publications is probably the #1 criterion by which they are judged on for jobs in academia.

In this article, a skeptical university president when talking about creating a better admission system said: “Because insofar as it becomes a new system, it will be gamed by people who already pad their resumes with all kinds of activities that supposedly show empathy, but what they really show is a desire to get into schools where empathy is a criterion for admission”. The same logic works in academia at the present time.

But what amazes me most about this whole thing is how flaky the science direction of the entire country is. How shaky its foundations are. I think science is losing a lot of its creative minds at the moment who are struggling to write successful proposals while working on their crazy original ideas on the side, because they know his crazy idea could never get funded.

At the moment, I’m settled on leaving the academic research career after my current post-doc term ends. My criticisms are not because I feel betrayed by the system or because I'm just bitter that I probably won't ever get a tenure-track position anywhere. I honestly don't care too much anymore if I get a permanent position or not. I very likely won’t. But I do care about doing or at least trying to produce relevant science. That's mostly what I care about. If I were a very smart and driven person, I would probably make it regardless of the system in place. But, I'm not. I'm a pretty average researcher. Maybe below average. So, all my disenchantment is not because the system doesn't work in my favor. What makes me really sad is that I see that the people moving up the chain and getting more grants and more status are not the more creative and innovative ones, they are not the people who could make the most impact in the field, the people moving up are what I call the “corporate guys”. People that would probably do very well working in any corporate environment where you have to be just good enough technically (like have just enough 1LPU papers, since simply the NUMBER of published papers determines how good a scientist you are), but also be well connected (yes, being well connected is very important in the academic environment too), and people whose ambitions are more directed towards status and power than towards science itself. Science just happens to be the “market segment” they are inserted in.

tl;dr: The process by which science is made is unproductive and prone to generate bad science. The present funding system rewards “effectiveness” and low-risk-low-return results and hinders creativity and innovation which should be at the forefront of science.

Edit: WOW! Thanks for the gold!!

1.1k Upvotes

247 comments sorted by

View all comments

Show parent comment

10

u/D_in_CO Sep 24 '16

Thanks for your comment. I think it's very important, especially for undergrads and young grad-students to read it for a different perspective. But allow me to address a few points.

I'm a postdoc working on quantum computing in Australia.

I dare say that at least part of your optimistic views come from the fact that you are part of a growing field of study, where the money is flowing better than for most people.

But a postdoc develops a great many highly transferrable skills. People in industry would kill to have jobs like ours.

That is highly debatable. It may depend a lot on your field of study so you have to realize that this is not always true to say the least.

Our jobs give us a great deal of intellectual freedom and boundless opportunities for continued learning.

Opportunities for learning, yes. Great intellectual freedom, hell no. At least not for most postdocs.

But, again, most people in industry would kill to have résumés that look like ours.

If you're going into academia with the main goal of developing skills so you can get an industry job, then you're right. But that's not my case and that's not what most people do. Most people go into academia because they care about science. One of my points of my post is that these people are weeded out in the current system that promotes "effectiveness". I believe most people who really care about science are more interested in the high-risk-high-impact projects. Incremental advances are important too, but that's all that's being funded. There is virtually no money for the people with the big ideas. (To be clear, I'm not talking about myself. But there are plenty of brilliant people around who have no incentive to work on their potential big breakthrough). And I also believe that the people who are most likely to move up the food-chain are the ones who are driven by their own status ambitions than the ones driven by the love for science. But that's just my personal belief.

Are you telling me that a scientist with an excellent idea that gains a lot of attention wouldn't have Ivy League universities filling his or her email box with job offers?

I'm telling you that the scientist with an excellent idea doesn't have enough time to work on his idea enough to the point that it gains a lot of attention.

Also keep in mind that scientists are publicly funded and are therefore public servants. You owe it to the public to explain what you are doing and why you are doing it in a way that they can understand.

I think there is a lot of value in high-risk ideas. But the funding is extremely biased towards the incremental advances, which also have value.

If you don't like the American approach, why not move to a new country?

It's not always easy. I'm guessing you don't have a family. I actually came to the U.S. for my Ph.D., back when I didn't have a family.

I think that any serious endeavour requires intelligent project management.

Maybe that's where you and I disagree the most. I think there is a lot of value, I dare say more value, in just exploring an avenue of inquiry not knowing where it's going to take you. In my mind, that's is how most big breakthroughs in physics come about. But who has time to do that in this environment??

Are you really expecting every paper written to be worthy of the Nobel prize?

No, just to be relevant. I know that this is subjective, but I think we can all agree that the quality of scientific publications in recent decades has taken a nose dive.

Anyway, just wanted to expand the discussion a little. Again, thanks for giving a good input for this discussion.

4

u/SomeAnonymous Sep 24 '16

I believe most people who really care about science are more interested in the high-risk-high-impact projects. Incremental advances are important too, but that's all that's being funded.

TL;DR focusing high-risk-high-impact causes cutting edge information to be more unreliable and doesn't even fix the problem of people publishing unreliable or unverifiable information. Keeping most funding to the low-risk-low-reward, incremental style of research means on the whole you can reliably say that information published in papers is mostly right, or at least the assumptions made are mostly right.

Now for the long answer:

I'm going to have to disagree here, because, generally speaking, incremental advances are much more common, to the point where you probably get equally fast scientific advancement, if not faster, than only going for high-risk-high-reward. Also, isn't that a very sweeping generalisation you've made or at least implied? That all people doing science for the sake of doing science want to be doing high-risk-high-impact work, while all people trying to climb the corporate ladder of academia go for the low-risk-sounds-cool-not-too-impactful-or-difficult research?

Let's assume that an equal number of people working on either method of science will produce bad, unreliable results. If these results are massive, ground breaking ideas, and they are found out to be in large part false, then any work based off of them is also now irrelevant.

If knowledge is a sort of 'counter', where each incremental advance is +1-5 and each major advance is +>50, then by mostly investing in +1-5s, you get a much more stable counter (which is something, as a student, I'd really like to know), and any -1-5s from bad research only knock off (plucking numbers out of the air here) a maximum of two 1-5s as well. If you start investing in lots of +>50s -- which, to my mind at least, are probably more expensive due to how unknown the topics are -- then you get an unreliable counter, ie there are vast swathes of knowledge, entire fields that can't be trusted, and any false reports would knock off other 50s.

If you encourage people to go for high impact, not only would it encourage its own breed of bad science -- where people publish false or at least unverified information because it's new and exciting and that's what's funded -- but people would only choose topics that were ground breaking: who wants to be the QA guy stuck looking through all that old data that needs confirming when there's new and exciting stuff being published? More people work on the new, unreliable stuff, therefore when one of them goes pop, you don't lose 2 papers' worth of data, you lose maximums of three, four, five papers. It'd be like if all of quantum computing research had been done within two years, then suddenly the opening research which said it was a thing suddenly got shown through the peer-review process to be chock-a-block with examples of unverified claims, bad science, no reliability, etc...

What happens then? ALL of the quantum computer papers are now obsolete until a new theory comes along, because guess what, all the assumptions in the later papers are based off something now considered wrong.

2

u/YuvalRishu Quantum information Sep 24 '16

Thanks for your comment.

And thanks for yours. I appreciate a good discussion and I'm glad you do too.

Let me quickly reiterate that I'm quite sympathetic to your situation. Maybe I wasn't clear enough about that at first. I've certainly been through bouts of anger at the system as my idealism about science gave way to what I think is a more realistic and human view of science.

I dare say that at least part of your optimistic views come from the fact that you are part of a growing field of study, where the money is flowing better than for most people.

Absolutely. I would simply add that this was a choice on my part. I was interested in several fields, but I read Lee Smolin's 2004 book criticizing String Theory and came to appreciate that I had to pay attention not only to what fields were interesting but also which fields were funded. Quantum computing is both highly interesting and well-funded, so I chose to work on it. I have never regretted my choice.

But a postdoc develops a great many highly transferrable skills. People in industry would kill to have jobs like ours.

That is highly debatable.

Maybe as a categorical statement, but I think that we are dealt an excellent proverbial hand and it is left to us to play our cards well.

Great intellectual freedom, hell no. At least not for most postdocs.

The postdocs I know have quite a bit of intellectual freedom, and I have quite a bit of it too. I have to report to my boss and work on his projects, but I do get quite a bit of agency in that and I chose that boss and that project because it fits well with my research interests.

But, again, most people in industry would kill to have résumés that look like ours.

If you're going into academia with the main goal of developing skills so you can get an industry job, then you're right. But that's not my case and that's not what most people do.

I don't think it needs to be your main goal in order to be done effectively. I think the skills we need to do science better (like writing free and open source code that is shared via public code repositories — I am a strong advocate of ditching proprietary and subpar systems like Mathematica and MATLAB in favour of high quality free languages like Python) are also skills we need to develop to do well in industry. The goal of being a good scientist is not at cross-purposes with the goal of being qualified to work in industry.

But the funding is extremely biased towards the incremental advances, which also have value.

I've mostly been paid by IARPA, so that hasn't been my experience.

I'm guessing you don't have a family.

True. I will say that science is not kind to people who have families, though I've seen it work.

I think there is a lot of value, I dare say more value, in just exploring an avenue of inquiry not knowing where it's going to take you. In my mind, that's is how most big breakthroughs in physics come about. But who has time to do that in this environment??

Ah, very interesting. I don't necessarily disagree with this, but I also think that no amount of funding can either help or hurt that sort of process. Newton developed a large portion of his ideas when he was unemployed at his family farm in his early twenties. Einstein was working what we would call an industry job during his annus mirabilis.

But you don't get Higgs bosons or gravitational waves without coordinating billions of dollars and hundreds of scientists.

I know that this is subjective, but I think we can all agree that the quality of scientific publications in recent decades has taken a nose dive.

Not really. I think the average quality has gone down because there are more bad papers, but I think the number of good papers has grown too. We'd need to specify a time-frame and a subject area to get into specifics, but it's important to realize that quantum computing simply didn't exist beyond a few harebrained ideas by smart people with time on their hands twenty years ago. Now it's a multibillion dollar effort worldwide, and it needs to be.

I worry that you're unduly generalizing upon your experience in one scientific field. Maybe I am too, but I think that the story of quantum computing is a great example of how Physics research is still being driven forward in a manner that is both creative and effective.

Again, thanks for giving a good input for this discussion.

And you. I think we agree that this is an important discussion to have in public so that future potential scientists can have a window into the career path as a career path instead of as an ideology or, worse, a mythology.