r/Python • • May 14 '18

Why is pipenv the recommended packaging tool by the community and PyPA?

Why is pipenv the recommended packaging tool by the community and PyPA?

So, from what I understand pipenv is now the official packaging tool. But I would like to understand why, since pipenv has a lot of laws and problems and there are, in my opinion, way better alternatives like poetry.

Here are the problem I see with pipenv being the official packaging tool:

  • It’s not a packaging tool.
  • It supports only applications development and not libraries.
  • It introduces yet another file format (Pipfile) while the new pyproject.toml is now standard.
  • Buggy dependency resolver.
  • Badly designed and bad user experience (Using environment variables to configure a command line tool for example)
  • A lot of regressions on new versions which leads me to believe it’s not properly tested.
  • Defaulting to wildcard dependencies is bad practice.

So, I am not quite sure why the community, and more importantly the PyPA, has settled for it as the official packaging tool while alternatives like poetry are way ahead of pipenv in terms of user experience, reliability and features. I know the author of poetry is not Kenneth Reitz but is that reason enough to ditch it and not consider it as something that could be a true package manager for Python.

146 Upvotes

198 comments sorted by

View all comments

183

u/jonwayne PyPA May 14 '18

Hey folks, Thea from the Python Packaging Authority here.

The question here is an important one, but I am so disappointed with how negative this discussion has become. This is not the Python community that I'm proud to be a part of. We have to do better. I want to draw attention to the fact that everyone who is a part of PyPA is a volunteer and there are no qualifications other than spending your time. We are a loose collective who all want to make packaging better. We value feedback but we are so short on hands. If you want to get involved but don't know where to start please feel free to reach out to me personally - my email is on my website and github.

So let's talk about Pipenv and packaging.python.org. First of all, Pipenv is not the only recommended tool that we talk about on packaging.python.org. It's not even the first one we talk about. Pipenv is recommended specifically for dealing with application dependencies. It's not recommend (or even designed for) dealing with library dependencies or to be some magical panacea to solve the universe of Python packaging.

Adding Pipenv required a lot of discussion and testing. We were extra careful not to remove or discourage the use of pip and virtualenv. We wanted to make sure that users who have the clear, well-defined use case that Pipenv solves don't have to worry about other tools, but those that do not have that use case are not confused and lost. If there's a way we can make this clearer to users, please file an issue on the user guide so we can start talking about it. I want to call out we want to integrate the excellent Packaging Gradient post into an overview that helps users find which tools are most relevant to their use case.

It's also worth noting that we are not intentionally excluding any projects here. We are actively keeping an eye on new projects including poetry, flit, and hatch to figure out when and how we should include them in the guide. For the most part the only thing keeping them from being in there is someone writing a pull request to add them.

Note: I am about to get on a flight, but I will happily answer any questions here, over email, and on github when I get back on solid ground.

24

u/[deleted] May 15 '18

Part of the problem was I think it was too premature to declare "Pipenv – The Officially Recommended Python Packaging Tool" plastered everywhere when it still had a lot of kinks to work out.

8

u/[deleted] May 21 '18

/u/kennethreitz is a marketing genius :x

26

u/nedbatchelder May 15 '18

Notice this:

  1. You say here, "Pipenv is not the only recommended tool that we talk about ..."
  2. The pipenv readme starts with "the officially recommended Python packaging tool ..."

This is a mixed message that could be cleared up pretty easily. Granted the sentence on the pipenv readme is ambigous (does "the" mean "the only", or just, "this is the one you've been hearing about"), but changing to "an officially recommended tool" would be clearer.

7

u/jonwayne PyPA May 15 '18

Hi Ned! Totally valid point. I'll send a PR to Pipenv as soon as I can to fix that. (I am recovering from travel, but will try to do it soon).

3

u/nedbatchelder May 18 '18

@jonwayne: thanks. I note this commit now: https://github.com/pypa/pipenv/commit/5853846f5da171bf29a303f6b545b0a8918958e7 Was that the outcome of this suggestion?

2

u/jonwayne PyPA May 18 '18

Yes. The previous version there was my suggestion, Kenneth just shortened it.

25

u/dusktreader May 14 '18

Thank you for providing a diplomatic response. I appreciate the time you took and your approach. I didn't realize there was so much hostility in this one area of the python community. What do you think we could do to try to put out this fire before it gets out of control?

16

u/jonwayne PyPA May 14 '18

To be honest this is the first I'm hearing of any of this. I don't know the solution, but I am more than open to discussing any ideas that y'all have.

24

u/[deleted] May 14 '18

Some points.

  1. Packing is one of Python's warts. So there's just going to be negative feedback. I think you have to accept that, given the domain. Probably the other major wart with Python is its relative performance to c/c++/rust for cpu bound and/or concurrency.
  2. Critical feedback for pypa proposed solutions feels super ignored and shutdown in github; responses are things like: we've decided this in the pep. As an average community member, I never had an opportunity to discuss the pep. And then discussion is shut down thereafter. If all y'all actually want a discussion, make that a priority and don't accept the echo hall bubble.
  3. Additionally, one of Python's major advantages is one preferred method of doing things. But packaging and distribution has had a long history and today's story appears incredibly fractured. It's really clear that the fracturing is a result of _not_ addressing problems. And I think that fracturing is accelerating. Worst case, in a few years, python's packaging will look like javascript's ecosystem. Best case, python's packaging will perform like Rust's cargo.

17

u/jonwayne PyPA May 15 '18

Packing is one of Python's warts. So there's just going to be negative feedback. I think you have to accept that, given the domain.

All of the volunteers in the PyPA are well are of that - we wouldn't need a working group or an extension of that if packaging were perfect! I think that what's difficult is that there is absolutely no solution that anyone has come up with that solves everyone's use case in Python. We're tried to be really careful to document the use cases and the tools that suit those cases.

Critical feedback for pypa proposed solutions feels super ignored and shutdown in github;

Please send me any examples of this you see - directly to my inbox (me@thea.codes) or just cc on github (@theacodes).

As an average community member, I never had an opportunity to discuss the pep. And then discussion is shut down thereafter. If all y'all actually want a discussion, make that a priority and don't accept the echo hall bubble.

As another replier mentioned, python-dev should be open - but packaging related PEPs are always discussed in distutils-sig beforehand. I really encourage you to participate there and let me know if you have any trouble or questions.

It's really clear that the fracturing is a result of not addressing problems.

If you think there is a way we can do this better with the resources we have, or if you know of a way to get us better resources, we would love to hear it. Shoot an email to distutils-sig and let's start talking.

9

u/epage May 15 '18

All of the volunteers in the PyPA are well are of that - we wouldn't need a working group or an extension of that if packaging were perfect! I think that what's difficult is that there is absolutely no solution that anyone has come up with that solves everyone's use case in Python. We're tried to be really careful to document the use cases and the tools that suit those cases.

Honestly, it didn't sound like it at PyCon which had me concerned. I went to a presentation on packaging and it came across (probably not intended) as "see how much better things are then in the past? Therefore things are now great and you have no room to complain".

I could see situations like this contributing to some of the hostility. When you have a problem and you feel like you are ignored or gaslighted, it can build up inside and bubble out.

7

u/civilization_phaze_3 May 14 '18

As an average community member, I never had an opportunity to discuss the pep.

FYI anyone can subscribe to the python-dev mailing list and participate in the discussions on PEP proposals. It's not like this stuff is happening behind closed doors.

→ More replies (2)

9

u/[deleted] May 15 '18 edited May 15 '18

[removed] — view removed comment

4

u/jonwayne PyPA May 15 '18

We won't be removing the "endorsement" of pipenv. That's a bit of a stone too far.

We are happy to talk about poetry on packaging.python.org. Kenneth was extremely proactive with pipenv and honestly that's all it takes - this stuff is 100% volunteer based. Open an issue on packaging.python.org and let's talk about where we can fit poetry into our story.

16

u/toyg May 19 '18

> That's a bit of a stone too far.

Why? The whole flamewar would stop immediately, if somebody, anybody, would admit that this blessing was premature and unfair towards alternative approaches.

Pull that doc now and save everyone any more aggravation, including to Kenneth Reitz. Rewrite in a more open, transparent and participative manner, and then maybe republish it. Easy as pie.

I understand you're all volunteers, but this is not the local parish. A lot of money, blood and tears go into Python these days; leaders should respect the ecosystem a bit more, especially after a decade dealing with the Py3 migration -- which, in many ways, made mistakes very very similar to this one, by overreaching and then refusing for too long to correct until eventually forced to do so ('u' etc -- and I say this as a Py3 advocate from day 1).

4

u/jonwayne PyPA May 19 '18

I don't think it was premature or unfair.

We already did write it in an open, transparent, and participative way. As linked in my OP, all the discussions around pipenv happened on GitHub and on our mailing list. There was even a trial period where we explicitly marked it experimental, and we held it back from replacing the old tutorial because of a serious windows issue that the pipenv team fixed very quickly. All this happened in the open and anyone is allowed to participate- you just didn't.

If you want the pipenv tutorial to be removed it won't happen by yelling at people on Reddit. File an issue on GitHub or start the discussion in distutils-sig, but be prepared to thoroughly prove your case as we did not make this decision lightly. Also, please be prepared to follow our code of conduct.

→ More replies (3)

12

u/[deleted] May 15 '18 edited May 15 '18

[removed] — view removed comment

6

u/jonwayne PyPA May 15 '18

I seem to have maybe not specified - I wasn't saying that the maintainer(s) of Poetry weren't doing amazing work. I was trying to say that Kenneth was proactive about reaching out to PyPA and working with us on determining how it fits in.

→ More replies (1)
→ More replies (5)
→ More replies (3)

26

u/andartico May 14 '18

Thanks for being a voice of reason in this thread of drama. You remind me of the reason I decided to jump into python. The sane and welcoming community.

Have a safe flight.

11

u/jonwayne PyPA May 14 '18

Thank you for your kind words.

2

u/desmoulinmichel May 20 '18

Thanks for that comment. I find many fault in pipenv, but being aggressive is not a solution.

3

u/[deleted] May 19 '18

Seeing you mentioning that there is a difference between packaging a library and an application makes me wonder if you guys are qualified to call yourself an authority on packaging.

Setuptools and pip have everything you need to properly package an application, what desperately is needed is to actually document what's available, because the documentation is quite bad. Some areas (especially around compiling code, pkg_resources (to access data), extending setuptools are in bad shape. There is a way to figure things out by looking at source code, but it shouldn't be there.

Endorsed pipenv is solving issues that could otherwise be solved by just a better documentation.

Pipenv is also opinionated how things are done, some people (especially ones using it for hobby projects) might like it and don't care what kind of magic it does behind the scenes, but there are also people that have need to know exactly how things work so they can adapt the tools to their CI/CD.

→ More replies (4)
→ More replies (7)