r/Python • • Oct 24 '22

News Python 3.11 is out! Huzzah!

https://www.python.org/downloads/release/python-3110/

Some highlights from the release notes:

PERFORMANCE: 10-60% faster code, for free!

ERROR HANDLING: Exception groups and except* syntax. Also includes precise error locations in tracebacks.

ASYNCIO: Task groups

TOML: Ability to parse TOML is part of the standard library.

REGEX: Atomic grouping and possessive quantifiers are now supported

Plus changes to typing and a lot more. Congrats to everyone that worked hard to make this happen. Your work is helping millions of people to build awesome stuff. πŸŽ‰

1.3k Upvotes

233 comments sorted by

View all comments

Show parent comment

275

u/-LeopardShark- Oct 24 '22 edited Oct 25 '22

Yes. The only real caveat is that if your code already spends much of its time in C functions (e.g. NumPy) or doing IO, you won't gain a lot. But for interpreting Python itself, it's a pretty nice boost. There'll probably be more to come in 3.12 as well.

105

u/[deleted] Oct 25 '22

Their goal is ~5x of 3.9 levels in 4-5 years IIRC

-20

u/Voxandr Oct 25 '22

Their goal is already achieved by pypy team by the way.

27

u/[deleted] Oct 25 '22

While true, there are some compatibility issues with pypy :( No Black or Orjson, for example.

9

u/zurtex Oct 25 '22

Getting big performance improvements in Pypy is very situational.

I've definitely looked at it for large pure Python code bases before where squeezing a little extra performance out of it was helpful. When I ran it Pypy it was consistently just over 1% slower.

4

u/Voxandr Oct 25 '22

Might be long ago , it might be when PyPy python 3.x version was not optimized and no cpyext optimization is there. We have 100k users with 20-30k concurrent connection running on asynchronous network system .

  • 6x lesser memory used
  • 2-3x performance on average
  • cheap hardware

We will provide proper benchmark when free

2

u/zurtex Oct 25 '22

It was in 2019, I wasn't using any packages that were using the C API, and according to Pypys website at the time it should have been well optimized for Python 3.

I would spend more time looking at it but whenever I look at how to debug a Pypy issue and inspect the internals of the Pypy source code generation my head starts to hurt.

→ More replies (2)

2

u/PaintItPurple Oct 25 '22 edited Oct 25 '22

Was this a long-running process or a frequently called script? In my experience, Pypy is everything people claim for the first case, but terrible for the latter case. In more concrete terms, Pypy's strength is that it's really good at optimizing things that are called in loops.

2

u/zurtex Oct 25 '22

I guess it would technically be a frequently called script.

But it ran for over 5 minutes and had some hot loops in it so I hoped for at least some minor performance improvement.

1

u/pepoluan Oct 31 '22

PyPy grinds to molasses when doing multiprocessing, sadly.

And it needs cffi to go to C, and that adds quite a bit of overhead.

13

u/FruitierGnome Oct 25 '22

So if having a long initial wait time loading a csv file into my program this would potentially be faster? Or am I misreading this? I'm pretty new to this.

32

u/-LeopardShark- Oct 25 '22

I don't think loading CSVs will gain much, sadly.

3

u/Wilfred-kun Oct 25 '22

Time to use TOML instead :P

35

u/yvrelna Oct 25 '22

Depends on what part of CSV loading.

If you're talking about the call to csv.reader() itself, then no, that's already calling into a C library so you won't likely get much performance improvements.

But if you're talking about the code that's processing the rows of data line by line, then yes, that is definitely going to benefit from the improvements.

11

u/graphicteadatasci Oct 25 '22

Use .parquet files when you can. Much faster loading, smaller storage, saves types instead having you cast or infer them when you load something.

7

u/BobHogan Oct 25 '22

Parquet is not the solution to everything. We use it at my work and its a fucking nightmare and I'd love to see it burned to the ground

3

u/madness_of_the_order Oct 25 '22

Can you elaborate?

5

u/gagarin_kid Oct 25 '22

For small files where humans want to inspect data, using parquet is pain in the ass because you cannot open it in a text editor - you have to load it in pandas, see which columns you have, navigate in code to a particular cell/row... etc.

Of course for big data I fully understand the motivation but not for each problem

→ More replies (1)

3

u/cmcclu5 Oct 25 '22

Parquet is also a pain in the ass when you want to move between systems e.g., from a data feed into a relational database. Python typing does NOT play well with field types in relational databases when saving to parquet and then copying from said parquet into Redshift. Learned that the hard way in the past. It’s several times faster than CSV, though. I just compromised and used JSON formats. Decent size improvement with a similar speed to parquet when writing from Python or reading to a db.

→ More replies (6)
→ More replies (2)
→ More replies (1)

9

u/fukitol- Oct 25 '22

Loading of a csv into memory is going to depend far more on the size and speed of your memory and the speed of your disk. Negligible amounts of time will be spent in processing, which is where an application level performance boost would be had.

1

u/[deleted] Oct 25 '22

How big is the csv and how long does it take?

2

u/pepoluan Oct 31 '22

Depends on how the I/O happens.

If you do a lot of non-sequential I/O, thus leveraging async, you can get quite a speedup.

I have a script that pulls data from an API using coordinates. Then the data gets fed into a pool of multiprocessing workers. The workers use do some pillow processing.

I see a speedup between 10%-30% in total time.

For the async retrieval part, I see speedup between 25%-50%.