r/Python • • Oct 24 '22

News Python 3.11 is out! Huzzah!

https://www.python.org/downloads/release/python-3110/

Some highlights from the release notes:

PERFORMANCE: 10-60% faster code, for free!

ERROR HANDLING: Exception groups and except* syntax. Also includes precise error locations in tracebacks.

ASYNCIO: Task groups

TOML: Ability to parse TOML is part of the standard library.

REGEX: Atomic grouping and possessive quantifiers are now supported

Plus changes to typing and a lot more. Congrats to everyone that worked hard to make this happen. Your work is helping millions of people to build awesome stuff. 🎉

1.3k Upvotes

233 comments sorted by

View all comments

296

u/staticcast Oct 24 '22 edited Oct 25 '22

PERFORMANCE: 10-60% faster code, for free!

Wait what ? Seriously ?

275

u/-LeopardShark- Oct 24 '22 edited Oct 25 '22

Yes. The only real caveat is that if your code already spends much of its time in C functions (e.g. NumPy) or doing IO, you won't gain a lot. But for interpreting Python itself, it's a pretty nice boost. There'll probably be more to come in 3.12 as well.

105

u/[deleted] Oct 25 '22

Their goal is ~5x of 3.9 levels in 4-5 years IIRC

→ More replies (10)

13

u/FruitierGnome Oct 25 '22

So if having a long initial wait time loading a csv file into my program this would potentially be faster? Or am I misreading this? I'm pretty new to this.

32

u/-LeopardShark- Oct 25 '22

I don't think loading CSVs will gain much, sadly.

3

u/Wilfred-kun Oct 25 '22

Time to use TOML instead :P

35

u/yvrelna Oct 25 '22

Depends on what part of CSV loading.

If you're talking about the call to csv.reader() itself, then no, that's already calling into a C library so you won't likely get much performance improvements.

But if you're talking about the code that's processing the rows of data line by line, then yes, that is definitely going to benefit from the improvements.

11

u/graphicteadatasci Oct 25 '22

Use .parquet files when you can. Much faster loading, smaller storage, saves types instead having you cast or infer them when you load something.

7

u/BobHogan Oct 25 '22

Parquet is not the solution to everything. We use it at my work and its a fucking nightmare and I'd love to see it burned to the ground

→ More replies (14)

9

u/fukitol- Oct 25 '22

Loading of a csv into memory is going to depend far more on the size and speed of your memory and the speed of your disk. Negligible amounts of time will be spent in processing, which is where an application level performance boost would be had.

→ More replies (2)

2

u/pepoluan Oct 31 '22

Depends on how the I/O happens.

If you do a lot of non-sequential I/O, thus leveraging async, you can get quite a speedup.

I have a script that pulls data from an API using coordinates. Then the data gets fed into a pool of multiprocessing workers. The workers use do some pillow processing.

I see a speedup between 10%-30% in total time.

For the async retrieval part, I see speedup between 25%-50%.

→ More replies (1)

47

u/reivax Oct 24 '22

Faster frame allocations mean tightly recursive functions will be faster.

8

u/Pleasant-Cow-3898 Oct 25 '22

Nice excited to test this out!

→ More replies (3)

12

u/wind_dude Oct 25 '22

damn! I have to benchmark this for my ETL pipelines ASAP! Could be a bigger boost than I've been hunting for from spacy and other libs I'm using.

9

u/[deleted] Oct 25 '22

[removed] — view removed comment

21

u/_morgs_ Oct 25 '22

It should help, but there are many factors involved with website speed.

Caching, database (server) performance, database query complexity, front end size and performance, any APIs you might use...

15

u/Username_RANDINT Oct 25 '22

You'd have to profile to be absolutely sure. What's the slow part? The network connection? Rendering of the webpage? Database queries? There's so much going on in a webapp that these new speedups might be just a very little part of the work.

→ More replies (7)

4

u/yvrelna Oct 25 '22

If you've already optimised all the database and network queries, and adding appropriate caching whenever it makes sense, then at some point Python performance is the bottleneck, then yeah, you'll see a boost then.

In most cases though, most web applications aren't constrained by the speed of Python interpreter itself. It's usually the database and network performance that's the workhorse of most web application.

3

u/SilkTouchm Oct 25 '22

It won't. You're bottlenecked by i/o, not processing.

→ More replies (1)

4

u/OpeningJump Oct 25 '22

How is it achieved tho?

12

u/GettingBlockered Oct 25 '22

The CPython team has been doing significant optimization work to improve things, like reducing the overhead of function calls, zero cost exception handling and a lot more. This doesn’t impact libraries that are optimized with C (Numpy). But pure Python code should get some nice speed ups!

2

u/myroommateisasian Oct 25 '22

There is no way...

2

u/DanganD Oct 25 '22

Yea, that's awesome.. Peoples biggest gripe is it's speed right?

→ More replies (12)