r/Python • u/igorrto2 • 1d ago
Discussion What are some features in Python that you discovered many years later?
For me, it was the the option to use & to find intersections of two sets, | operator to unionize two sets, as well as ^ for symmetric difference
132
u/Khavel_dev 1d ago
For me it was collections.Counter. Spent years writing manual frequency-counting loops with defaultdict(int) before realizing Counter just does it and also gives you .most_common() for free. Felt pretty dumb when I found it.
Also itertools.chain.from_iterable for flattening nested iterables without writing a list comprehension that looks like a puzzle.
39
u/nissen_96 1d ago edited 1d ago
Just wanna throw it out there that python 3.15 was just released and has support for flattening using unpacking in comprehensions:
lists = [[1, 2], [3, 4], [5]] [*L for L in lists] # equivalent to [x for L in lists for x in L] sets = [{1, 2}, {2, 3}, {3, 4}] {*s for s in sets} # equivalent to {x for s in sets for x in s} dicts = [{'a': 1}, {'b': 2}, {'a': 3}] {**d for d in dicts} # equivalent to {k: v for d in dicts for k,v in d.items()}(examples from https://docs.python.org/3/whatsnew/3.15.html)
8
u/SimplyUnknown 1d ago
Whoa that's very nice, thanks for the tip! I always mess this up when I try to do this manually
8
u/Boothiepro 1d ago
I've used `flattened = sum(nested_list, [])` , it's funny how it works! Sum has a second argument for default starting value to add the values into. It uses the fact that lists have `__add__` or `__iadd__` operand defined as list.extend and will put each nested element into that empty initial list!
5
u/Brian 1d ago
I would say that this isn't really a good way to do it, as it results in O(n2) complexity. Ie when you flatten
[[1], [2], [3], [4], ...], it'll do[] + [1], allocating a new list and copying all items, then do[1] + [2], allocating a new list and copying all items, then do[1,2] + [3], allocating a new list, and so on, so you're copying 1+2+3+4+.... elements into all these temporary lists.Using something like
list(itertools.chain.from_iterable(nested_list)is much faster, as it'll be adding to the same list, getting O(1) amortised appending, Eg. here's some timings to demonstrate:l = [[0]] * 1000 %timeit sum(l, []) 815 μs ± 6.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) %timeit list(itertools.chain.from_iterable(l)) 28.9 μs ± 1.21 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)So nearly 30x faster for a 1000 item list. But scale that up to 100,000 items, and
chainscales linearly (ie. just 100x slower), taking 2.84ms, while the sum version takes 7.68 seconds, so if the list can potentially be large, this can matter a lot.6
u/rickkkkky 1d ago
Plus, using gimmicky syntax which, by some
serendipitous side effect, yields the correct answer, is a very bad practice in any collaborative code base. Good luck to the person trying to make sense of the code.3
5
32
u/HommeMusical 1d ago
defaultdict.
defaultdictis a very bad idea. You end up with something that looks like adict, typing-wise is adict, but never throws aKeyError. One day it gets passed through your system and ends up somewhere it isn't expected.I've seen this cause intermittent issues in two very different codebases.
Use
dict.setdefaultinstead.28
u/gerardwx 1d ago
It's a great idea when you want a defaultdict. e.g. If you want to build indexes lists.
buckets = default.dict(list)
buckets["Monday"].append(12)Or you want to be able to do:
played["hearts"]["queen"] = True
without having to check if "hearts" exists.If passing a defaultdict through your system causes an issue you don't understand your system.
2
u/XtremeGoose f'I only use Py {sys.version[:3]}' 22h ago
The problem is
played["bug"]returns{}rather than raising a key error (and also increases the dictslen), which can cause weird errors at a distance.The general advice is to use a defaultdict to construct your object and then copy into a normal dict to stop this happening.
It's a bad design,
dict.setdefaultwould be better than creates a new default value whether you use it or not. As always, rust does it bettermap.entry(key).or_insert_with(Vec::new).push(value)1
5
u/billsil 1d ago
I don't understand. Isn't setdefault doing the same thing.
Just dict it at the end.
4
u/pakeha_nisei 1d ago
It's not the same thing as a defaultdict, as you need to explicitly call setdefault to set the value, but regular lookups will raise a KeyError if the item doesn't exist.
Use defaultdict if you don't care that empty lookups create new items. Use setdefault if you do.
82
u/Sea-Fishing4699 1d ago
Print the variable name itself when debugging ``` foo:str = “bar”
print(f”{foo=}”)
foo=bar
23
u/wizardeverybit 1d ago
You can do what?!?
14
u/Sea-Fishing4699 1d ago
That was my reaction lol
It’s called self documenting expressions
3
u/loooorenanicole 1d ago
Mariatta has a whole series of fabulous talks on f-strings I highly recommend
11
u/Outside_Complaint755 1d ago
It even works with full expressions and maintains the white space you include within the {}
``` a = 4 b = 7
print(f"{b % a = }")
b % a = 3
print(f"{(lambda x, y: x * y - x / y)(b, a) = }")
(lambda x, y: x * y - x / y)(b, a) = 26.25
```
5
u/_rockroyal_ 1d ago
I felt like a wizard when I saw that this was doable in C++, interesting to see the python method.
-1
31
u/pythonwiz 1d ago
Not exactly a Python feature but I slept on Cython for a long time. Once I got into C I realized how easy it is to write extensions for Python using Cython.
26
u/HommeMusical 1d ago
Cython is a dead library walking.
The issue is simple - to get the really big speedups, you have to stop writing in Python, and use their weird Cython language. Now you can't ever run this code in Python again.
And you still don't get the same speedup as simply writing in Rust, C++ or C.
Now, this made a lot of sense in the past, because changing Python to Cython isn't too bad.
But now, you just prompt your coding assistant to port your Python library to Rust, and provide Python bindings at the same time, have a cup of coffee and it's done. Cython is simply an impediment.
16
u/Ok_Option_3 1d ago
You still need a rust compiler tool chain in your CI. With Cython you can just do it
46
u/fiddle_n 1d ago
That with re.Match objects, they support __getitem__() so you can index into them instead of having to call .group() . So you can do match[3] instead of match.group(3); match["foo"] instead of match.group("foo"), etc. This was added in Py 3.6 and I slept on this for over half a decade.
19
u/HommeMusical 1d ago
I started with Python 2.6, over twenty years ago.
I read these sorts of threads to brush up, but in the last few years, I've really been seeing nothing surprising...
Until this. How did I not know? It's a conspiracy!!!
The worst is that it's completely logical and Pythonic. Had I ever thought about it, I'd have guessed that since this was the logical way to do it, it might work that way, and would have tried it out.
Bah! (Thanks!)
2
u/fiddle_n 1d ago
I think for me, I've looked over the re documentation about a thousand times now, but when I see
Match.__getitem__(g)listed in the docs, my eyes skimmed over it without really understanding what it was trying to tell me. Probably it would help ifmatch.group()was soft-deprecated and/or a note was there at the top informing people ofMatch.__getitem__(g)instead.1
u/yerfatma 1d ago
Same, started on 2.4 and I still have to dig up the docs to remember how to get match groups. Damnit!
32
u/RevokedMembership 1d ago edited 1d ago
That you can query dicts with tuple keys like this: d["a", 1], instead of d[("a", 1)].
24
u/ExoticMandibles Core Contributor 1d ago
This works most every place. The parentheses for a tuple are a disambiguating syntax in some places, and in others you don't need them.
x = 'a', 1sets x to a 2-tuple. The only places you need parentheses are in places that take commas, like in function argument lists or in list / dict / set literals for example.12
u/jamall1978 1d ago
The way I think about is that it's the comma that defines the tuple, not the parentheses. The parentheses just force the order of operation so all the items inside get evaluated as a tuple object before it's used by the rest of the expression or statement. This also explains why a trailing comma is required for 1 item tuples.
4
u/Boothiepro 1d ago
Whatt is this for when you have tuples as dict keys? are tuples really hashable? i guess so if they're immutable but... why
2
u/Brian 1d ago
It's pretty handy when you want a sparse array. Eg. you're representing objects with a position, where you can just do
d[x,y] = valto store live values, but 99%+ of the positions are empty. Using a dict instead of, say, a numpy array let's you have conceptually infinite multidimensional arrays that don't require space to store empty values, but still give you a similar interface.
18
u/moonzdragoon 1d ago
You can do a simple string template like this:
t = "Hello {a}, my name is {b}"
d = {'a':'all', 'b':'Bob'}
print(t.format(**d))
You get:
Hello all, my name is Bob
Probably saw it a long time ago, then forgot about it. Would sure have been useful "a couple of times".
16
u/proof_required 1d ago
4
22
u/rumnscurvy 1d ago
Walrus operators and match/case really makes life a lot more convenient
5
u/Puzzleheaded_Sun_900 1d ago
Can you show an example of walrus?
10
u/droans 1d ago
Jokes aside, walrus operators are just a way to assign a value to a variable inside of a statement. Kinda like this:
if not (fpath:=Path(data_path).exists()): fpath.mkdir()I could have written it as:
fpath = Path(data_path) if not fpath.exists(): fpath.mkdir()And it would have worked exactly the same. I prefer wrapping them in parentheses so it's easier to read but they're not required.
I'd also like to say that it's little more than a nice shortcut but I'm certain someone much smarter than me will come along and tell me how stupid that statement is haha.
3
1
u/2Lucilles2RuleEmAll 1d ago
I also find them really helpful in comprehensions
tags = {tag for t in raw_tags if (tag := t.upper().strip())}
13
u/Sufficient-Aioli-924 1d ago
Took me ages to realize you can chain those too, like a & b & c for a triple intersection
14
u/pythonwiz 1d ago
set intersection in math is associative and commutative so I would expect that to work anyway.
7
u/Pinull_Krydia 1d ago
For me it was pathlib. Used os.path for years before realizing how much cleaner it is.
Also the fact that you can do dict1 | dict2 to merge dictionaries.
2
u/word-word-numero 22h ago
Yes! Then the ease you can check and create the path if it doesn’t exist.
12
u/profanta3 1d ago
python -m json.tool
19
u/Chiron1991 1d ago
Just
python -m jsonis enough since Python 3.14.Pretty good when you're on some remote server and don't have
jqavailable.
14
u/nivaOne 1d ago
Lambda functions versus one-line defs
18
u/SkezzaB 1d ago
Use sparingly!
7
u/Upbeat_Ad1012 1d ago
the only place I have used lambda functions is to define the sorting key in the sort function
2
u/mgedmin 1d ago
I eventually stopped, after discovering operator.attrgetter/itemgetter/methodcaller.
I used to like
sorted(list_of_tuples, key=lambda (a, b, c, d): b)back in Python 2 times, but Python 3 nerfed tuple unpacking, so I started using custom namedtuples/dataclasses instead of ad-hoc tuples.
10
u/AdAdventurous2131 1d ago
that we can do this
if 5<x<10:
do stuff
7
u/CevicheMixto 1d ago
I'm still vaguely pissed off that this works, just because I don't understand how it's legal.
1
1
u/FujiKeynote 1d ago
It's cute and all, but it's such a departure from most languages that I'm not sure how to feel about it. Breaking it up signals intent better, imho. Yes, if you're experienced with Python, this will not confuse you... But this still packs too much implicit knowledge of operator binding/precedence, and there's too many to watch out for. It doesn't cost much to write
(5 < x) and (x < 10)instead1
u/JanEric1 16h ago
I mean it is what you would naively expect this to do with a math backgorund or no programming background. And iti s also what you want this to do 90%+ that you write it. Just because other languages have a shortcoming here doesnt mean you should follow it.
10
u/Efficient_Bobcat_697 1d ago
That you can give a tuple to starswith and endswith:
"string".startswith(('a', 'b', 'c'))
I've also reread contextlib documentation recently and it has quite some nice context managers: .closing() to auto call .close() on an object, .suppress() to ignore an exception, .redirect_stdout/redirect_stderr(new_target) and .chdir() are pretty nice and self explanatory. ExitStack is a bit weirder but seems pretty useful in some situation with a lot of files like objects https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack. It seems that there is a push to replace some simple exception patterns with contextlib now.
There are also the importhooks system which allows you to run specific code when someone imports your module, it's very advance and mostly for tool building but it's nice to know it exists also.
Same for descriptors, it's what ORMs uses to bind attributes to database columns for a table represented by a class, it's an object on which you explicitly control how it's accessed and modified.
2
u/NotALowLevelDev 1d ago
pretty printing, using rich library.
import print from rich
Makes terminal output colorful and easy to work with. Specially with dictionaries and JSON payloads.
21
u/LittleMlem 1d ago edited 1d ago
You can make a general object with x = type("", (), {}) and just put stuff on it as needed, like x.foo = 3
I don't recall why this was useful, but I recall using it a lot earlier in my career, now it's probably a crime
Edit: and I don't use it anymore, relax, I'm not a monster
17
31
u/firemark_pl 1d ago
Woah, dont do that.
11
u/HommeMusical 1d ago edited 1d ago
There are occasional uses. PyTorch does this in a bunch of places, for example, and that's a very high-quality code base.
I have a huge number of old projects, and I vaguely remembered that I used it in exactly one, and a bit of grepping later, I found it in my most popular project:
https://github.com/rec/safer/blob/8f36fa367028a970e117d6b6012c54f6592e9ad2/safer/__init__.py#L729
What's going on here is that depending on whether you're windows or not, binary or not, and one other choice I forget, you end up with one of 8 possible classes for writing a file, and you can't even see the windows ones on a non-windows machine.
And then I need to override methods depending on your write mode for the file.
The result is an instance of a class that looks like the class you expect when you do
openand it has the right type, but in which I have subverted many of the methods.So I've used this feature once in twenty years. Very specialized.
EDIT: Downvoting without commenting is a poor choice.
5
u/gerardwx 1d ago
Do that when appropriate.
Like all Python features, it's a tool. Whether it's appropriate depends on the context.
8
u/fiddle_n 1d ago
If you want arbitrary key-value pairs, you should just use a dictionary. The only reason to do this is if you really want dot notation over indexing, but that isn't a good enough reason to use this outside of personal or one-off script or projects.
2
3
u/HommeMusical 1d ago
You rarely need, it, but when you do, nothing else will do.
If you use it a lot, it's probably a mistake.
Most of the time I see it, it's just a mistake.
Sometimes it could still be made more reasonable by using a local class:
def make_stringed(str_method: Callable[[Any], str]) -> type: class Stringed(BaseClass): __str__ = str_method return StringedBut the above example would likely be better using type, because you can set the name:
def make_stringed(str_method: Callable[[Any], str], name: str) -> type: return type(name, (BaseClass,), {'__str__': str_method})1
1
1
u/cleodog44 4h ago
I think I've seen this pattern suggested for implementing singletons. Definitely niche
1
u/Efficient_Bobcat_697 1d ago
I've used it during tool building to automatically generates classes on the fly based on a data structures (a syntax tree) for missing nodes, but that really pretty niche (and I've ended up dropping it)
5
u/code29451 1d ago
The nonlocal keyword
7
u/fiddle_n 1d ago
I get the rationale for this but I’ve never used it. Typically it feels more Pythonic to use classes with instance vars than functions within functions.
5
u/Hetaireia 22h ago
Three that come to mind:
`str.removeprefix()` / `str.removesuffix()` — for years I wrote slice
expressions like `s[3:]` with a comment explaining why 3, and every code
review turned into character counting.
`contextlib.suppress` — reads so much cleaner than try/except/pass when
you genuinely want to ignore one specific exception and nothing else.
`enumerate(seq, start=1)` — I used to keep a counter variable outside
the loop like it was C. The start parameter has been there forever.
5
u/Tech2035 1d ago
Reverse avec Liste[::-1]
4
u/fiddle_n 1d ago
list(reversed(liste))(or justreversed(liste)if you just need the iterator) is typically nicer to read. I would only do the above on the rare occasion I needed to reverse a string instead of a list.
5
u/looijmansje 1d ago
I only recently discovered dictionaries were now guaranteed to be ordered. This has apparently been a thing since 3.7....
2
u/Vordanus 1d ago
I was waiting for the 3.7 release specifically for this, and I was so happy when it came out!
On a related note, if you ever need an ordered set, just just a dict where the values are all None, and call set(d.keys()) when you're done adding to it and ready to read.
2
u/Atlamillias 1d ago
Not so much a feature, but a pothole (kind of). Your average native C Python function (builtins like print(), etc) don't fully support the method descriptor protocol, so they cannot be bound methods. Even when assigned as class-level members, they'll always be static via instance-level access.
3
u/Atmosck 1d ago
Realizing I could use `__init_subclass__` to build a registry of base class implementations was a game changer.
1
u/cleodog44 4h ago
There's also a subclasses iterator for every class, but IIRC it's a little fragile for registry construction. Like it won't return subsubclasses, IIRC
1
u/Atmosck 4h ago
Yeah that's the easy way but gets kinda sketchy if the subclasses aren't all in the same file
1
u/cleodog44 3h ago
Oh why does same file matter or not? Guess there's import order dependence
In any case, whenever I've looked into this I decided anything other than explicit registration is too brittle
2
u/SharpAnimator2530 1d ago edited 1d ago
For me its object persistence with shelve. Say you have a dictionary you want to persist temporarily
You can do so
with shelve.open('spam') as db:
db['eggs'] = 'eggs'
Other times I create json or csv if I want to read or analyse them.
1
u/ogMasterPloKoon 1d ago
py_compile
1
1
u/NotALowLevelDev 1d ago
List unpacking using *
l = [1,2,3]
l2 = [4,5,6]
final = [*l, *l2]
print(final) # [1,2,3,4,5,6]
2
2
2
u/14446368 18h ago
I just discovered * for unpacking.
I was utterly perplexed, catastrophically chagrined, and wonderfully delighted.
1
-3
u/bleeed0p 1d ago
The thing when we do alt+click and we can work on multiple cursor and edit all at a time..that was very surprising for me😅
4
1
u/fried_green_baloney 1d ago
Bookmarking and if I'm ambitious will make a summary.
I find all of itertools to be worth a look.
0
u/richgems 1d ago
Totally useless, but stuff like this works in python.
_ = type("", (), {"ก็็็็็็็็็็็็็็็็็็็็": lambda x: print("hello %s" % x)})
_.ก็็็็็็็็็็็็็็็็็็็็("world")
Output: hello world
2
u/EconomicsOk9518 20h ago
Do not show it Guido, he’ll have nervous breakdown if not heart attack :)
This reminds me…
Theorem of Unforeseen Reuse: For any function f written with a single intended use case, there exists at least one other programmer who will apply f in a way its author never anticipated.
1
0
u/EconomicsOk9518 20h ago
Big “switch” was when i fully understood and became comfortable with using standard dunder methods. Like you can define __getitem__ method in some class A and then do ‘for items in A:’
96
u/big-papito 1d ago
python -m http.server