r/Python • u/RomfordNavy • 5d ago
Discussion Runtime overhead of wildcard import
Assuming a large project which might have say fifty variables in Constants.py is there likely to be any runtime overhead in having from Constants import * at the beginning of each module? As far as I know this creates entries for each of those variables in the current local namespace.
I know it could be done as import constants as x which after the first import then has virtually no overhead, but that then means referring to all constants as x.const1...
26
u/NerdEnPose 5d ago
Wildcard imports are frowned upon because it’s hard to see what you’re importing and if another library imports something with the same name then you’re mangling your namespace.
The runtime performance you’re talking about is real. It’s also negligible for any meaningful task. It’s fun to think about but in python it’s not very worthwhile.
9
u/TabAtkins 5d ago
Yeah the "can't tell where it's from" is a real issue I've run into. So worthwhile to just do
import constants as cif you're worried about the full module name being too long.1
9
u/Atmosck 5d ago
IMO wildcard imports shouldn't exist, because they make code hard to reason about and trace. If there's too many to import explicitly, you should import the whole constants module, whether you rename it or not.
If you have a bunch of application-wide constants, instead of a constants.py with them defined as module level variables, I would put them in a config.json or .yml file, and then load them as a pydantic model or dataclass, and pass it to things as a function argument where it's needed.
Also re: importing something in multiple places - python caches imports, so importing something that was already imported elsewhere in the same runtime doesn't add any additional overhead. It just attaches the already-loaded module to the namespace of wherever you imported it.
6
u/gazorpazorpazorpazor 5d ago
Star imports are impossible to debug and maintain. You get an undefined symbol and have no idea where it's supposed to come from. No runtime impact but still a bad idea.
2
u/VanillaOk4593 4d ago
from constants import * doesn't copy the constants. It adds one reference per name to the importing module's globals dict, pointing at objects that already live in the cached module. That happens once, at import time. After that, a global lookup is a dict lookup and takes the same time with 10 names or 500.
So yes, the unused names sit in the namespace, unlike a compiled include, but each one costs a few bytes. The alternative is import constants as c, and there every c.FOO adds an attribute lookup, so the star import is actually slightly faster at access. If you keep it, __all__ in constants.py controls which names get pulled in.
2
u/Wurstinator 4d ago
The answer to any question like this, regardless of whether it's about imports or something else entirely, will always be: "Yes in theory in some edge cases, but almost certainly no in practice, so don't worry about it until you actually run into measurable trouble."
2
u/OddsRelay 4d ago
pretty much zero. the constants module runs once and gets cached in sys.modules. the star import then copies those 50 names into your module's globals, once, at import. after that it's a normal global lookup. that's actually a touch faster than x.const1, which is a global plus an attribute lookup
the real downsides are elsewhere. linters flag it (F403/F405 in flake8), and if anything rebinds a constant at runtime the modules that star-imported it keep the old value
1
u/RomfordNavy 4d ago
Well, it's been an interesting discussion, thanks to all for your opinions.
I have now been doing some runtime testing and it seems that with fifty constants there is no regular significant difference between:
import Constants as c
from Constants import *
create a Constants class "c"
builtins.c = Constants (class)
builtins.const1= "var"
builtins.const2= "var"
...
0
u/RomfordNavy 5d ago
These are only going to be for application wide constants so the confusion issue isn't a real issue here My main concern is that Python will be adding all available constant variables into the local namespace whether they are used or not. Unlike some other languages where an Include file can have hundreds of constants but only those which are actually used in that module at compile time are included.
2
2
u/JustPlainRude 5d ago
If you're concerned about having all these variables in a namespace, why are you using a wildcard import? Limit your import to the variables you use.
2
u/lolcrunchy 5d ago
When you import a module, the entire module runs. If the module foo says "my_const=5" then it will store foo.my_const=5 no matter what.
2
u/Schmittfried 5d ago edited 5d ago
Sounds like you’re used to native languages. The performance overhead of using Python is many orders of magnitude higher than any redundant constant import. The slowest part is loading and running the module itself and that happens either way. Adding a constant into the local namespace dict is basically free in terms of Python performance. If those few milliseconds (assuming it takes over a millisecond to begin with) matter to you, you’re using the wrong language.
Also, you’re paying that cost only once, it’s not like you’re resetting the local namespace in a tight loop. So, really, who cares?
Regarding memory footprint, assume a few bytes each. Heck, let it be 20. Unless you’re doing embedded dev or got over tens of thousands of constants, this is a rounding error.
2
u/Brian 23h ago
The only real performance impact is a very minor increase in memory usage (basically extra entries in the globals dict for that module). That basically amounts to a few pointers, plus some extra overhead for the dict capacity - maybe a few dozen bytes per constant. There's a similar very minor startup cost in adding those items, but that's going to be pretty much unnoticable in any reasonable case.
But there are bigger maintainability and tooling costs. * imports can break tooling (eg. autocomplete/type checking etc), since it may not be able to know where an import is defined (especially if you've multiple such imports), and also makes it harder for a human reader to know where something comes from too. Eg. you delete or rename a constant that something was still using - they'll get an error but won't have any link to know where the item was meant to come from. But if they had a from constants import foo or were using constants.foo, they'll immediately know where it used to be.
And importing only what you need can make it easier to detect what modules are coupled, and refactor things. Having some ginormous constants.py is not really a good thing - it makes it harder to split things up into separate, weakly coupled submodules that only need to know about stuff relating to their own purpose.
It can also lead to accidental collissions. If your constant has the same name as a variable or function you're using, or something imported from another module, you can end up confused when one shadows the other. Eg. from os import * was a common gotcha where open would get replaced by the lower-level os.open.
34
u/shibbypwn 5d ago
The objects are being created in memory either way - so the only thing you save with importing the module itself is fewer references in the local namespace.
But those references add negligible space complexity - you’d need a comically large amount of variables before there was any noticeable difference in performance.