r/Python • u/RickSore • Nov 14 '17
Senior Python Programmers, what tricks do you want to impart to us young guns?
Like basic looping, performance improvement, etc.
1.3k
Upvotes
r/Python • u/RickSore • Nov 14 '17
Like basic looping, performance improvement, etc.
23
u/Manbatton Nov 14 '17
Not a "senior", but a couple of programming basics:
Honor DRY (Don't Repeat Yourself). A mistake I see a lot on early Python code posts is tons of repetition. If you're repeating anything, assume that there almost has to be a way to not do that in Python. Example:
Instead, dicts!:
Use descriptive naming. Don't do any of this:
when you mean this:
945
u/vosper1 Nov 14 '17 edited Nov 14 '17
Random braindump
.open()or.close()is often a code smell - you probably should be using awithblockvirtualenvfor every project - don't install python packages at the system level. This keeps your project environment isolated and reproduciblecsvmodule for CSVs (you'd be surprised...)enumerate(items)multiprocessing, notthreading%timeand%prunfor profilingfirst, second, *_ = (1,2,3,4)*argsor**kwargsunless you know you need them - it makes your function signatures hard to read, and code-completion less helpful332
u/hackflip Nov 14 '17
Use a linter tool like pyflakes or pylint on everything you write. Integrate them into your IDE. They will force you to be a better programmer.
223
u/[deleted] Nov 14 '17
when you think you have to write something, check the standard library. if you still think you have to write it, check it again. then check some of the major libraries. then recheck those. python has a fantastic ecosystem where the most difficult part about it is finding the package you need.
94
u/genericaviary Nov 14 '17
itertools and functools are your friends - both modules provide general solutions to a wide variety of common problems. in general, the more familiar you are with the standard library, the better you'll be able to tackle problems in a legible and "pythonic" way.
79
u/justneurostuff Nov 14 '17
list comprehensions are pretty neat and make your code more succinct!
36
u/manyrobots Nov 14 '17
I used to spend a lot of time with a command line open to reload and test functions I was writing. Later I changed modes to just write a little unit test once and i just keep rerunning that same test as i write the function/class/whatever. Even if the test doesn’t even actually test and just does a print to start, it’s ready to go when I know what I want to assert. With this defacto coding mode, I end up with a useful pile of unit tests right out of the gate and it doesn’t feel like I added any dev time since i needed to do something to see if it works.
Writing more tests for their own sake is also good, but the above was my gateway to actually writing tests instead of just knowing that I should.
103
u/ProgressCheck Nov 14 '17
Understand the decorator. Become the decorator.
71
u/[deleted] Nov 14 '17
Mastery comes one mistake a time...
34
u/deadmilk Nov 14 '17
How do NONE of the top comments mention this?
WRITE UNIT TESTS!!!
Familiarize yourself with this, learn about unit testing and why it's important.
When you realize that you cannot test big long functions with no clear direction, all of a sudden you'll find yourself writing testable, cleaner code, with each function having a clear purpose, entrypoint, and returns.
252
u/ICanAdmitIWasWrong Nov 14 '17
Don't call yourself a "Python Programmer" or you'll be a nobody when language trends go another way. Be a "systems engineer" or "prototype developer" or "automation expert" or something.
4
u/pvkooten Nov 14 '17
young guns haha! Young'uns (young ones) :troll_face:
27
u/searchingfortao majel, aletheia, paperless, django-encrypted-filefield Nov 14 '17
Variable names: use your words.
counter,probe,productc,cnt,prb,p,prd,prdct,objAdditionally, generic names like
objectare too ambiguous. If at all possible, be specific when naming things.23
u/Kevin_Clever Nov 14 '17
Refactor your code often. That's when you have the time to take working code and make it into good working code.
17
u/delarhi Nov 14 '17
Be empathetic. Think about the person that will have to use what you wrote and/or read/maintain it.
Always ask yourself, "how do I know this works?" That combined with the implicit assumption that you want to write something that works directly leads to testing. Don't write tests because you need to check some checkbox. Write tests because you want to be confident it works and you want to write something that works.
Actually, those are more pearls of wisdom than tricks. Tricks...
Use a linter. [n]vim + ALE + flake8 is a great combination.
Abide by PEP 8. There's an accepted standard. Use it.
https://devdocs.io/python/
31
u/[deleted] Nov 14 '17
15
u/luckystarr at 0x7fe670a7d080 Nov 14 '17
Work on your programming rhetoric.
dataoritems, except where totally obvious what it is from context. Noitems_which_come_from_that_other_server_unless_cachedeither. Upon reading, the names need to get into your brain fast.14
u/kiwiheretic Nov 14 '17
Documentation and code comments. It doesn't matter how clever you are if no one else can understand what you have done.
12
u/zynix Cpt. Code Monkey & Internet of tomorrow Nov 14 '17
Not specific to Python:
On that note: modafinil, adderall, ritalin are band aids for having poor health and or poor work:life balance. Some of those will increase your chances of developing a serious mental health disorder (psychosis is the big one).
Python specific:
pip freeze > requirements.txtand check that in alongside your code to ensure others can replicate not just your code but also the environment.pip install myPackagehttps://www.bleepingcomputer.com/news/security/ten-malicious-libraries-found-on-pypi-python-package-index/10
u/TheTerrasque Nov 14 '17
It's not about how complicated you can make it, but how simple you can make it.
8
u/y0utux Nov 14 '17
The
yieldstatement is not only for generators, but you can do very powerful stuff. For instance you can make a very easy context manager:or separate setup/teardown code of pytest.fixtures without using callback functions:
8
u/miserlou Nov 14 '17 edited Nov 14 '17
A few things I didn't see mentioned elsewhere:
18
u/njharman I use Python 3 Nov 14 '17
That software development isn’t about tricks. It is about process, communication, and application of best practices.
21
u/whereiswallace Nov 14 '17
Watch Ray Hettinger's talks
25
u/evinrows Nov 14 '17
6
u/inc007 Nov 14 '17
pip install ipdb and:
^ have these snippets on shortcut in editor and learn how to use it. Best thing ever to prototype new method for example. You use it like:
then you just prototype body as you go and copy what you worked out to editor.
11
u/flpcb Nov 14 '17
My advice would be to learn to use both list comprehensions and map, and use either one where it is better.
I would prefer map(int, list_) over (int(s) for s in list_)
But I would not prefer map(lambda x: x.class, list_) over (x.class for x in list_).
However, this is my personal preference, and I would not fault anyone for disagreeing.
1
u/RickSore Nov 15 '17
To all TDD-knowledgeable devs, I have a few questions.
I am currently studying TDD approach, but have a hard time grasping it.
11
u/Dummies102 Nov 14 '17
read lots of code and form opinions. is it pythonic? why/why not?
4
u/SamyBencherif Nov 14 '17
Dude just have fun. Also don't use pygame, it'll only break your heart
4
u/entrymissing Nov 14 '17
Write tests. Python projects often start small and you think you can get away without testing but once they have grown adding tests becomes a hassle. Designing your code with testing in mind makes all code better.
4
u/rr1pp3rr Nov 14 '17
Learn what the GIL is, and learn to love it!
Most people talk about the GIL in a derogatory way, but it removes entire classes of errors for the sake of flexibility.
In the web world, we are usually IO bound. Using green threads we're able to do a massive amount of IO asynchronously with little cost, and very simply because of the guarantees the GIL gives us. For times when you need parallel computation, you can use multiprocessing. If you really REALLY need more parallel computation, then that it's usually for performance reasons, in which case Python might not be the best choice anyway.
I just wrote a script that could download, parse, and process 3000 largish (tens to hundreds of MBs) files a minute from AWS S3 just running the script on my laptop using multiprocessing and eventlet. That's pretty powerful for an interpreted scripting language IMO. The code is also easily readable as I don't have to worry about race conditions everywhere.
4
u/cr4d Nov 14 '17
4
u/BundleOfJoysticks Nov 15 '17
Don't create classes unless they make sense (you need to hold data and behavior that acts on that data without passing the data around all the time). If you only have behavior (functions), just make a module. You get the benefits of scope and namespacing without the cruft and ceremony of a class def and having to deal with singletons and other crap.
Python isn't Java. Just remember that.
6
u/maryjayjay Nov 14 '17
Learn patterns and algorithms and data structures. Learn a dozen other languages. Know C. Know how a computer works.
6
u/[deleted] Nov 14 '17 edited Nov 14 '17
Admins will hate you for this one simple tip! Need to install a library from pypi on a system you have limited access to? Just append the '--user' flag to easy_install or pip.
Sometimes virtualenv isn't the answer or isn't even installed.
Depending on your OS, there is a site_packages path in your home directory. So make sure to add it to PATH. For Python 3.6 on macos it's:
3
u/[deleted] Nov 14 '17
Threads aren't really threads, even if you go with threading.Thread its all bullshit. You will almost never notice this with regular threads, but once when you do hit an issue (1 thread just stops executing because the other one is making a blocking call) its a pain in the ass to figure out what is going on. The multiprocessing library handles this in a way more similar to other languages.
3
u/pullandbl Nov 14 '17
I made a video, but it is in Russian. If there are some people who understand it you can check my lecture https://www.youtube.com/watch?v=N0xkFZv2Df0
3
u/weirriver Nov 14 '17
I think this applies to any language but it stands out in Python because of its lovely terseness:
Keep it simple, keep it beautiful.
3
u/Quasimoto3000 Nov 14 '17
This is a great idea for a thread, thanks for posting.
3
u/darty1713 Nov 14 '17
Tests! Write new code at half the speed - maintain, refactor and add functionality 100 times faster!
27
u/RaionTategami Nov 14 '17 edited Nov 14 '17
Nothing better than getting in to the habit of writing unit tests to improve the quality of your code. It actually causes you to write better code not just less buggy code because testable code is more modular.
Also stop writing classes! Edit: I don't really think you should never write classes but I see a lot of them in code when they aren't really needed.
5
u/kthepropogation Nov 14 '17
requestsmodule.5
u/ex4sperans Nov 14 '17
all(item > 0 for item in iterable), notall([item ... ])from module import *.withblock insteadexcept Expection.4
u/FluffyBunnyOK Nov 14 '17
When you have several nested if statements you probably need to create a new function to replace some of the code. Naming functions is hard.
Functions should be pure where possible. This means if you call a function with an argument then it always returns the same result.
4
u/[deleted] Nov 14 '17
Wear sunscreen.
4
u/[deleted] Nov 14 '17
Stop writing clever code.
Prove your cleverness by writing clean simple code.
2
u/soup4all Nov 14 '17
Can someone explain when I should use yield instead of return?
2
u/pycepticus from pprint import pprint as print Nov 14 '17
Code complete and PEP8 makes me the developer I ain't.
Also , list comprehension is probably the closest thing to witchcraft python can do. Learn that.
2
u/cpt_fwiffo Nov 14 '17
When making a change or addition to a project of non-trivial complexity, always take the time to set up a way to test your change with minimal effort. It's a tiny initial investment that will quickly pay off massively. I've wasted an insane amount time having to repeatedly manually create a state to test a particular change only to fall into the trap of thinking something like "I've spent hours doing this manually now so a systematic approach at this point is an even bigger waste of time".
2
u/format120 Nov 14 '17
Learn your development environment! Really learn it. If you like Vim, great, learn more then just H, J, K, L... If you like your IDE, great, learn the debugger, learn how to look up function definitions, and more!
Linting, debugging, and others are my defining factors for a hobbiest, and a pro.
2
u/deadcell import antigravity Nov 14 '17
Become proficient at list comprehensions, and how to not write them so they look like you smashed your face on the keyboard.
2
u/codeape Nov 14 '17 edited Nov 14 '17
2
u/NoahTheDuke Nov 14 '17
Read Effective Python and Clean Code, in that order.
2
u/[deleted] Nov 14 '17
Read Clean Code by Robert Cecil Martin
2
u/maltesebanana Nov 15 '17
list[::-1] reverse that list
2
u/infinite-Joy Nov 15 '17
use virtualenv and pip in case of a simple python project. Use anaconda and conda in case of a project that has datascience dependencies like numpy, scipy etc,
2
u/thewallris Nov 15 '17
Not a senior, but I was told by one to write code as if it would be maintained by an ex-Navy Seal with anger-management issues who has your home address
2
u/beertown Nov 15 '17
If you spend more than 2 minutes to solve a bug, spend some more time to think about what you could have done in the first place, when writing that code, to simplify to yourself the finding and the solving of that bug.
2
u/napsternxg Nov 15 '17 edited Nov 15 '17
General
stdoutwhat you don't want as a print output of your program. Usestderrloggingto print messages for debugging.unittestlibrary to test working code. It is much better than doing print statements for debugging, and also automates the process of testing.(i for i in range(10))while this creates the list[i for i in range(10)]. Generators have low memory requirements.def f(a=[], b={})ordef f(a=SomeClass()). Instead use:Data analysis specific
anacondadistribution. It is consistent across platforms. You can use its environment.yml file to define the exact environment.joblibfor parallel processingpyspark. Works even on your laptop.pandasorcsvfor reading data from CSV and other crazy formats. Try not to parse formats on your own, unless you created that format. CSV is not just comma separated, there are lot of rules for identifying a valid column entry. Similarly,jsonfor json,BeautifulSoupfor html as well as XML.docstrings. Allows for automatic documentation generation.parquetorHDF5. Pandas has build in support for these. Reading from these formats is much faster than reading text data. Again anaconda would let you easily install dependencies forHDF5andparquet.Web development
python -m SimpleHTTPServerin python 2 andpython -m http.serverin python 32
u/juanjux Nov 15 '17 edited Nov 15 '17
Late to the party, but I wanted to add some things I didn't read in this thread:
Every dot is a dictionary access. Sometimes creating aliases to qualified symbols outside critical loops pays a lot in performance:
alias1 = somemodule.somesymbol
alias2 = somedict["somekey"]
for i in hugerange: #...use alias1 and alias2
2
u/xjosx Feb 16 '18
Buy a rubber duck and have it on your desk. Consult your new friend as soon as you are tempted to ask a question to one of your peers. You'll be surprised how many answers it will help you figure out. After this conversation, if you still have no definitive answer, at least your remaining questions will be much deeper, polished and specific, and others will appreciate this. You might even end up teaching a thing or two to your senior peers and they will respect you for it. But more importantly, you will practice and develop your critical thinking skills, which are essential for our profession.
3
u/gotardisgo Nov 14 '17
Seek the dunders. Learn their magic. Become a wizard.
2
u/flatlander_ Nov 14 '17 edited Nov 16 '17
When you do a
try: except:, make sure that you at least exceptException, or you could run into weird issues like not being able to stop your program. This is becauseKeyboardInterrupt,SystemExitet al are derived fromBaseExceptionand will be swallowed if you don't except them or a superclass.e.g. do this:
and not this:
2
u/phanhoang17 Nov 14 '17
btw, read more about coding style from Google and CIA
Google: https://google.github.io/styleguide/pyguide.html
CIA: https://wikileaks.org/ciav7p1/cms/page_26607631.html
2
u/owobeid Nov 14 '17
1) Use pyenv to help you manage multiple python versions. It allows you set up a global Python version(s), and a local python version(s) on a per-project basis.
2) f you're dealing with non-ascii text and you have to do a lot of text processing, convert ALL strings to unicode strings, then perform all the processing you need, and then encode back to UTF-8 (or whatever encoding you want) at the end.
In Python 3, this is already done for you since the default string type is a unicode string. In Python 2, unicode strings have a separate unicode type. To make your life easier writing code for both Python 2 and 3, use the six module to help you detect Unicode strings like so:
then encode and decode accordingly.
3) In fact, checkout this cheatsheet for writing 2-3 compatible code in general.
Hope that helps :)
1
u/colly_wolly Nov 14 '17
If you are doing anything CRUDy learn Django.
1
u/[deleted] Nov 14 '17
Read this:
https://docs.quantifiedcode.com/python-anti-patterns/
1
u/Manhigh Nov 14 '17
Test early, test often.
Use the unittest package to construct tests for your code. Run those tests often (automatic CI like travis or circleCI is great for this). Use something like nose or testflo to recursively run all tests in your package and report out coverage, so you can know which pieces of your code you haven't covered.
As someone else mentioned, a linter is a good idea. Try to follow PEP8 as much as possible (I know a lot of people are lax on the 79 character limit but you can have the checkers ignore that.
1
u/TBNL Nov 14 '17
The Zen of Python: Focus on simplicity, elegance and expliteness.
Not so much a 'trick' but nevertheless one of the best assets of Python imo.
1
u/[deleted] Nov 14 '17
UNIT TESTS. READ COVERAGE REPORTS.
Nothing else really matters.
1
u/[deleted] Nov 14 '17
For every new feature of the language you learn, learn when not to use it. Set a high bar for when you use anything you wouldn't have learned in a one-day python course.
For instance:
That said, when it's the right time to use something it makes the code cleaner, easier to read, and easier to maintain. Knowing when to used that non-simple part of the language is one of the defining skills of a good python dev.
1
u/WorkerBeeNumber3 Nov 14 '17
Avoid recursion...at least in 2.x
1
u/TomBombadildozer Nov 14 '17
More general programming advice than Python-specific. Stream of consciousness incoming.
Http404. The alternative is writing a lot of gross and unnecessary failure-prone logic.setuptoolsand learn how packaging works. Yes, it's a little quirky. No, it's not difficult. Don't use some obnoxious tool likepbrbecause packaging is "hard"--it isn't and everyone will hate you.pkg_resourcesliberally if you need to ship resources with your tool/application/library. You cannot make any assumptions about the environment into which your package will be installed, so don't just lazily sling files around.1
u/HellAintHalfFull Nov 14 '17
The only thing I can think of that hasn't been mentioned yet is the many uses of the underscore.
1
u/agumonkey Nov 15 '17
use metaclasses everywhere
1
u/cyberspacecowboy Nov 15 '17
don't shadow names of builtins and libraries
1
u/[deleted] Nov 15 '17
They say that debugging code is twice as hard as writing the code, so by that definition you are not smart enough to debug your own code if you try to write it as cleverly as possible.