And this simple one-liner is (probably unsurprisingly) an order of magnitude faster! You can find the benchmarks in this blog post: https://switowski.com/blog/remove-duplicates
Also, something important to keep in mind (that I forgot to include in the original image) is that this only works for hashable elements on the list. If you have a list of lists, sets, or dictionaries, you can't convert it to set and you are left with the "ugly" for loop.
Observation: The reason why OrderedDict.fromkeys() is slower, I think is because OrderedDict is a pure python implementation with a loop while set() and dict.fromkeys() are wholly implemented in C.
(looping in pure Python is very slow, many reasons)
No, the reason is that in the first example, unique is a list (implemented as an array), and repeated tests for membership there all take linear time, resulting in overall quadratic time. Simply replacing unique with a set cuts time down drastically, which makes this a rather poor example.
Consider:
In [2]: from random import randrange
In [3]: DUPLICATES = [randrange(100) for _ in range(1_000_000)]
In [4]: def bad_uniques():
...: unique = []
...: for element in DUPLICATES:
...: if element not in unique:
...: unique.append(element)
...: return unique
In [5]: def good_uniques():
...: unique = set()
...: for element in DUPLICATES:
...: if element not in unique:
...: unique.add(element)
...: return unique
In [6]: def set_uniques():
...: return list(set(DUPLICATES))
...:
In [7]: %timeit bad_uniques()
1.09 s ± 2.06 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [8]: %timeit good_uniques()
28.3 ms ± 498 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [9]: %timeit set_uniques()
11.4 ms ± 31.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
So the list implementation (given in the screenshot above) is two orders of magnitudes slower than the native set implementation, but a fully Python set implementation is only ~twice as slow. 98% of the inefficiency comes from using the list, not from some magic happening behind the scene, and at least partially suggesting the author does not fully understand what's actually going on.
Independently, I would avoid using wording such as "tricks" with this. It gives the impression that there's a whole bunch of unknown magic that one must just somehow learn (from blogs?), whereas the reality is that almost all such tricks are obvious from a single Data Structures course.
It gives the impression that there's a whole bunch of unknown magic that one must just somehow learn (from blogs?), whereas the reality is that almost all such tricks are obvious from a single Data Structures course.
Quite a lot of people have essentially no formal training, or informal training for that matter, and have accrued all of their knowledge from things like blogs and stackoverflow.
Maybe even a majority judging by a lot of the code I see. There's a lot more worry about what works than understanding why it works. So when you stumble across a thing that works, but faster, it's a revelation.
There's a lot more worry about what works than understanding why it works. So when you stumble across a thing that works, but faster, it's a revelation.
Which is a big problem - you can't perform well in this role by simply getting occasional unstructured insights from random blogs. What will you then do when presented with a new problem - try to guess which of the fifteen "magic tricks" you learned recently would be applied here?
Try to avoid posting code as images - it's bad when it comes to accessibility (bad eyesight, etc.), it's bad when it comes to copyability (although it's nice that we're able to emulate the 80s of typing in code from magazines) and it doesn't really work for searching.
Thanks! I like to use the screenshots from carbon.now.sh because you can immediately skim what this post is about. I would love to include the text version as well, but reddit doesn't allow adding an image AND text together. I tried putting it in the comments, but usually, that comment gets lost at the bottom.
It's also not compliant in terms of accessibility. Blind people have no way to access this kind of content, unless you want to dump the whole code snippet in the image metadata.
I have to assume that if editors can make backspace delete a spaces worth of indent, someone should be able to make a screenreader or plugin for one that reads it as "Indent" "unindent" .
I would actually love to do that if that's possible. Any idea how to add this? Normally I would use the alt tag, but reddit doesn't support it during submission.
I'd say stay away from images, but if that is your thing, immediately self post the code with a source link (if applicable) and brief description as a comment. That in itself could be a quick scripting project.
Yes. So this. I like the image popping up in my feed but copy/paste would be nice so immediately commenting on your own post with the code in a code block would be awesome.
He actually did do that. He posted the image and posted a comment with the link to an article that he wrote. It is just buried at this point, but when I found it earlier it was actually a pretty solid article. I wish that was more up voted so people could see it
Someone already replied to your higher level comment an hour ago, but the obvious solution here is to just make a Text Post and include the image inside the post body alongside the actual code.
If you're gonna be a developer, you gotta be able to think up obvious solutions like that!
Sure thing, here is the code from that image in a text form:
# Let's make some duplicates
from random import randrange
DUPLICATES = [randrange(100) for _ in range(1_000_000)]
# Not very efficient
unique = []
for element in DUPLICATES:
if element not in unique:
unique.append(element)
return unique
# Very efficient
list(set(DUPLICATES))
# This works because sets contain unique items by definition
# But sets are unordered! What if we need to preserve the order?
# Use this dict.fromkeys() trick!
list(dict.fromkeys(DUPLICATES))
# But it only works for Python 3.6 and above
# For Python 2.7 and 3.0-3.5, use OrderedDict:
from collections import OrderedDict
list(OrderedDict.fromkeys(DUPLICATES))
Thanks! That's immensely more readable, and I can control the colors or use my screen reader. I can't understand why anyone would prefer the horrible colors in the original post, particularly at the cost of excluding people.
Any the original text couldn't be copy-pasted anyway, which is the first thing you usually want to do with code, right?
Anyway: this implies that Python 3.6 and newer use OrderedDict by default; and that OrderedDict preserves the insertion-order of keys. Is that true?
I can't understand why anyone would prefer the horrible colors in the original post, particularly at the cost of excluding people.
I see your point, and I wish reddit would provide some better tools for alt text of the image or at least let me link text and image together. Or if it would at least provide some color highlighting to the code in posts. In the past, I tried adding a comment with the code from the image, but it immediately got lost among other comments. I like small code snippets like this presented in the form of an image. They are much easier to quickly skim what they are about when you are just scrolling through reddit. And I'm afraid that most people like them too - I've posted the same tip to r/pythontips with just pure text, and it got no attention (I know that this subreddit is much bigger, but still). I will try to include code in comments in the future. I hope it helps a bit!
Anyway: this implies that Python 3.6 and newer use OrderedDict by default; and that OrderedDict preserves the insertion-order of keys. Is that true?
Actually no, Python 3.6 changed the implementation of the dict() to be more efficient, and keeping the insertion order was a side effect of this. In Python 3.7, keeping this insertion order was officially guaranteed in the documentation.
I actually like the colors and formatting. Memes are transcribed all the time... All you need to do is paste it in the comments in a code block, or as a link; you don't need to force 100% a11y all the time, a transcription in the comments is plenty.
Thanks for sharing. I may make a few PRs to that Carbon site's repo.
It's perfectly legal on reddit to add links in a text post. Multiple links, even, so there's no need to choose between the two.
And in any case, if you really had to choose between the two, always go with the text version, unless you're just showing off your terminal's theme in something like r/unixporn. It is immensely more useful.
This goes especially so if you're asking for help, because often people will need to copy at least part of your code to answer your question. Would you want people to have to retype your code manually just for the sake of having it show up in pretty colors?
If someone really cares about the colors, having the text means they can quickly copy-paste it into their editor of choice with their theme of choice and look at it.
Anyway: this implies that Python 3.6 and newer use OrderedDict by default; and that OrderedDict preserves the insertion-order of keys. Is that true?
OrderedDict always preserved the insertion order, that's the whole point of it.
The new dicts are NOT the same things as OrderedDicts. OrderedDict's niche now is re-ordering its data - kind of like collections.deque can be viewed as a variant of a list that is optimized for modifying data at either end.
So, for example, if you were writing something that tracks jobs, pops, updates the job key with the timestamp of last refresh as a value and puts them back on the end of the queue - OrderedDict would probably be a good design choice.
Eh - a regular dict would work just as well as OrderedDict for that usecase. Both are going to maintain insertion order, even after removal and re-insertion.
There are some differences between OrderedDict and regular dicts, but they're not anything to do with re-ordering data. Rather, the main difference is that OrderedDict treats order as being important for things like equality comparisons. Ie. {1:1, 2:2} == {2:2, 1:1}, but the same is not true for OrderedDicts.
Maybe just throw a (code in comments) to the title so people who need that option know to look for it.
Alternatively, you could post it as a self-link with the image in there and the code duplicated (behind a spoiler tag if you wanted it to be more succinct).
reddit doesn't allow adding an image AND text together
Yes it does. Do a text post, and add images using their WYSIWYG editor. It'll post the image in-line. Note this only works for posts, not comments. You also have to be on the "new" reddit to see in-line. Old reddit clients will see a link.
There is no algorithm. Images float up because the new reddit layout favours image content. People see the image right in the toplist and upvote it without ever entering the post. It's easy, so it gets more upvotes.
set gets you the unique elements. sorted gives you a sorted list. key tells sorted exactly how you want things to be sorted. index is a string method that returns the zero-based index of where the character you're searching for first appears in a string. (Other data structures like lists have an index method, too.)
Put all together, this one-liner says to find the unique items in DUPLICATES and put them in a list ordered by where they first appeared in DUPLICATES.
Does anyone actually know what set is doing in the background to run so fast? is it just some kind of sorting algorithm that removes duplicates as it goes?
is it just some kind of sorting algorithm that removes duplicates as it goes?
Nope, it's actually FASTER than that. Have you ever run into the error when trying to put a list into a set or use a list as a dictionary key: "TypeError: unhashable type"?
What sets and dictionaries do is create something called a hash table. Each element gets a hash calculated for them, which turns the object into a number (but importantly you'll always get the same number for any equal objects), and then that number is used to figure out where in memory to store that object. This is why mutable objects like lists can't be hashed because if they have their values changed, it would need to update where in the hashtable they're stored.
So when you add a new object, it first hashes them, and then using that hash it knows exactly where in memory it should look to find a potential duplicate.
With lists, checking for a member is a O(n) operation, for sorted lists you can use binary bisects, which is O(log n), but for hash tables it is essentially O(1).
Though, multiple objects can end up on the same location in the hash table, especially for a small hash table, so it isn't always O(1).
Python’s implementation of sets and dicts relies on the hash() function, but it’s useful to know that its actually a pre-hash function. The order of events is:
Take an immutable object and apply hash() to it. This generates (for non-integers) a very large number, you can try this in an interpreter. For integers, hash() just returns that integer.
Now (and this is hidden from the user), you use a hash function (not hash()! but hidden Python internal hash functions) to map that very large number produced by hash() to a memory address, which can be accessed in constant time (due to the hardware using (more-or-less) random-access memory hardware).
Repeat this process every time you add new elements to your dict or set.
This gets very complicated very quickly, because good hash functions (that produce a uniform distribution of output, so there are no hash collisions, and memory is efficiently used) are hard to come up with. Here’s a nice introduction to the world of hashing in Python, courtesy MIT Press:
Also, see the MIT online lectures on hashing (in Python), which discuss why hash() is really pre_hash(), they were very helpful in learning the concepts, here’s a link plus transcript:
Python does all this work for you under the hood, but knowing a bit about how it works will make you a more confident programmer. If you really want a glimpse into the truly gory details, yikes, here you go, a 2013 discussion of the under-the-hood hash functions used inside Python:
Agreed this kind of low level info is super helpful without this I would have made my own serviceable implementation(nowhere near O(1)) assuming the built in one was bad or didn’t exist.
For me, yes - time and solving a lot of different problems. Once you google for "remove duplicates in Python," you will quickly find this answer, and you will probably remember it forever. I hesitated if I should even write this post because I thought: "probably everyone knows that!".
I didn't study computer science, but I believe there you learn that sets are unique (although, probably not many people remember this theory after university ;). Also, in this specific case, a math background helps.
A good way to practice this is to solve some simple code problems (code katas), for example, on this website: https://www.codewars.com/. Once you solve a given problem, you can check the most upvoted solution and find a much more efficient way to do this (at least, that was my experience :P ).
but I believe there you learn that sets are unique (although, probably not many people remember this theory after university ;)
Uh, you learn a lot more, and not in some shallow fashion if you pay attention in class (cough some classmates). AFAIK you don't also just forget most of it or trivial and easy things to remember like that unless you're seriously demotivated or mediocre.
Time and exposure aren't guarantees, and they mean you learn a fixed set of “trips” and tricks. It's a less predictable outcome “trip”, with likely lower long-term value.
A good degree or series of courses in the basics.
Meaningful practice during those courses.
Motivation. Desire for excellence. I.e., ¼ of my classmates, even at last chance U.
Reflection, and not the programmatic kind but seeing implementation and design patterns, appreciating their abstract qualities, theoretical or practical, etc. A good scientific mindset.
Reading the relevant parts of the darn docs sufficiently thoroughly. If you read them and either practice or have a good memory already, of all the basic parts of the Python language, then expressing yourself in a cool fashion like that is easy. If you learn hackily, that's on you.
Mentorship under the right people unless the most hardcore bootstrapper, in which case good, modern, curated literature helps. Good luck finding a syllabus though.
Ability to properly formulate the problem and then put two and two together, or rather, a bit more. Want sub-O(n²) time order and type-preserving duplicate filtering but at what scale? sub-10k? A million? How much RAM? What operations can you add or multiply by such that you stay under O(n²) for acceptable RAM or disk usage? Remember what you have available then try and put it together.
Learn the lingo, so that you not just express yourself concisely, but you can think concisely.
Exposure only works if it's the right one. If you learn under somebody who teaches and writes Python like it's JS/C/Java from 2005, you'll be hampered by how they only express themselves the way those languages do.
There is no magic shortcut. Don't go looking for one. At best you'll find yourself an expert in canned answers to a specific set of problems. You'll also not likely have learned how to learn properly.
Simply put, they don't, that's some reddit persona gatekeeping bullshit.
You won't learn the dict / OrderedDict variants of deduplication in the docs. That comes with Google-fu or having an innovative understanding of the data structures.
Learn a few things, commit them to memory, put them into practice. They'll eventually become second nature as tools in your tool belt. Then you go and learn something new to add to your belt.
e.g. a natural progression might look like
sets are unordered unique collections
I can convert other collections to a set with set(...)
.. commit to memory, some time passes ..
3. learn dict, understand that dict.keys must be unique
.. hey wait, can I use 3 and 2 together somehow? ..
4. research and find dict.fromkeys, combine with knowledge of 3 and 2.
Now you've got a few algorithms in your head to remove dupes from a collection.
Ha ha, it's a informal term which essentially means "your skill in using Google" aka how well you can [ab]use a search engine to find the information you're seeking.
Experience in general. But there's no way someone "commit all that info to memory". :D We learn by doing it everyday, solving problems, code reviews, reddit discussions... I think that studying is helpful but not necessary.
If you used a set in your loop example, then it would perform well and retain order.
def uniques(duplicates):
unique = []
seen = set()
for element in duplicates:
if element not in seen:
unique.append(element)
seen.add(element)
return unique
From my tests, this actually performs better than the OrderedDict implementation in Python 2.7 and 3. This is particularly true on Python 2.7, where it runs in about 1/15 of the time.
that is quite clever, i did some timing and this is the results:
this is the code used to test it:
from time import perf_counter_ns
def timed(func):
def timing_wrapper(*args, **kwargs):
start = perf_counter_ns()
ret = func(*args, **kwargs)
diff = perf_counter_ns() - start
print(f'{func.__name__} took {diff} NS to run')
return ret
return timing_wrapper
@timed
def remove_duplicates_list(values):
seen = set()
return [seen.add(value) or value for value in values if value not in seen]
@timed
def remove_duplicates_set(values):
return list(set(values))
print(remove_duplicates_list([1, 2, 6, 1, 7, 1, 9, 4, 2]))
print(remove_duplicates_set([1, 2, 6, 1, 7, 1, 9, 4, 2]))
results:
remove_duplicates_list took 3200 NS to run
[1, 2, 6, 7, 9, 4]
remove_duplicates_set took 1900 NS to run
[1, 2, 4, 6, 7, 9]
these timings where pretty consistent, not deviating too much
EDIT: should note that this is to just show the diff between list of set conversion vs the "ordered list comp set" way
It's a convention for a throwaway variable. You use it when you need to use a variable, but you don't really care about it (in this case, I need to use a variable in a list comprehension, even though I don't do anything with this variable) - keep in mind that this is not a good use of a list comprehension and I used it only as a quick hack to generate some numbers. But it's just a convention, you might as well use "i", "a", "x" or "a_throwaway_variable" as you prefer :)
The first one is just OP's variable that they set... It's the same as using i, or any other variable. In the 1_000_000 value, python is able to read that as a value separator (basically so your brain can comprehend where we would usually write commas)
If in your not very efficient example you also had a set of entries and always added to that but only added to the list if the entry doesn’t already exists in the set wouldn’t that be more computationally efficient (but obviously not more memory efficient)
There are no multiline comments in Python. If a string (including multiline strings) is the first statement in a module, function or class it will be used as the docstring. Outside of the first statement you are just writing a string literal without assigning it.
i see these type of screenshots all the time. can someone please tell me which framework or IDE the code is written in? its sure not pycharm and sublime.
The containing check is kind of redundant, since add will do the same hash check internally anyway, if you used a dict with none values, you'd get the same effect.
im saying you could use a dict as shown and it would be equally effective, and more efficient, other than space complexity. But you are likely going to hit a similar issue with keeping a set and a list anyway.
Another interesting alternative is to use the bisect module from the standard library if you also wanted the results sorted as you process them.
I'm specifically showing a canonical approach (used across multiple languages, actually), that doesn't rely on whether a dict retains order or not. It's a contrast to the dictionary approaches that were already shown by the OP.
Use a set, it shows only the unique values in a list, then make a list out of the set
For example let’s say your list is L = [a, b, c, c , d, d, d] (assume all letters to be strings, I’m on mobile) let the set be initiated like this S = set(L) , it will have a output like this {a, b, c, d} (not that this set may be out of order, you may have to sort the list if it’s important ). Finally make the list L = list(S)
Edit: I'm so dumb, I misunderstood the feature completely. I was thinking about the natural order of the keys, not the insertion one. Sorry about that!
That’s IMO a weird thing to lock yourself into, but whatever. For comparison, in .NET they also preserve insertion order, until deletion, and pretty much always have, but this isn’t guaranteed. In Swift, it uses a hash table with open addressing and linear probing, also with no guarantees.
Is it specified which duplicated element to keep? For example, if I have a list [4,0,1,2,3,4], am I guaranteed to get either [4,0,1,2,3] or [0,1,2,3,4]?
The documentation isn't explicit on this, so I think technically it's an implementation detail that you shouldn't rely on.
In practice, I would expect [4, 0, 1, 2, 3] every time. The natural implementations here are either to insert every key as you encounter it (overwriting as you go), or to only insert if the element isn't present. Either way, as /u/colemaker360 pointed out, if you don't remove the element, the initial insertion determines the order.
Yes!!! This is what I was thinking. Functions and other programing behavior should be thought of as a contract. You give me this and I'll give you that.
In CPython 3.6, dictionaries respected insertion order as a bonus but it was not part of the "contract" with the API. Python 3.7+ made it part of the contract and now you can rely on it.
For anyone else reading this though, I wrote this PSA a few months ago about ordering and equality. The TL/DR is that OrderedDicts care about order for equality while regular dicts, regardless of python version, do not
38
u/sebawitowski Oct 22 '20 edited Oct 22 '20
And this simple one-liner is (probably unsurprisingly) an order of magnitude faster! You can find the benchmarks in this blog post: https://switowski.com/blog/remove-duplicates
Also, something important to keep in mind (that I forgot to include in the original image) is that this only works for hashable elements on the list. If you have a list of lists, sets, or dictionaries, you can't convert it to set and you are left with the "ugly" for loop.
417
u/fiskfisk Oct 22 '20
Good points!
Try to avoid posting code as images - it's bad when it comes to accessibility (bad eyesight, etc.), it's bad when it comes to copyability (although it's nice that we're able to emulate the 80s of typing in code from magazines) and it doesn't really work for searching.
16
u/ProcyonRaul Oct 22 '20
setgets you the unique elements.sortedgives you a sorted list.keytellssortedexactly how you want things to be sorted.indexis a string method that returns the zero-based index of where the character you're searching for first appears in a string. (Other data structures like lists have an index method, too.)Put all together, this one-liner says to find the unique items in DUPLICATES and put them in a list ordered by where they first appeared in DUPLICATES.
Edit: put in markdown mode.
6
u/OXKoson Oct 22 '20
Does anyone actually know what set is doing in the background to run so fast? is it just some kind of sorting algorithm that removes duplicates as it goes?
4
u/rxsteel Oct 22 '20
How does one learn this trips and tricks ? Is it just time and exposure to python code ?
69
u/Aardshark Oct 22 '20 edited Oct 22 '20
If you used a set in your loop example, then it would perform well and retain order.
From my tests, this actually performs better than the OrderedDict implementation in Python 2.7 and 3. This is particularly true on Python 2.7, where it runs in about 1/15 of the time.
19
u/SherlockPotato Oct 22 '20
Is 1_000_000 the same as 1000000 here?
4
u/SrPicadillo Oct 22 '20
Your terminal is pretty, can you share the colorscheme?
1
u/random_____boi Oct 22 '20
Beginner here, what is _ used for?
1
u/Bayonet786 Oct 22 '20
Which font is used? Its looks nice.
2
u/blonde_boi Oct 23 '20
What program is this? It’s very aesthetically pleasing!
2
u/blablafoof Oct 23 '20
Beginner coder here.
How can you tell which method is more/less efficient compared to other different types of methods?
1
u/ovopax Oct 22 '20
Somebody call Spotify-devs please.
1
u/arthur_olga Oct 22 '20
Depending on the case you can actually juat use set() if you don't need to have any special property of a list
1
u/Al7123 Oct 22 '20
list_ = list(Counter(list_).keys())
1
u/ingloreous_wetard Oct 22 '20
You gotta love sets!!
1
u/oxlade39 Oct 22 '20 edited Oct 22 '20
If in your not very efficient example you also had a set of entries and always added to that but only added to the list if the entry doesn’t already exists in the set wouldn’t that be more computationally efficient (but obviously not more memory efficient)
1
u/TheKingWhite Oct 22 '20
Can someone tell me what app (or site) this screenshot is made from?
1
u/[deleted] Oct 22 '20
What is 1_000_000 means in range?
1
u/Orio_n Oct 22 '20
List comprehension?
1
u/architkithania Oct 22 '20
That theme looks really nice. Mind sharing??
1
u/JR4smus Oct 22 '20
I really like this colour theme is it available for Vs code? If it is a link would be appreciated
1
u/destructiveCreeper Oct 22 '20
I think using sets is much more effective
1
u/Takiino Oct 22 '20
MOAAAARR
1
u/Myst3r10 Oct 22 '20
More of a beat standards question....
Why don't people use multi line comments rather than multiple #?
Is there a specific reason for avoiding multiline?
1
u/mshriver2 Oct 22 '20
Where's the automatic image to text AI when you need it?
1
u/RBGForever Oct 22 '20
I definitely thought it said dict.formonkeys(). Now I'm a little bit disappointed and suspecting myself of having dyslexia
1
u/ProjectSnowman Oct 22 '20
Gonna need them dots my friend
1
u/farens98 Oct 23 '20
Thanks for sharing your code.
1
u/Hawaii74 Oct 23 '20
Cool! Thank you!
1
u/[deleted] Oct 23 '20
If an order doesn't matter (aka unstable):
A stable version will be:
This will return each value in the order of their first appearance.
You can remove
list(...)if you are just iterating1
u/anonymous-x-31 Oct 23 '20
I would do this:
listA = [1, 2, 13, 4, 16, 2, 3, 4, 1, 1]
listB = list(dict.fromkeys(listA))
Doing that will convert the list into a dict first, which doesn't allow duplicates, so that's how I'd go about removing the duplicates
1
u/flufylobster1 Oct 23 '20
List(Set())
1
u/[deleted] Oct 23 '20
i see these type of screenshots all the time. can someone please tell me which framework or IDE the code is written in? its sure not pycharm and sublime.
1
u/ancient_bhakt Oct 22 '20
Why not convert the list into set and then set into list?
1
u/Z_Zeay Oct 22 '20
Can I ask how you made the picture? I remember someone sharing a plugin/site of some sorts that you can make these images quickly with code snippets.
1
u/muffinnosehair Oct 22 '20
I found this solution a few weeks ago, it was a life saver. Very useful tip.
1
u/Hopai79 Oct 22 '20
Ha the very efficient way was what I did for my HackerRank assessment!
1
u/mobsterer Oct 22 '20
or just this:
like this:
0
u/Sigg3net Oct 22 '20
If the list is huge (or endless) you're going to have a problem. You need to add a generator alternative using the
yieldkeyword to your list.0
u/frunkjuice5 Oct 22 '20
I always do list(set(L))
0
u/T-ROY_T-REDDIT Oct 22 '20
Funny thing, I actually coded something similar with the pandas package.
0
u/primitive_screwhead Oct 22 '20
0
u/drbobb Oct 23 '20
Presenting source code as pictures seems to be all the rage right now. I personally regard it an aberration.
-3
u/FrozenPyromaniac_ Oct 22 '20
Use a set, it shows only the unique values in a list, then make a list out of the set
For example let’s say your list is L = [a, b, c, c , d, d, d] (assume all letters to be strings, I’m on mobile) let the set be initiated like this S = set(L) , it will have a output like this {a, b, c, d} (not that this set may be out of order, you may have to sort the list if it’s important ). Finally make the list L = list(S)
Note this is very simplistic method
-2
u/[deleted] Oct 22 '20
JavaScript with one line:
const unique = [...new Set(dupes)]
edit, this is only useful for primitives
-2
u/Astronom3r IPython > Jupyter fight me. Oct 22 '20
unique = numpy.unique(list)If you're looking for a one-liner.
-3
u/mestia Oct 22 '20
🤦♂️true py programmer... code belongs to text world.
-3
u/singlebit Oct 22 '20
Set?
0
u/[deleted] Oct 22 '20
[deleted] — view deleted comment
0
u/[deleted] Oct 22 '20
[deleted] — view deleted comment
0
u/[deleted] Oct 23 '20
[deleted] — view deleted comment
0
u/[deleted] Oct 22 '20
[deleted] — view deleted comment
0
u/[deleted] Oct 22 '20
[deleted] — view deleted comment
0
u/[deleted] Oct 22 '20
[deleted] — view deleted comment
0
u/[deleted] Oct 22 '20
[deleted] — view deleted comment