To remove duplicates from a Python list while preserving the order of the elements, use the code list(dict.fromkeys(list)) that goes through two phases: (1) Convert the list to a dict using the dict.fromkeys() function with the list elements as keys and None as dict values. (2) Convert the dictionary back to a list using the list() constructor. As dictionaries preserve the order of the keys, the list ordering is preserved.
Problem: How to remove duplicates from a Python list while keeping the order of the list elements preserved?
You may find this question a little awkward. What has removing duplicates to do with preserving the order of the elements? The reason is simple: a well-known and efficient way to remove duplicates from a list is to convert the list to a set—which is duplicated-free—and converting it back to a list. Here’s what you may find everywhere:
lst = [42, 42, 'Alice', 'Alice', 1] dup_free = list(set(lst)) print(dup_free) # ['Alice', 42, 1]
The back-and-forth conversion list(set(lst)) removes all duplicates from the list. However, it doesn’t preserve the order of the elements. In the example, the string 'Alice' now appears before the integer 42.
So, how to remove duplicates while preserving the order of the elements?
The most Pythonic and blazingly fast approach is to use a dictionary:
lst = [3, 3, 22, 22, 1] result = list(dict.fromkeys(lst)) print(result) # [3, 22, 1]
The dict.fromkeys() method creates a new dictionary using the elements from an iterable as the keys. Python dictionary keys are unique by default so converting our list into a dictionary will remove duplicates automatically. Once this has been done with our initial list, converting the dictionary back results in the duplicate-free list.
This is the most Pythonic way to remove duplicates from a Python list while preserving the order.
Is this method fast? Like sets, dictionaries use hash tables, which means they are extremely fast.
Do you want to develop the skills of a well-rounded Python professional—while getting paid in the process? Become a Python freelancer and order your book Leaving the Rat Race with Python on Amazon (Kindle/Print)!
Do Python Dictionaries Preserve the Ordering of the Keys?
Surprisingly, the dictionary keys in Python preserve the order of the elements. So, yes, the order of the elements is preserved. (source)
Countless online resources like this argue that the order of dictionary keys is not preserved. They assume that the underlying implementation of the dictionary key iterables uses sets—and sets are well-known to be agnostic to the ordering of elements. But this assumption is wrong. The built-in Python dictionary implementation in cPython preserves the order.
Here’s another example:
lst = ['Alice', 'Bob', 'Bob', 1, 1, 1, 2, 3, 3]
dic = dict.fromkeys(lst)
print(dic)
# {'Alice': None, 'Bob': None, 1: None, 2: None, 3: None}You see that the order of elements is preserved so when converting it back, the original ordering of the list elements is still preserved:
print(list(dic)) # ['Alice', 'Bob', 1, 2, 3]
However, you cannot rely on it because any Python implementation could, theoretically, decide not to preserve the order (notice the “COULD” here is 100% theoretical and does not apply to the default cPython implementation).
If you need to be certain that the order is preserved, you can use the ordered dictionary library. In cPython, this is just a wrapper for the default dict implementation.
Source Article: How to Remove Duplicates From a Python List?
Removing Duplicates From Ordered Lists For Older Versions
Dictionaries only became ordered in all Python implementations when Python 3.7 was released (this was also an implementation detail of CPython 3.6).
So, if you’re using an older version of Python, you will need to import the OrderedDict class from the collections package in the standard library instead:
from collections import OrderedDict lst = [1, 1, 9, 1, 9, 6, 9, 7] result = list(OrderedDict.fromkeys(lst))
The output is the following duplicate-free list with the order of the elements preserved:
print(result) # [1, 9, 6, 7]
Interactive Code Shell
Let’s try this method in our interactive Python shell:
Exercise: Run the code. Does it work?
You can find more ways to remove duplicates while preserving the order in this detailed blog article:
Related tutorial: Python List: Remove Duplicates and Keep the Order
Where to Go From Here?
Enough theory. Let’s get some practice!
Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.
To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?
You build high-value coding skills by working on practical coding projects!
Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?
🚀 If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.
If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.