We may need to work with lists in Python, sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods.
Naive method
First, we will look simple naive method to remove list duplicates. The basic logic is :
- Define a result list.
- Loop the input list.
- Check if the input list item is in the result list, if not append it to the result list.
numbers = [10,2,3,1,5,2,10,3,2,5] #input list
result = [] # To store the result
for item in numbers:
if item not in result:
result.append(item)
print(f"List after duplicates removal : {result}")
Output
List after duplicates removal : [10, 2, 3, 1, 5]Based on the ordering of the output list items we can remove duplicates by :
- Using set ( Not keeping the list insertion order)
- Using OrderedDict or dict (Keeps the list insertion order)
1. Using set (Not keeping the list insertion order)
This is a common way to remove list duplicates. We can use set to remove Python list duplicates. Create a new set instance using the input list. The set does not allow duplicate elements.
numbers = [10,2,3,1,5,2,10,3,2,5]
numbers = list(set(numbers))
print(f"List after duplicates removal : {numbers}")
Output
List after duplicates removal : [1, 2, 3, 5, 10]This method will not keep the list output order. We got output [1, 2, 3, 5, 10] instead of keeping the correct order [10, 2, 3, 1, 5]. If we are not concerned about the order of the output we can use this method for duplicates removal.
2. Using OrderedDict or dict (Keeps the list insertion order)
If we want to remove list duplicates by keeping the list insertion order we can use the OrderedDict.fromkeys function.
from collections import OrderedDict
numbers = [10,2,3,1,5,2,10,3,2,5]
result = list(OrderedDict.fromkeys(numbers))
print(f"List after duplication removal : {result}")
Output
List after duplication removal : [10, 2, 3, 1, 5]We can see that this method preserves the list ordering.
Starting with Python 3.7 built-in dictionary(dict) keeps the insertion order. So if we are on Python 3.7 or later we can use dict.fromkeys function to remove list duplicates.
numbers = [10,2,3,1,5,2,10,3,2,5]
result = list(dict.fromkeys(numbers))
print(f"List after duplication removal : {result}")
Output
List after duplication removal : [10, 2, 3, 1, 5]Conclusion
There you have it, different ways to remove Python list duplicates. If you are not concerned about list order it is better to use set based method. You can read about list append vs extend performance here.