Python provides several ways to remove duplicates from a list, enabling you to efficiently work with unique elements. In this blog post, we will explore different methods along with code examples to help you remove duplicates from a list in Python.
1. Using a Set
One straightforward approach is to convert the list into a set, which automatically removes duplicates due to its unique property. Afterward, we can convert the set back to a list if needed.
Example 1 :
my_list = [1, 2, 3,3, 2, 4, 1, 3, 5]
unique_list = list(set(my_list))
print(unique_list)
Output:
[1, 2, 3, 4, 5]Example 2:
In this example, the original list my_list contains duplicate strings. By converting it to a set using set(my_list), duplicates are automatically removed since sets can only contain unique elements. Finally, we convert the set unique_set back to a list using list(unique_set), resulting in the variable unique_list containing only the unique strings from the original list.
# Define a list with duplicate strings
my_list = ['apple', 'banana', 'orange', 'banana', 'grape', 'apple']
# Convert the list to a set
unique_set = set(my_list)
# Convert the set back to a list
unique_list = list(unique_set)
# Print the unique list
print(unique_list)Output:
['banana', 'grape', 'apple', 'orange']2. Using a List Comprehension
Another concise method involves using a list comprehension to iterate over the list, checking if each element is not already present in the sub-list preceding it. Example 1 :
my_list = [1, 2, 3,3, 2, 4, 1, 3, 5] unique_list = [x for i, x in enumerate(my_list) if x not in my_list[:i]] print(unique_list)
Output:
[1, 2, 3, 4, 5]Example 2:
In this example, we have a list of strings my_list that contains duplicates. The list comprehension iterates over each element in my_list while ensuring that only elements that haven’t been encountered before (i.e., not present in my_list[:i]) are included in the unique_list. As a result, the unique_list contains only the unique strings from the original list.
# Define a list with duplicate strings
my_list = ['apple', 'banana', 'orange', 'banana', 'grape', 'apple']
# Use list comprehension to create a unique_list without duplicates
unique_list = [x for i, x in enumerate(my_list) if x not in my_list[:i]]
# Print the unique list
print(unique_list)Output:
['apple', 'banana', 'orange', 'grape']3. Using the dict.fromkeys() Method
We can utilize the dict.fromkeys() method to create a new dictionary using the list’s elements as keys. Since dictionaries cannot have duplicate keys, this method effectively removes duplicates. Finally, we can convert the dictionary keys back into a list.
Example 1 :
my_list = [1, 2, 3, 2, 4, 1, 3, 5]
unique_list = list(dict.fromkeys(my_list))
print(unique_list)[1, 2, 3, 4, 5]
Example 2:
In this example, the original list my_list contains duplicate elements. By passing my_list as an argument to the dict.fromkeys() method, a dictionary is created with unique keys, effectively removing the duplicates.
# Define a list with duplicate strings
my_list = ['apple', 'banana', 'orange', 'banana', 'grape', 'apple']
# Use the dict.fromkeys() method to create a dictionary with unique keys
unique_dict = dict.fromkeys(my_list)
# Convert the unique dictionary keys back to a list
unique_list = list(unique_dict)
# Print the unique list
print(unique_list)Output:
['apple', 'banana', 'orange', 'grape']4. Using the collections.OrderedDict Class
The collections module provides the OrderedDict class, which maintains the order of elements while removing duplicates. By creating an OrderedDict from the list and converting it back to a list, we obtain a unique list in the original order.
Example 1
from collections import OrderedDict
my_list = [1, 2, 3, 3, 2, 4, 1, 3, 5]
unique_list = list(OrderedDict.fromkeys(my_list))
print(unique_list)
Output:
[1, 2, 3, 4, 5]Example 2
In this example, the original list my_list contains duplicate strings. By using the OrderedDict.fromkeys() method with my_list, an OrderedDict is created with unique keys, effectively removing the duplicates while preserving the order of elements.
from collections import OrderedDict
# Define a list with duplicate strings
my_list = ['apple', 'banana', 'orange', 'banana', 'grape', 'apple']
# Create an OrderedDict to preserve the order of elements while removing duplicates
unique_ordered_dict = OrderedDict.fromkeys(my_list)
# Convert the unique OrderedDict keys back to a list
unique_list = list(unique_ordered_dict.keys())
# Print the unique list
print(unique_list)Output:
['apple', 'banana', 'orange', 'grape']
