Learn the coding skills for your next job

Removing Duplicates From a List in Python

Let’s go over a few idiomatic ways to remove duplicates from lists in Python.

Method #1 – Create a new list (simplest)

This is the easiest algorithm to code, but because it requires creating a new list, also requires more memory and is a bit slower.

def remove_duplicates(original): deduped = [] for item in original: if item not in deduped: deduped.append(item) return deduped
Code language: Python (python)

We take advantage of Python’s in keyword here, only adding each item to the final list if it isn’t already present.

Learn Python, get hired

Python has been growing in popularity for over 20 years, and today is used by tens of thousands of companies to deliver reliable software. If you are interested in a career in Python get started by writing real code with us.

Method #2 – Create a new list with syntactic sugar (less code, harder to understand)

def remove_duplicates(original): deduped = [] [deduped.append(item) for item in original if item not in deduped] return deduped
Code language: JavaScript (javascript)

This is the same exact code from a performance standpoint but only uses one line. If you’re into code golf, then this might be your solution.

Method #3 – Use the built-in “set” data structure (fast, loses order)

A set() is a group of values that doesn’t contain any duplicates. By casting a list into a set and back, you remove all duplicates. The main drawback here is that you’ll lose your ordering.

def remove_duplicates(original): return list(set(original))
Code language: Python (python)

This method will be faster in most circumstances than the previous two because each transfer is O(n) in big-o notation terms. A group of two O(n) operations is faster than one O(n^2) operation. As a bonus, it even uses less code.

Method #4 – Use an ordered dictionary (fast, maintains order)

By using the collections libraries’ OrderedDict type, we can maintain the ordering of the list while maintaining the same Big-O that we had with a set().

from collections import OrderedDict def remove_duplicates(original): return list(OrderedDict.fromkeys(original))
Code language: Python (python)

Trying to find your next programming job?

If you are a self-taught developer having trouble finding your first programming job, we've got your back! We have the learning resources and tight-knit dev community that you need to land the coding job you've been looking for. To get started, create a free account and join our Discord community.

Have questions or feedback?

If we've made a mistake in the article, please let us know so we can get it corrected!