Dedupe preserving order in Python

Order-preserving dedupe - a daily-driver data cleanup.

def dedupe(items):
    seen = set()
    out = []
    for item in items:
        if item not in seen:
            seen.add(item)
            out.append(item)
    return out

How it works

1. Tracks already-seen items in a set.
2. First occurrences are kept, in their original order.
3. Later duplicates are skipped.

Keywords and builtins used here

• dedupe
• def
• for
• if
• return
• set

The run, in numbers

Lines

8

Characters to type

123

Tokens

40

Three-star pace

90 tpm

At the three-star pace of 90 tokens a minute, this run takes about 27 seconds.

Type this snippet

Step 5 of 10 in Collections, step 27 of 72 in Language basics.

← Previous Next →

Dedupe preserving order in other languages

• Rust