See the answer
import pandas as pd
orders = pd.read_csv("data/orders.csv").drop_duplicates()
customers = pd.read_csv("data/customers.csv")
customers = pd.concat([customers, customers[customers["customer_id"] == "C015"]])
print(len(orders), customers["customer_id"].duplicated().sum())
m = orders.merge(customers, on="customer_id", how="left")
print(len(m))
print((orders["customer_id"] == "C015").sum(), (m["customer_id"] == "C015").sum())
try:
orders.merge(customers, on="customer_id", validate="many_to_one")
except pd.errors.MergeError:
print("MergeError")
fixed = orders.merge(customers.drop_duplicates("customer_id"), on="customer_id",
how="left", validate="many_to_one")
print(len(fixed))
Output (from running the code)
120 1
124
4 8
MergeError
120
A merge repeats a left row once for every match on the right. Customer C015 appears twice in customers, so each of C015's 4 orders comes out twice and 120 orders become 124 rows. validate="many_to_one" raises a MergeError straight away, and drop_duplicates("customer_id") on the customers table brings the count back to 120.
Common mistake
Fixing it with drop_duplicates() on the merged result: the rows are not exact copies once the customer columns differ, and the damage should be fixed before the merge, not after.
Likely follow-up
The two C015 rows differ: one says Delhi, one says Jaipur. What do you do? Don't drop either blindly. Check which row is newer or which one the source system treats as current, for example by signup date or a last-updated column, keep that one and tell the data owner. Dropping the first row is a guess.