A nightly pandas preprocessing job now takes 6 hours

A nightly pandas preprocessing job now takes 6 hours | Buncho

function                                   cumulative_s    share
DataFrame.apply(axis=1) -> clean_row         10,900       50%
DataFrame.apply(axis=1) -> make_key           5,950       27%
loop: iterrows + df.loc[i, "flag"] = ...      2,600       12%
groupby("user_id").apply(user_features)       1,100        5%
DataFrame.merge(lookup, on="sku_text")          700        3%
read_csv + parse                                390        2%

peak memory: 22 GB   (machine: 32 GB, other jobs also run on it)
read_csv columns: 41 loaded, 9 used downstream

Code excerpt

df = pd.read_csv("events.csv")                        # all columns, default dtypes

def clean_row(r):
    name = r["product_name"].lower().strip()
    price = r["price"] if r["price"] > 0 else None
    return pd.Series({"name": name, "price_clean": price})

df[["name", "price_clean"]] = df.apply(clean_row, axis=1)
df["key"] = df.apply(lambda r: f"{r['user_id']}_{r['sku_text']}", axis=1)

for i, row in df.iterrows():
    if row["price_clean"] is not None and row["price_clean"] > 500:
        df.loc[i, "flag"] = 1

feats = df.groupby("user_id").apply(lambda g: user_features(g))
df = df.merge(lookup, on="sku_text", how="left")

A nightly pandas preprocessing job now takes 6 hours

Scenario data

How to answer

Hints

AI feedback

Community solutions