Pinterest had three different teams each training their own image-understanding model. Each worked. Together they were a slow-motion disaster — three times the cost, three times the maintenance, and three chances to drift apart.
Visual Cropper, Lens (camera search), and Shop the Look each had its own bespoke visual embedding. Independently reasonable; collectively unsustainable. The teardown is less about a model architecture and more about a kind of technical debt that ML systems accumulate quietly.
To let computers compare images, you convert each image into a visual embedding — a vector capturing what's in it — using a trained model. Pinterest had several visual-search features, and each had grown its own embedding model.
That means three models to train, three to serve, three to store vectors for, and three to debug when they misbehave. Worse, they can drift: the same image understood differently by each, so the product feels inconsistent. Every new feature added another model and another increment of this debt.
Pinterest replaced the bespoke models with one unified multi-task visual embedding that serves all visual-search surfaces. It's trained with metric learning over CNN features — the model learns to place visually/semantically similar images near each other in vector space — with a multi-task objective so the single embedding works well across Cropper, Lens, and Shop the Look rather than being tuned for just one.
The payoff compounds in the right direction. Now there's one model to train, serve, and store; one place to improve. And because every surface reads the same embedding, an improvement to the shared model lifts all features at once — the inverse of the old world, where each surface improved alone and the others drifted.
Worth knowing
This is the ML-infrastructure version of paying down technical debt, and the economics are the point. Three models isn't 3x the work — it's worse, because they interact, drift, and each new feature multiplies the burden. Consolidating to a shared foundation isn't 'cleanup'; it changes the slope of your future cost curve and turns single improvements into fleet-wide wins.
The gap it reveals
Teams treat 'each feature gets its own model' as natural — it's locally optimal and ships fast. The realisation is that proliferating bespoke models is compounding technical debt (multiplied training/serving/storage cost plus drift), and that a unified multi-task embedding is a leverage point where one improvement benefits everything. Seeing consolidation as a cost-and-quality strategy, not janitorial work, is the systems-level view.
In the interview room
In ML-system design, proposing a shared embedding/feature foundation across use cases — rather than a model per feature — signals platform thinking: "I'd train one multi-task visual embedding so we maintain and improve a single model, avoid drift across surfaces, and let each gain lift all of them." It shows you optimise for the org's long-run cost curve, not just one feature's launch.
The reframe
The locally optimal choice — every team builds exactly what it needs — is often globally expensive. Pinterest's win was zooming out from 'is each model good?' to 'what does maintaining all of them cost us forever?' Shared foundations feel slower to build and are almost always cheaper to live with. The leverage is in what you don't duplicate.
Three models that each work can still be the wrong answer. Build the one that makes all of them unnecessary.
Primary source →
pinterest-engineering — Unifying Visual Embeddings for Visual Search at Pinterest