The cheapest cloud compute comes with a catch: it can be yanked away mid-job. Most teams avoid it for anything important. Razorpay used it for almost everything — by being surgical about the one piece that mustn't die.
Processing 1PB+ per day strained their third-party platform, and they wanted to run on cheap Spot instances — but Spot interruptions kept killing the Spark driver, which takes the whole job down with it. The naive lesson is 'don't use Spot for big jobs.' Razorpay's actual lesson is more useful: use it for the parts that can die, not the part that can't.
Cloud providers sell spare capacity at a steep discount as Spot instances — often ~70% cheaper — with one string attached: they can reclaim the machine with little notice. For replaceable work, that's a great deal; if a worker vanishes, you just redo its chunk elsewhere.
The danger is putting something irreplaceable on Spot. In a Spark job, the driver coordinates everything; lose it and the entire job dies. So the rule isn't 'avoid Spot' — it's 'never put the coordinator on Spot.'
Razorpay migrated to Amazon EMR and used YARN node labels to physically separate the two kinds of work. The Spark driver is pinned to On-Demand nodes (full price, never reclaimed) — a small, fixed cost for the one component that can't tolerate interruption. The executors/tasks, which are replaceable, run on Spot nodes: if AWS reclaims one, the lost task simply reruns elsewhere and the job continues.
They also did "fat executor" tuning — sizing each executor to use cores and memory more efficiently — for 13–15% faster jobs. Net result: 11% better performance and 21% lower cost, while keeping the bulk of compute on the cheap tier.
Worth knowing
The transferable principle is classifying work by its tolerance for interruption. Almost any large compute workload has a small, stateful, must-not-die core and a large, stateless, easily-redone bulk. Map the core to reliable capacity and the bulk to cheap interruptible capacity, and you capture most of the discount with almost none of the risk. The trick is the classification, not the tool.
The gap it reveals
Most engineers see Spot as 'risky, avoid for real work.' The cost-savvy realisation is that interruptibility is a property you can route around: identify the irreplaceable component (the driver/coordinator), give it stable capacity, and run everything replaceable on Spot. Knowing to segregate workloads by fault-tolerance — not to avoid the cheap tier wholesale — is real cloud-cost engineering.
In the interview room
Cost rarely comes up unprompted, so raising it stands out: "for batch compute I'd run replaceable executors on Spot for the discount, but pin the driver/coordinator to On-Demand so an interruption costs a task, not the job." That one sentence shows you understand both the pricing model and the failure model — and can trade them off deliberately.
The reframe
Cheap-but-unreliable resources aren't off-limits; they're a tool you aim precisely. The discipline is asking, of every component, 'can this be interrupted and redone?' — then placing it accordingly. Razorpay didn't avoid the risky-cheap option; they understood their own job well enough to know exactly which 5% couldn't touch it.
Don't avoid the cheap, fragile tier. Know which part of your workload can afford to die on it.
Primary source →
aws.amazon.com — How Razorpay Achieved 11% Performance Improvement and 21% Cost Reduction with Amazon EMR