How a Retail Analytics Team Replaced a Failing In-House Flipkart Scraper.JPG
                     How a Retail Analytics Team Replaced a Failing In-House Flipkart Scraper
The Client
The retail analytics function inside a large Indian consumer goods group 
— a central team serving five brands across electronics and home 
categories, all selling on Flipkart. Around 1,200 SKUs under coverage, 
feeding pricing, supply, and category teams across the group.
Client details are anonymised. Figures are representative of the 
engagement.
The Problem: A Scraper Everyone Depended On and Nobody 
Owned
The scraper had been built two years earlier by a data engineer who no 
longer worked at the company.
It had started, as these things do, as a small internal project. One 
engineer, roughly six weeks, a Python service and a cron job. It worked. It 
was celebrated. It went into production, informally, in the way that 
internal tools go into production — which is to say, nobody ever decided 
it was production, and so nobody ever gave it an owner.
Two years later, five brands were making pricing decisions on its output.
The engineer who built it had left. The scraper had been inherited, 
reluctantly, by a data engineer who described it in our first conversation as 
"about fifteen percent of my job and one hundred percent of my anxiety."
What Was Actually Wrong
The team's initial framing was that the scraper "kept breaking." That turned 
out to be the least of it.
It broke, and it broke at the worst possible times. Eleven parser 
failures over eighteen months. Two of them during Big Billion Days — which 
is not a coincidence but a structural consequence: sale events introduce 
page elements the parser has never seen, and they do it precisely when the 
data matters most.
It failed silently, which was far worse. Three months before we were 
engaged, a Flipkart front-end change had caused the parser to begin 
returning the Plus member price in the price field for a subset of records. 
The pipeline did not break. It kept running. It returned numbers. The 
numbers were plausible.
For roughly five weeks, one brand's pricing team believed a competitor had 
cut prices across a range of SKUs. They responded. They cut their own prices 
to match 
a competitor move that had not happened.
Nobody caught it. There was no null-rate monitoring, no delta gate, no range 
check. The parser asked for a price, got a number, and wrote a number. The 
number was for the wrong customer tier.
It captured the wrong things. One row per product page. No seller array 
— so the brand-protection team, which had asked for unauthorised-seller 
data twice, was told each time that it "would be a big change." No variant 
matrix — so in_stock: true on a listing whose main configuration had been 
unavailable for a fortnight. Single pincode — so the entire tier-2 and tier-3 
picture, which was where the group's growth was, was simply absent.
It had no capacity headroom. During the last sale event it had saturated 
on day one, silently dropped roughly 40 percent of its target records, and 
reported success.
Nobody knew what it was supposed to capture. This was, in the end, 
the finding that settled the internal argument. There was no expected-record 
count. So there was no way to know what had been missed. The pipeline 
could not distinguish "captured everything" from "captured 60 percent," and 
it reported both identically.
The Honest Cost
The team's CFO had approved the original build on the basis of a simple 
comparison: six weeks of one engineer, versus a vendor contract. The 
build won easily.
That comparison was not wrong. It was incomplete, in a way that only 
becomes visible in retrospect.
Cost line Budgeted at build time? Actual over 18 months
Initial build ✅ Yes ~6 engineer-weeks
Infrastructure and proxies ✅ Yes As modelled
Parser maintenance ✅ No ~11 incidents, ~3–5 days each
Sale-day firefighting and on-call ✅ No 2 events, senior engineer, weekends
Seller array, variants, pincodes — all 
Feature requests never delivered ✅ No
deferred indefinitely
Data quality gates ✅ No Never built
One five-week episode of 
Decisions made on wrong data ✅ No
unnecessary price cuts
Add the maintenance lines and the in-house scraper had consumed 
substantially more engineering time in maintenance than it had in 
construction — while delivering a fraction of the fields the business had 
asked for.
And the last line is the one that ended the debate. The unnecessary price 
cuts, sustained across a range of SKUs for five weeks, cost more in 
foregone margin than several years of a vendor contract.
The Solution
Product Data Scrape replaced the in-house pipeline over four weeks, 
running both in parallel for two of them.
• A schema that models the platform — all price tiers preserved 
separately, nested seller arrays with F-Assured and default-seller flags 
and stable seller IDs, full variant matrices with structured attributes, 
structured bank offers with cap-aware computation, pincode resolution 
across a 60-pincode panel, sale-event deal flags.
• Validation gates between capture and delivery — range checks, 
event-aware delta checks, cross-field consistency, schema 
conformance, and, critically, per-field null-rate monitoring alerting on 
change rather than level. This is the gate that would have caught the 
Plus-price incident within a day.
• Expected-record counts. Every run knows what it intended to 
capture. A shortfall is an alert, not a silence.
• Capacity provisioned for the peak, with degradation rules defined in 
advance that protect hero-SKU capture under load, and a T-7 rehearsal 
before every sale event.
• The maintenance burden moved off the client entirely.
The Results
Metric In-House Scraper Product Data Scrape
Fields captured 9 60+
Seller array ✅ ✅ Full, with stable seller IDs
Variant matrix ✅ ✅ Full
Pincode coverage 1 60
Data quality gates 0 6 gate types
0 delivered; 3 caught and 
Silent-failure incidents (12 months) ≥1 known, unknown true count
quarantined
Sale-day record completeness ~60% (undetected) 99%+, with completeness reported
Engineer time on pipeline ~15% of one FTE, ongoing ~0
Brand-protection team served ✅ Deferred 2 years ✅ Enforcement queue live in week 3
Figures are representative of the engagement outcome.
The reassigned engineer's time went to a demand-forecasting project that 
had been backlogged for a year.
And the brand-protection request that had been deferred for two years — 
because adding the seller array to the in-house scraper was "a big change" 
— was live in the third week, because it had never been a big change. It 
had simply been in a schema nobody had the capacity to extend.
The Lesson
The build-versus-buy conversation is almost always held on the wrong axis.
The question is not can we build this? Of course you can. A competent 
engineer will have a working Flipkart scraper in a few weeks, and it will 
feel like a win.
The right questions are:
• Who owns it in eighteen months? Not who builds it. Who is on call 
when it breaks during a sale, after the person who built it has moved 
on.
• How will you know when it is wrong? Not broken — wrong. Silently 
returning a plausible number from the wrong price tier. If you cannot 
answer this, you do not have a data pipeline. You have a source of 
confident errors, and the confidence is the dangerous part.
• What will it cost when a decision is made on bad data? Put a 
number on it. It is almost always larger than the entire engineering 
cost of the alternative, and it is the only line in the comparison that 
nobody ever includes.
The engineer who inherited this scraper had the most accurate summary of 
the whole engagement, and he offered it in the first meeting: "It's not that 
Wit doorke sWn'itt wh oPrrko. dItu'sc tth Daat tI ah aSvcer anpoe way of knowing when it doesn't."
Product Data Scrape replaces in-house Flipkart scrapers with a maintained, 
validated, schema-complete data feed: all price tiers, nested seller arrays 
with stable identifiers, full variant matrices, structured offers, pincode 
panels, sale-event capacity — and validation gates that catch the silently 
wrong values before they reach a decision.
If you have an in-house scraper and no null-rate monitoring, we can tell 
you in one parallel run whether it is currently lying to you.
Product Data Scrape — turning marketplace complexity into decision-ready 
data.