Imagine this: you’re scrolling through Instagram. Your favourite creator just posted a rave review of a dress.
“This fits perfectly! Link in my bio 💕”
You tap the link, make a purchase, and moments later, three things happen in parallel:
- You get your gorgeous dress.
- The brand records a sale.
- The creator pockets a commission, maybe ₹200 on your ₹2,000 purchase.
Simple, right?
Not even close.
What is Reconciliation?
Before diving deeper, let’s clarify what reconciliation actually means. Financial reconciliation is the process of comparing two sets of records to ensure they match and are accurate. In our context, it’s the complex system of verifying that sales, commissions, and payments align perfectly across multiple platforms, brands, and creators.
Think of it as the accounting equivalent of making sure everyone’s version of the truth matches up — the brand’s sales records, our internal tracking, and the creator’s commission calculations all need to tell the same story down to the last paisa.
Behind the scenes, this seemingly smooth transaction triggers a monstrous reconciliation process at our startup — a process so complex and fragile that it kept two engineers working overtime every month to make sure everyone got paid correctly. I spent a whole year wrestling this beast, It’s a saga worth sharing, to say the least..
The Monster in the Data
When I took over the reconciliation project in December 2023, it felt like stepping into a war zone. The reconciliation system was a patchwork of hastily written scripts, connected with minimal safeguards. Our team members were manually cleaning data and running reconciliation scripts in environments that weren’t isolated from production systems — a practice that created both performance and security risks.
Every month-end, the data files grew to unmanageable sizes, system resources were pushed to their limits, and our database CPUs Utilisation consistently reached concerning levels. We lived in constant fear of traffic spikes — one unexpected surge and the entire system would freeze, triggering a cascade of problems. To make matters worse, preliminary reconciliation data was visible to creators before proper validation, creating potential trust issues when numbers needed correction.
The three major pain points:
- Customer Experience (CX) Was Compromised
Creators were seeing unverified numbers and often raised concerns publicly before we had a chance to validate them. Our manual quality checks were slow and error-prone due to time constraints — we were racing against the clock, running on zero sleep, surrounded by ten people for moral support, all while the creators’ trust hung in the balance. - MIS and Finance Were Flying Blind
The data was a mess. Mismatched sales, missing fields, and no clean, timely numbers for GMV (Gross Merchandise Value) or invoicing. We had to send reconciliation sheets mid-cycle to brands, who’d manually fill order statuses, and then we’d patch it all together just before deadlines. Finance wanted data early, but the whole process was delayed by manual data cleaning and late uploads. - Commission Payment Challenges Created Financial Risk
We paid creators for an entire month’s sales — even if those orders were later returned. It was like tipping a waiter for a meal you didn’t eat. The system marked sales as reconciled once brands approved them, but if reconciliation wasn’t completed by month-end, duplicate orders slipped through — not just into the creator payouts, but also into the next month’s brand invoicing. The result? Total chaos.
Why Was This So Hard?
Building effective reconciliation systems requires both technical expertise and a deep understanding of business processes. Our challenges stemmed from several factors:
- Data Format Diversity: We dealt with 100+ brands, each with their own data format and quirks. No standardization in sight.
- Inconsistent Data Delivery: Brands frequently changed their data formats without notice, requiring constant adaptation.
- A Last-Minute Data Dump: Most data arrived at month-end, leaving us just 4–5 days to clean, process, test, and finalize payments. Some e-commerce brands, especially those with massive volumes, sent their data on the very last day — leaving us with barely a day to make it all work.
- Commission Madness: Each brand had its own unique commission structure — ranging from flat rates and tiered slabs (based on new vs. returning users) to creator-specific rates, category-based variations, seller-level tweaks, and even mid-cycle changes. Often, multiple commission types ran simultaneously, making it hard to track. To make things harder, there were human factors too. If a brand changed its commission structure and informed us late, we had to retroactively correct those commissions during reconciliation — further complicating an already chaotic month.
- Missing and Mismatched Data: Sales amounts didn’t match between us and brands, commission percentages conflicted, some data was missing altogether. Fraud and ineligible creators added manual exceptions.
- Evolving Systems: As our sales tracking, commission structures, and reporting needs evolved, the reconciliation team had to adapt continuously while meeting tight deadlines.
- Excel Was a Joke: The data volume was too massive for spreadsheets. Jupyter notebooks were our only friends.
- Deadline Pressure: Creators expect their earnings accurate to the last paisa and on the 1st day of the month. No delays allowed.
- Reporting Headaches: Multiple stakeholders — MIS, creator dashboards, brand portals — all required accurate, timely data.
The Bandwidth Crunch and Scale Horror
You might think a month is plenty of time to do this. Nope.
One-third of the month was just execution time. Two engineers worked feverishly in the last week to clean, test, and patch the system. The first 23 days’ work often broke under the load of data scale and changing requirements on day 24. You couldn’t send the file back because you won’t get it back before the month ends!
The complexity of our reconciliation (Reco) process created significant resource constraints. One-third of each month was consumed by execution time alone. Two engineers worked intensively during the final week to clean, test, and stabilize the system. Work completed during the first three weeks often broke under the increased data load and changing requirements in the final days.
This was just the opening chapter of our reconciliation saga — a story of chaos, late nights, and relentless problem-solving. In the next part of this series, I’ll share how we began transforming this chaotic process into a more structured, secure, and reliable system. The journey from manual chaos to controlled processes taught us valuable lessons about building resilient reconciliation solutions that I’m excited to share.
Stay tuned. The ride’s just getting started.