A Tale of Two Products

As a technologist, it is incredibly rewarding, stimulating and challenging (not necessarily in that order!) to be on, or lead, a team that is building products that are in the following categories:

• A seed-stage/growth-stage product — a product that is being built from the ground up, and there is a race to get customers to try it and get feedback and iterate very fast and improve.
• A scale-stage product — a product that has product-market fit, an active — and growing! — customer base, where you’re racing to add features, ship bugfixes and improvements.

To be on a team that is building products in either of these stages can do wonders for learning and professional/career growth. It is arguable that maybe the first category can power “exponential” growth, especially for engineers in the first few years of their professional journey; while the second category is well-served by — and propels further — someone that is more experienced. At any rate, either of these can be incredibly rewarding.

What happens if/when one has the opportunity — and the challenge — of leading a team that is building/shipping products in both categories?! This can be an opportunity of a lifetime — while at the same time being incredibly challenging, always exhilarating and sometimes taxing. I had this opportunity in the last year or so at The MotherShip — and I thought I’d use this break to reflect on this. I want to look at what I learned, the trials and tribulations, and the sweet satisfaction to be had when one pauses to catch their breath on what is a rocket ship! 🚀 🚀 🚀

Straddling Two Boats

Engineering products — and leading teams that are building and shipping products — has two key pillars: Building and Shaping. (There is a third pillar — Selling; but I’m going to park this aside as far as this post is concerned. This is also because the selling part is less of a focus for senior engineers, and more at the intersection of product and field/business teams.) So, Builders and Shapers it is! For the engineers on the team, it is mostly builders. (Again, this is an over-simplification — senior engineers on the team are using their influence to shape the product while actively contributing to building it out as well!) For the builders, then — how different is it to work on teams building seed- (or growth-) stage products versus scale-stage products? Here’s what I have experienced:

1. (Rate/volume of) Code contributions. For a seed-stage/growth-stage product, there is a lot of code being written, usually very rapidly. Effective engineering teams are also investing in putting the build/test/deploy infrastructure for the code that’s getting constructed. In contrast, for a scale-stage product, the velocity may be lower; often, Management may decide to trim their investments in the scale-stage product, prioritizing only the critical customer asks. Even if the velocity is similar, the *intensity* is much lower. I define intensity as the ratio of PRs getting checked in, to the number of developers contributing those PRs.
2. Design choices. This one is obvious. For a growth-stage product, the team is focused on making the “correct” set of trade-offs for technology, architecture and design choices. For a scale-stage product, the team is, often, if not always, constrained by the architectural legacy and/or the design choices made by previous/past contributors.
3. Technical debt. The trade-offs, vis-a-vis technical debt, that the team is making are markedly different from growth-stage and scale-stage products! For the former, the team is focused on ensuring that they do not *accrue* technical debt early on. For the latter, the team is focused on paying back, or keeping a handle on, technical debt that is *already accrued*.

There may be other factors, but I view these as the most significant.

As an engineering leader, these are the considerations that will inform decision making — decisions like staffing each of the set of products. Of course, there’s the overriding factor — everyone wants to be building the shiny new thing! There’s usually no/few takers for “keeping the lights on” on the legacy product/service. This is often a lopsided perspective, though. Ensuring that the team operates a reliable, resilient service, prioritizing shipping critical feature asks, bugfixes and improvements is something that that phrase scarcely does justice to!

Another crucial consideration is that the operability and observability needs are *vastly* different for these products. The scale-stage product needs:

1. Top-notch observability. Effective monitoring — focused on low TTD and low/no noise is a boon. Another non-negotiable is a set of dashboards that help the on-call engineer quickly and effectively triage and isolate the issue. A definite good-to-have — almost must-have — is right automation for self-mitigation. Or at the least, a set of SOPs/guidelines that the on-call engineer can work through for mitigating a high fraction of issues.
2. Reliable deployment gates. With a legacy product, adding a seemingly simple feature can interact with multiple features/feature usages that already exist. While unit/integration/black-box/functional tests help, having a set of synthetic workloads in a staging environment that put every build being deployed through the paces to tease out issues before they might escape into production will reduce a lot of disruptions.
3. Fault-injection frameworks. This complements — or augments, or forms a part of — reliable deployment gates. This goes a long way toward smoking out issues in staging/pre-prod.

Having these is *crucial* to staffing the scale-stage product with a lean crew, so that the seed-stage/growth-stage product can get all the “oxygen” it needs!

The growth-stage product does not (yet, at this stage) need the above — though effective teams will start taking baby steps to have the MVP versions of each of these. All the same, not having these — or having extremely stripped down versions of these — may not impact business outcomes for the growth-stage product. For the scale-stage product, though, not having these can be costly — in terms of handling support tickets due to a buggy build making it into prod, or in terms of hotfixes that will disrupt (at least some engineers on) the team.

Finally, the support commitments on the products are — very obviously — as stark as night and day! The scale-stage product may be doing very healthy revenue, with (many) hundreds, or (tens of) thousands of users/customers. Again, no one wants to do “phucking” support — but it is a business need, and as a leader one has to be creative in terms of meeting this need with as lean a crew as possible, that is disrupted to the minimum.

I found straddling these to be a significant challenge — at least early on, till I settled into my own groove (and as did the team). Here is what I found the most challenging:

1. Completely different tech stacks. The tech stacks were *completely different*! This is not just programming languages or frameworks, but just completely different technology choices at a *fundamental level*!
2. Staffing challenges. Attrition! Onboarding new engineers! Onboarding new engineers has its own curve — but this is even steeper when you have to get these engineers on the on-call rotation for the legacy product because of attrition on the team!
3. Support commitments. The scale-stage product was doing (triple digit!) millions in revenue. Being on top of support tickets, shipping fixes and improvements was obviously necessary — but it took a toll. (The reasons are not hard to guess — we were lagging on some of the markers I wrote about above.)

How did I go about it? Well, there’s no magic bullet. I immersed myself in the product, learning about the product, the technology, hiring and onboarding people, building the right relationships across the crew … I knew I had to focus on getting one thing “right” first. Of course, in the moment, it looked like getting a bunch of things “just about right”. I picked the scale-stage product to set things right — this, to me, was all about customer-obsession and using that as a launchpad. Put the customer first — they are paying for the scale-stage product; and if you lose those, then you lose the opportunity to pitch your seed-stage product to them as well. Keep them onboard — and to do that you need to run a tight ship on ops — and they’ll often be wanting to try out a new flavor; at least for newer workloads before they commit to migrating existing ones. Put the customer first — use that to see how they use the product, what usage scenarios do they care about more than the others, what are their pain points …

And then work backwards from that. Do we have the right dashboards? Umm, well, we could do better. Alright, let’s do that. What do we need to do to build context on this product? Right — what partnerships do I need to build to enable that? What are the gaps in our automation? What is our telemetry telling us about issues *just below* the threshold that do not result in support tickets, but are affecting our customers, nonetheless? Let’s allocate some capacity to getting these. Of course, it was heads-to-the-desk work.

But wait, you say, you still had to build out the seed-stage product?! Yes, the team did have to build that out. And we did that by allocating some engineers on the team that were completely plugged out of *anything* to do with the legacy product. Including on-call rotations. Have a lean (or, well, lean-enough) crew focus on bugfixes, critical improvements that will contribute to service reliability. And then there were (reasonably) senior engineers that were contributing to build the new product, but were on-call for the legacy flavored one. This was an opportunity to learn how customers used the legacy product, their pain points and then use that to shape the new product. Oh, there’s this use case for legacy drivers that we need to support. Ah — customers really care about performance for these workloads, and are less-sensitive to performance about these other workloads. And such.

How did we do? I want to believe we did “good enough”. What is good enough? No escalations on the legacy product. We got the high-impact things into the beta milestones for the new product flavor.

What did we not do so well? We did not pay back nearly enough tech debt on the legacy product. We continue to be slowed down due to this. On the new product, we did not go deep enough into the new technology stack, the frameworks, the internals to build *deep* proficiency fast enough. This meant we shipped with fewer differentiators.

For me, as an engineering leader, this challenge was worth every moment living it. It helped me grow — as a leader, as an engineer. It helped me see some of my blind spots; I believe I am now better prepared to address some of them! Yes, it was tiring — but the satisfaction of taking on this challenge, and emerging on the other end mostly whole; that was energising. Onward and upward!

It was the best of times. It was the worst of times. Charles Dickens started his tale with these words. I conclude mine with these!