The Reformist CTO’s Guide to Impact Intelligence

Example #1: A Customer Support Chatbot

What’s the contribution of an AI customer support chatbot to limiting call volume (while maintaining customer satisfaction) in your contact center?

Figure 3: Downstream Impact of an AI Chatbot

It is not enough anymore to assume success based on mere solution delivery. Or even the number of satisfactory chatbot sessions which Figure 3 calls virtual assistant capture. That’s proximate impact. It’s what the Lean Startup mantra of build-measure-learn aims for typically. However, downstream impact in the form of call savings is what really matters in this case. In general, proximate impact might not be a reliable leading indicator of downstream impact.

A chatbot might be a small initiative in the larger scheme, but small initiatives are a good place to exercise your impact intelligence muscle.

Example #2: Regulatory Compliance AI assistant

Consider a common workflow in regulatory compliance. A compliance analyst is assigned a case. They study the case, its relevant regulations and any recent changes to them. They then apply their expertise and arrive at a recommendation. A final decision is made after subjecting the recommendation to a number of reviews and approvals depending on the importance or severity of the case. The Time to Decision might be of the order of hours, days or even weeks depending on the case and its industry sector. Slow decisions could adversely affect the business. If it turns out that the analysts are the bottleneck, then perhaps it might help to develop an AI assistant (“Regu Nerd”) to interpret and apply the ever-changing regulations. Figure 4 shows the impact network for the initiative.

Figure 4: Impact Network for an AI Interpreter of Regulations

Its proximate impact may be reported in terms of the uptake of the assistant (e.g., prompts per analyst per week), but it is more meaningful to assess the time saved by analysts while processing a case. Any real business impact would arise from an improvement in Time to Decision. That’s downstream impact, and it would only come about if the assistant were effective and if the Time to initial recommendation were indeed the bottleneck in the first place.

Example #3: Email Marketing SaaS

Consider a SaaS business that offers an email marketing solution. Their revenue depends on new subscriptions and renewals. Renewal depends on how useful the solution is to their customers, among other factors like price competitiveness. Figure 5 shows the relevant section of their impact network.

Figure 5: Impact Network for an Email Marketing SaaS

The clearest sign of customer success is how much additional revenue a customer could make through the leads generated via the use of this solution. Therefore, the product team keeps adding functionality to improve engagement with emails. For instance, they might decide to personalize the timing of email dispatch as per the recipient’s historical behavior. The implementation uses behavioral heuristics from open/click logs to identify peak engagement windows per contact. This information is fed to their campaign scheduler. What do you think is the measure of success of this feature? If you limit it to Email Open Rate or Click Through Rate you could verify with an A/B test. But that would be proximate impact only.

Leverage Points

Drawing up an impact network is a common first step. It serves as a commonly understood visual, somewhat like the ubiquitous language of domain driven design. To improve impact intelligence, leaders must address the flaws in their organization’s idea-to-impact cycle ( Figure 6). Although it is displayed here as a sequence, iteration makes it a cycle.

Any of the segments of this cycle might be weak but the first (idea selection) and the last (impact measurement & iteration) are particularly relevant for impact intelligence. A lack of rigor here leads to the vicious cycle of spray-and-pray ( Figure 1). The segments in the middle are more in the realm of execution or delivery. They contribute more to impact than to impact intelligence.

Figure 6: Leverage Points in the Idea to Impact Cycle

In systems thinking, leverage points are strategic intervention points within a system where a small shift in one element can produce significant changes in the overall system behavior. Figure 6 highlights the two leverage points for impact intelligence: idea selection and impact measurement. However, these two segments typically fall under the remit of business leaders, business relationship managers, or CPOs (Product). On the other hand, you—a tech CXO—are the one under productivity pressure resulting from poor business impact. How might you introduce rigor here?

In theory, you could try talking to the leaders responsible for idea selection and impact measurement. But if they were willing and able, they’d have likely spotted and addressed the problem themselves. The typical Classic Enterprise is not free of politics. Having this conversation in such a place might only result in polite reassurances and nudges not to worry about it as a tech CXO.

This situation is common in places that have grown Product and Engineering as separate functions with their own CXOs or senior vice presidents. Smaller or younger companies have the opportunity to avoid growing into this dysfunction. But you might be in a company that is well past this orgdesign decision.

Action #1: Introduce Robust Demand Management

Product may own idea triaging and prioritization, but they don’t always document their rationale for idea selection very well. Whether it takes the form of a business case or a justification slide deck, a good one ought to answer all the questions in the Robust Demand Management Questionnaire.

A commonly understood impact network helps answer some of the above questions. But what's absolutely essential for robust demand management is answers to the above, not the impact network. Answering the above makes for SMART (Specific, Measurable, Achievable, Relevant, Time-bound) ideas. Else they might be VAPID (Vague, Amorphous, Pie-in-the-sky, Irrelevant, Delayed). It is impossible to validate the business impact of VAPID ideas post tech delivery. This leads to the bad effects of Figure 1.

To mitigate this scenario, you must assert your right to allocate the bandwidth of your teams, an expensive business resource, to adequately documented ideas only. Do so for significant efforts only, not for every story or bug. Define your own thresholds, two person-weeks for example.

Make a distinction between prioritization and scheduling. The former is the act of assigning priority to a work item. The latter is about slotting the work-item into a work-cycle (e.g. sprint). Many organizations don't make this distinction and think of prioritization as inclusive of scheduling. Rethink this. Product still gets to prioritize. Scheduling has always been subject to practical considerations like dependencies or the availability of certain team members. It shall now also require answers to the above.

If the questions above were answered as part of idea triage, Engineering must obtain access to them. Robust demand management means that engineering teams only take up work that is documented as above, in addition to your usual documentation requirements (e.g. PRD). This means it is not just you, your teams too must understand the what, how and why of impact intelligence. More on this later.

Note that adequately documented does not necessarily mean well justified. Robust demand management does not mean Engineering makes a judgement call as to whether something is worth doing. It only makes sure that the projected benefits and timelines are documented in a verifiable manner. Product still gets to assign priority. To get the work scheduled, they could even answer “we don’t know” to some of the questions posed. At least we'll know how much of engineering capacity gets allocated to well-informed vs. ill-informed prioritization.

I helped Travelopia, an experiential travel company, implement an early version of robust demand management. Here’s a conference video where they talk about it.

This approach will have its detractors, especially among those at the receiving end of such robustness. They might deride it as gatekeeping. You must take the lead in explaining why it is necessary. A later section provides some guidance on how you could go about this. For now, I’ll only list the common objections.

1. This will slow us down. We can’t afford that.
2. Self-Censorship: Let’s put our house in order first.
3. It’s not agile to consider all this upfront.
4. Innovation isn’t predictable.
5. Our PMO/VMO already takes care of this.
6. This isn't collaborative.
7. We don’t have the data.

The last one is more than an objection if it is a fact. It can be a showstopper for impact intelligence. It warrants immediate attention.

Action #2: Pay Down Measurement Debt

Measurement debt is best addressed through a measurement improvement program. It comprises a team tasked with erasing blind spots in the measurement landscape. But it would require separate funding, which means a tech CXO might need to convince their COO or CFO. If that’s not feasible, consider doing it yourself. 

Take the lead in reducing measurement debt. Advise your teams to instrument application code to emit structured impact-relevant events at meaningful points. Store it and use it to build analytics dashboards that will help validate proximate and downstream impact. They must be built alongside new functionality. Ensure to only fill the gaps in measurement and integration. No need to duplicate what might already be available through third party analytics tools that Product might already have in place. Measurement debt reduction might be easier if there's a product operations team in place. Your developers might be able to work with them to identify and address gaps more effectively.

The effort may be considered as part of coding for non-functional (cross-functional) requirements. Think of it as another type of observability: the observability of business impact. Do it only for important or effort-intensive functionality in the beginning. It is a bit unconventional, but it might help you be a more impactful CTO.

Read more about measurement debt here

Action #3: Introduce Impact Validation

When you adopt impact measurement as a practice, it allows you to maintain a report as shown in the table below. It provides a summary of the projection vs. performance of the efforts we discussed earlier. Product does this usually, and if so, Engineering should ask to participate. If Product isn’t doing it, Engineering should take the lead and drive it in order to avoid the spray and pray trap explained earlier. Otherwise, you won’t have an alternative proposal when you get badgered about developer productivity.

You now have the opportunity to conduct an impact retrospective. The answer to the question, “By how much and in what time frame” (item 3(b)(i) in the Robust Demand Management Questionnaire), allows us to pencil in a date for a proximateimpact retrospective session. The session is meant to discuss the difference between projection and performance, if any. In case of a deficit, the objective is to learn, not to blame. This informs future projections and feeds back into robust demand management.

It's okay if, in the first year of rollout, the actuals are much weaker than what was expected. It might take a while for idea champions to temper their optimism when they state expected benefits. It should have no bearing on individual performance assessments. Impact intelligence is meant to align funding with portfolio (of initiatives) performance.

Impact measurement works the same for downstream impact, but impact validation works differently. This is because unlike proximate impact, downstream impact may be due to multiple factors. The table below illustrates this for the examples discussed earlier. Any observed improvement in the downstream metric cannot be automatically and fully attributed to any single improvement effort. For example, you may notice that call volume has gone up by only 2.4% in the last quarter despite a 4% growth in the customer base. But is it all due to the customer support chatbot? That requires further analysis.

Retrospectives for downstream impact are meant to attribute observed improvements to the initiatives at play and to other factors. This is called contribution analysis. This is harder for Engineering to drive because they require all contributing initiatives, even those outside Engineering, to participate. They are best scheduled monthly or quarterly, convened by a business leader who has a stake in the downstream metric in question. Therefore, they might be a bridge too far, even for a reformist CTO. Nonetheless, you could still make sure that the measurements are in place for the retrospective to take place, should the business leader so choose.

For the sake of completeness, Figure 7 shows what the results of a downstream impact retrospective might look like for the example of the customer support chatbot.

It shows that call volumes only rose by 2.4% quarter-on-quarter despite a 4% growth in the customer base. The model assumes that if nothing else changes, the change in call volume should match the change in the customer base. We see a difference of 1.6 percentage points or 160 basis points. How do we explain this? Your data analysts might inform you that 60 bps is explained by seasonality. We credit the rest (100 bps) to self-service channels and ask them to claim their contributions. After a round of contribution analysis, you might arrive at the numbers at the bottom. You could use some heuristics and simple data analysis to arrive at this. I call it Simple Impact Attribution to contrast it with more rigorous methods (e.g., controlled experiments) that a data scientist might prefer but which might not always be feasible.

Figure 7: Example of Impact Attribution

Action #4: Offer your CFO/COO an alternative to ROI

These days, no one knows the ROI (return on investment) of an initiative. Projections made to win approval might not be in strict ROI terms. They might just say that by executing initiative X, some important metric would improve by 5%. It is not possible to determine ROI with just this information. But with the results of impact validation in place as above, you might be able to calculate the next best thing, the Return on Projection (ROP). If the said metric improved by 4% as against the projected 5%, the ROP, also called the benefits realization ratio, is 80%. Knowing this is way better than knowing nothing. It’s way better than believing that the initiative must have done well just because it was executed (delivered) correctly.

ROP is a measure of projection vs. performance. A tech CXO could encourage their COO/CFO to make use of ROP to make better investment decisions in the next round of funding. Asking for a thorough justification before funding is good, but they are based on assumptions. A projection is invariably embedded in the justification. If they only decide based on projections, it incentivizes people to make unrealistic projections. Business leaders may be tempted to outdo each other in making unrealistic projections to win investment (or resources like team capacity). After all, there is no way to verify later. That’s unless you have an impact intelligence framework in place. The book has more detail on how to aggregate and use this metric at a portfolio level. Note that we are not aiming for perfect projections at all. We understand product development is not deterministic. Rather, the idea is to manage demand more effectively by discouraging unrealistic or unsound projections. Discourage spray and pray.

Action #5: Equip Your Teams

It can feel lonely if you are the only senior exec advocating for greater impact intelligence. But you don’t have to run a lonely campaign. Help your delivery teams understand the big picture and rally them around to your cause. Help them appreciate that software delivery does not automatically imply business impact. Even feature adoption does not. Start by helping them understand the meaning of business impact in different contexts. I have found it useful to explain this with an illustration of a hierarchy of outcomes as in Figure 8 The ones at the top are closest to business impact. The lower-level outcomes might support or enable the higher-level outcomes, but we should not take that for granted. Impact intelligence is about tracking that the supposed linkages work as expected. When your teams internalize this hierarchy, they’ll be able to help you implement robust demand management even more. They’ll begin to appreciate your nudges to reduce measurement debt. They’ll start asking Product and business leaders about the business impact of functionality that was delivered.

Figure 8: A hierarchy of outcomes