There are some important differences between managing a solutions vs a platform product, in terms of providing product value. One of the key differences is in how you determine and track valuable key performance indicators (KPIs).

This post focuses on the differences for managing KPIs not just for platforms overall, but for content management systems – products used by web sites to build their content, which include the tools that editors use to publish content to the site.

There are lots and lots of helpful articles on what KPIs are, and how to best determine which KPIs are best for your business (including this one, this one, and this one). So here is just a quick overview of what KPIs are and how you can go about choosing them.

How to Choose Your KPIs

KPIs are a set of measurements meant to indicate whether or not a product is successful. Because so many other articles have written on this subject, this is a quick (oversimplified) plan for one way you could go about choosing KPIs from your product.

This plan borrows ideas from the product innovation model used by Netflix (see its blog post here) and many other product companies to quickly determine success for product innovations. (Remember: once you choose your KPIs, you should stick with them for a while! This could be a good way to at least get you started.)

Plan:

1. Start with your goals. Don’t try to paste another product’s goals onto your product. Your product and business is unique.
2. Come up with a hypothesis for what metrics might allow you to prove that you’re delivering value, according to your goals.
3. Brainstorm ways to cheaply implement a test for whether or not the metrics you chose will consistently test for value.
4. Repeat the test until you find metrics that work for your product.
   1. Remember: measure metrics that reliably reveal cause and effect. See more on this important point here.
   2. Measure only the most important things. Keep the list small (try three items per category).
5. Once you’ve found metrics that reliably test whether your product is providing value, track those metrics over time to see if your product efforts improve (or maintain) success.
6. Re-check every so often that your KPIs are still the right indicators of your product’s success.

KPIs for Solutions Products vs Platform Products

Okay! That’s done. Moving onto the differences between solutions products and platform/CMS KPIs.

Because a [solutions] product is something you sell and a platform is common infrastructure that you build products on, their goals and therefore the ways they define success are, naturally, pretty different.

For instance, solutions products often focus on revenue growth and customer growth as key drivers of product success, so they are measured via KPIs such as increased MRR (monthly recurring revenue added in a given month) or the rate at which prospects are converted to customers. See here for more metrics ideas and suggestions.

Platform/CMS products are not usually directly involved in generating revenue, so many of the KPIs that solutions product managers might measure are not relevant. That, however, does not mean that platform or CMS products do not have an impact on a business’s bottom line.

So what are some good KPIs for a CMS product manager to think about? For our product, we break our goals into four categories: financial goals, systems goals, user (customer) goals, and innovation goals.

Financial Goals

Because I work for a media company with thousands of editors producing content for websites, one of the goals of the CMS is to drive efficient production or publication of the words, images, videos, and other media that make up the site’s content.

Some KPIs for increasing efficiency of content production include:

• Percentage decrease in time-to-publish
• Reduction in cost (in employee hours) to program pages
• Reduction in cost (in employee hours) maintaining pages and doors
• Increased number of newly published content items per week/month

Also, CMSes (particularly enterprise CMses) are often implemented to save costs – i.e. in order to not have to pay vendors to handle tasks the company can handle on its own, and to drive efficiency.

A simple way to test whether the CMS is saving costs is to determine the cost of a hosted solution vs the cost of the staff and third-party services. Of course, this cost must also be set against the opportunity cost of allowing for greater custom innovation (see below).

Reporting these KPIs to stakeholders who are funding the CMS, or another enterprise platform, keeps them up-to-date on the value of their financial investment.

Systems Goals

Much of what matters to the users of the CMS and to the business depends on the efficiency of the platform.

A CMS product team must work with the engineering management team to quantify data quality, integration efficiency, infrastructure reliability, and other systems-related concerns.

To drive efficiency, a CMS product manager may work with an engineering manager to determine the metrics such as the following for the following categories:

• Data quality
  • Percentage decrease in reported bugs
  • Percentage increase in specific types of log errors
• Release efficiency
  • Decrease in number of bugs created in QA or after release
  • Decrease in integration costs (in engineering hours) spent on releases
• Reliability
  • Percentage decrease in time outages occur
  • Percentage decrease in MTBF (mean time between failures)
  • Percentage decrease in MTTR (mean time to repair ) – i.e. consider which failures matter most to users and stakeholders and measure how often they happen, and how long they take to be repaired

Working with engineering management to determine and report on KPIs is a vital piece of a CMS or platform product manager’s role.

Innovation Goals

Another key reason to use an enterprise CMS is to create custom solutions to support innovative content.

To measure whether or not a CMS is enabling greater innovation, the product management team can measure the percentage increase in the revenues from custom features vs. the cost of producing the feature.

Also, if the goal is that the CMS allows readers on the end site to experience more innovative content, this article: ‘Ready-to-use KPIs to Track Content Management Performance’ suggests that product managers can “measure reader’s attitude to published content” – i.e. intentionally track reader sentiment or behavior related to custom features enabled by the CMS.

User (Customer) Goals

As Jody Roberts says, we “can’t talk about measuring the success of a CMS, without talking about measuring the success of that CMS’ consumers”. As with all products, the users of that product should be satisfied (or more) with the product. CMS product managers can use simple surveys to measure customer temperature tests over time: are they unhappy, neutral, happy, or very happy with your product?

Then, of course, there’s the handy Net Promoter Score. Even in the context of an internal CMS, product managers can ask users whether they’d recommend the CMS to new employees (or would encourage the business to continue funding the product).

Conclusion

Though the definition of success might be different for solutions vs platform products, a CMS, like any product, needs to provide value for its users (customers) and the business, and be measured against metrics that prove its value.

To do so, a product manager must understand their goals, work with their engineering counterparts and stakeholders to determine the right KPIs to track those goals, and then continually review whether or not the KPIs are really enabling them to report valuable insight on the success of the product.

Whether your platform goals are to increase efficiency, increase data quality, or delight users, it’s key to think of KPIs that will track and show value for your unique product.