Web applications are typically a complex mix of infrastructure, platforms, services and content. Unlike the mainframe-based applications or LAN client-server applications of the past, no single, simple set of metrics had existed to measure the performance of online applications.

I had an opportunity this week to discuss this issue with four web operations experts at a Coradiant round-table discussion series. Each participant runs very large sites, and each is a clear leader in their respective industries. These four IT executives represented very different industries: A pharmaceutical Software-as-a-Service (SaaS) provider, the highest volume online financial services organization, a regional healthcare provider, and the leading travel industry search site.

Ultimately, the goal is to overcome the visibility gap between IT executives and their web applications. And what’s clear is that watching user experience provides these organizations with a single tool set to measure online performance, find problems and govern IT effectiveness at the senior executive level of the organization. We’ve been working on technology to watch user experience since 2000, and it’s great to see this approach taking hold.

Each of these IT executives uses elements from four major groups of tools, but it’s clear that our User Performance Management (UPM) now gives IT a single, comprehensive capability to perform their jobs well. Because of this — and because UPM focuses on the user- and business-level perspective — it is quickly becoming the prevalent view of web operations health throughout their organizations.

There are four main ways to look at online performance:

• Platform Management: Monitoring the health of hardware platforms and infrastructure that applications run on
• Synthetic Testing: Repeated tests of certain user-initiated web processes using automated test scripts
• Web Analytics: Collecting information to analyze the source of visits, site navigation and buying trends
• User Performance Management: Monitoring actual end user traffic to identify errors and problems and to measure delivered performance

Web analytics and User Performance Management are based on data from user transactions, while synthetic testing and platform management use tests and platform metrics. Web analytics and synthetic tests are often the domain of marketing, while User Performance Management and platform management are more likely to be used by operational teams.

Let’s look at each in detail.

Platform management

The inherent complexity of modern applications means that there is less and less relationship between actual user experience and the health of platforms. A server can be unavailable—but load-balancers may hide the problems so that users aren’t affected. Similarly, a network can be forwarding packets perfectly—but users are getting content errors.

Server logs provide huge chunks of information that is difficult to correlate to actual user problems. Finding problems based on these logs often takes days of complex effort on the part of experts.

Platform monitoring tools are increasingly employed for diagnostic purposes and forensic root cause investigations. Platform monitoring is a necessary, but not sufficient, part of a web monitoring strategy.

Watching platforms tells you whether components functioned; but User Performance Management tells you whether the whole system delivered and what the resulting user experience was.

Web analytics

Web Analytics measures the effectiveness of online campaigns and web conversions. Marketing organizations use web analytics to optimize the process of converting viewers into buyers.

Modern web analytics has evolved into a powerful set of page-tagging and navigational analysis tools integrated into content management systems. Because it is not economical for web operators to capture and store the tremendous amount of information that collection can generate, analytics is most often delivered through a hosted service.

Web analytics shows purchases, lead sources, and search effectiveness, but doesn’t show whether poor performance affects conversions. Failed transactions often go unnoticed.

Analytics shows who did buy; but User Performance Management shows whether they could.

Synthetic testing

Synthetic testing gives an estimate of application performance. Synthetic testing uses scripts to provide a rough baseline to compare performance over time and to compare to competitors. They also act as a “sanity check” when a user complains of performance from a particular region.

Because of their repeatability synthetic testing has been particularly appealing to marketing organizations. Unfortunately, it has also lulled companies into a false sense of security. Modern, dynamic web applications generate unique pages for every visitor, and many of them are one-time transactions. Most organizations answer confidently when someone asks, “is your site working?” But they are far less certain when asked, “is your site broken?”—because someone, somewhere, may be having an error on a page that isn’t being tested.

Synthetic testing shows whether a site is working; but User Performance Management shows whether it is broken.

User Performance Management

Watching real users allows web operators to detect every error as it happens. More importantly, capturing users’ sessions means that there is a record of the failure. This makes it dramatically easier to reproduce, diagnose, and resolve the problem. User Performance Management also means organizations know what each user’s experience was like, as well as how the site really performs under production conditions.

This addresses some of the biggest challenges the panelists had:

• Assuring and effectively reporting service levels
• Finding and fixing problems quickly
• Knowing the effect of changes on their end users
• Real-time visibility into online user experience

Four essential elements

All four monitoring technologies are employed to run web applications. But it is clear that User performance Management gives web operators the ability to deliver fast, error-free, available applications and communicate IT effectiveness with the rest of the business.