The browser cache sucks. It barely knows about the context, what really ought to be cached and what is safe to expire. Plus the client doesn’t have much control on the cache. And the browser cache only caches resources. Not fragments.

localStorage is awesome as a flexible cache. Especially for fragments, see my blog post: http://00f.net/2010/09/22/transparent-client-side-fragment-cache/

Let’s use it, it’s just perfect for this kind of task. The only thing to fear of is what will happen once marketing companies realize how powerful it is.

To paraphrase: When distributing load time among categories, how are the following cases handled?
– Time ‘gaps’ where the client is processing JS, but nothing is being downloaded
– Multiple objects being downloaded in parallel

First, let me provide some additional context for the problem space. The focus of this type of reporting is for operational monitoring, as opposed to pure performance measurement. So the goal is quick problem identification, isolation, and resolution.

The characteristics of time ‘gaps’ due to client JS processing are dependent on the available resources on the client machine, which can vary greatly depending on the type of computer and what other things are happening on it. Since the focus of operational monitoring is on the delivery of content to the user, these client-side processing gaps are ignored in the data. So, both the category times and page load times do not include these gaps.

When multiple objects are loaded in parallel, the time is shared among all the objects loading at any moment in time. For example, if the load time of a page is divided into 10ms slices, and 5 objects load in parallel during a particular time slice, then each object is assigned 2ms of time (10 / 5) for that slice. Those times are summed for each object (or category) to determine the contribution to page load time.

This approach fulfills the goals above in helping understand the distribution of load time among categories (or domains), and quickly identifying drivers of a performance change. However, it is not intended as a development tool to estimate time savings for removing content. If page content is changed, the load behavior will change (the order objects are retrieved, the amount of parallelism, etc.), so the distribution of time to the various categories (or domains) will also change.

-Eric

There’s a paper that describes this:

http://home.pacbell.net/ciemo/statistics/WhatDoYouMean.pdf

There’s also a question about whether you should run IQR filtering on the data before analysing it or not. IQR filtering tends to get rid of outliers, which in the case of web performance data applies mainly to excessively high numbers that come out of DNS misses (ie, an ISP’s first DNS server failed and the second responded), which isn’t something you can control, though you may want to know about it any way.

I’ll be speaking about this at confoo.ca in Montreal in March if you’re interested.

Really a very useful addition to Webpagetest.
I hope he can think of a way to build this into the browser plugin version, but that’s probably wishful thinking.

Several of the core participants in the best practices working group are now involved in an Ad Performance Scoring Working Group, trying to develop a method of measuring and scoring the load performance of ads, from both qualitative and quantitative perspectives.

As far as research into the impact of slow ads on ad effectiveness – within the IAB that type of work would generally be performed by the Research Council. This idea has been discussed as a project for the Research Council, but I not aware of active work in this area.

Having an ad scoring method would be a good foundation to build from for that type of research.

-Eric

Do you know if the IAB has done any work in this area since the release of the Best Practices? For instance: research on the impact of slow ads on ad effectiveness.

thanks,
Aaron Peters

Another tenet per #3 is is the old adage of turning metrics into information, specifically useful information to those that can make a difference. I’ve always found two avenues for this:

1. Recognize metrics you observe that important to the business and communicate (per your “so what” example).

2. Recognize “pain” being experienced by the business/executives, and root out the appropriate metric to improve and follow. In the old CompuServe days, the “pain” was the VP of Ops being pinged whenever other executives couldn’t load the CompuServe Main Menu! The metric ended up being availability, which led to a whole treasure trove of other metrics (Capacity, publishing, server metrics…)

Mark

The next step after calculating the revenue impact would be to calculate an operational support cost for addressing the incident. This includes:

- On call time spent on addressing the issue
- NOC attention to the issue
- Additional monitoring/tools established to detect in the future
- Post Mortem meetings/attention
- Future dev/process work needed to fix the bug/root cause

In an ideal world, if adequate costing information exists, you should be able to correlate reduced TCO for operational support with improved Availability, in addition to improved customer sat and revenue.

Mark