The Super Bowl was Sunday and hundreds of millions tuned in for the game and of course the commercials. The game was excited, especially the 4th quarter, but as exciting as the game was the commercial breaks drove a different type of excitement. Last year almost 112 million watched in the US alone. For advertisers this can be a make or break opportunity.
According to multiple sources a 30 second commercial cost $5-5.5 million dollars leaving no room for errors. The ad story, copy, call to action, production quality, etc all has to be perfect to maximize the investment. The advertisement is used to drive eye balls to the brands, and convert viewers into paying customers. This means a company must have systems and infrastructure in place to provide a great user experience. The customer experience starts with the ad, but materializes either in the store or on the brand's web site.
The whole process is very scientific, but a single broken cog can cause the entire system to collapse. A poorly performing web site has many opportunities for a broken user experience. Slow web sites can destroy a company and brand in a matter of seconds.
35% will have a negative impression of the brand due to poor performance. 22% will never return to a slow site. About of a site's visitors will abandon the site if it takes longer than 3 seconds to load.
The majority of web traffic originates on mobile devices today, only amplifying the need to have a fast web experience. 33% of in person purchases are influenced by mobile experiences. 74% will Abandon a Page that takes longer than 5 seconds to load. 63% wont do business with a company after a bad mobile experience.
Web site performance is a key indicator if user satisfaction. Provide a slow experience and you will disenchant customers, maybe for good. Provide a great user experience they can be your fans for the rest of their life.
While many will critique the quality of the ads, we decided to critique the quality of each company's web site performance. We are going to pick a couple of winners and some big time losers when it comes to their web sites. There were 60 ads shown during the game. We chose to analyze 31 ads listed in a Washington Post article the Friday before the Super Bowl.
So who will spend $5,000,000 wisely and who is going to regret the decision?
How We Tested
Love2Dev performs detailed web performance audits and provides actions clients should take to improve their overall performance. The Super Bowl Ad comparison used a pair of WebPageTest.org runs for each company's home page for both desktop and mobile. When a super bowl ad directed traffic to a different site we evaluated that site.
We ran two different WebPageTest tests, one for desktop and one for mobile. We used Chrome for both tests, that way it should be the same browser platform on both platforms. For mobile we chose a Nexus 5 smart phone because we felt it was a fairly average phone, not too high end, not too low end. All tests were run from the Dulles Virginia location.
To keep the analysis simple we chose a few key performance indicators that highlight common performance issues we see sites make everyday. We are also going to offer some quick fixes you can apply to your site so you can perform with the super bowl crowd.
Key Performance Statistics
Time to First Byte
Steve Souders long ago stated the web performance golden rule is that 80% of your performance problems are in the client-side, not the server-side. He has since amended that statement to be 95/5 with the rise of mobile. Time to First Byte, or TTFB measures how long does it take your server to send the first bytes of your HTML to the browser. A good target time is 250 milliseconds or faster.
If average time to first byte is 500ms or greater then you need server-side optimizations. Common bottlenecks include poorly optimized database queries and web servers. We recommend using a NoSQL database or front-end caching service like MongoDB, Rediis or one of the cloud hosted services. You should also consider using a static web site and bypassing dynamic web servers like IIS/ASP.NET, Node/Express, etc. When using static pages there is no on-demand rendering that can slow down a response.
Since moving to a static site model we routinely see 100ms or faster response times from the sites we develop and manage. In fact it is not unusual to see 20-30ms response times. While your experience may vary, 100ms should be achievable.
The biggest loser in the time to first byte statistic is 84 Lumber. Their site takes a long time for both desktop and mobile. However our test took almost 35 seconds for the first bytes to register on mobile. Of course this is unacceptable for anyone. Comparing that time to the recent HTTPArchive data that places 84 Lumber in the bottom .1% of TTFB times.
84 Lumber paid the price during the game, their site and crashed hard.
You can see the drastic difference in this chart comparing all the pages TTFB values:
If it takes a long time to get the first bytes from the server, it triggers a cascade of poor performance since the browser cannot start to load the remaining assets till the markup is loaded.
Besides improving database queries and eliminating traditional web server infrastructure you can improve your TTFB by applying proper server caching. In . Other platforms have similar features. But the idea is you tell the server to create an in memory instance of the page as rendered the first time it is requested. You can control how long that page is cached and vary the value cached by queryString parameters and header values. When a web server pulls from memory it returns the asset much faster.
Disk I/O is very expensive, when a file is cached it is retrieved from memory, not the disk. This means the server hardware is less taxed. When you need less hardware you can save money on hosting and get more life out of the hardware. However today you should be using a cloud infrastructure, like we here at Love2Dev use.
Of course a last resort is to throw more hardware at the problem. The reason you need to add more servers is due to simultaneous request capacity. Remember browsers open 6 parallel connections over HTTP/1. Ethernet cards have limitations to the number of connections supported. The last time I checked it was about 65,000, but that was nearly 10 years ago.
Today with the cloud you can quickly scale a web site without extra hardware. In fact if you properly used a CDN to distribute rendered assets your site should 'just scale'.
A technical factor that hinders rendering is a poorly optimized critical rendering path. The critical rendering path refers to the logic flow a browser follows to render content.
A late start render time on desktop is exacerbated many times over on mobile. Mobile devices account for about 75% of web traffic these days. They are very underpowered compared to their desktop counterparts. This is why we chose to run a desktop and mobile test run for each site.
The average desktop page started rendering content 3.94 seconds, while the mobile versions averaged 3.36 seconds. At least mobile leaded reasonably fast due to sites doing a decent job implementing responsive images.
An early render time is critical to comforting your customer and engaging them with your message.
Speed Index is a metric created by Patrick Mehan. Patrick is the genius that created WebPageTest.org, which records a video of each page loading and evaluates how many pixels are rendered at different points in the entire loading cycle. The speed index is 'the area under the curve', which is a way of simplifying some calculus so you can tell how much of a page was rendered over time.
Love2Dev works to achieve a speed index around 1000, which indicates the page appeared to be rendered within 1 second (1000ms). The average super bowl advertiser's page scored AAA, with the best mobile score being GoDaddy at 1455. The worst score (non 84 Lumber) was earned by Audi at 19584. That correlates to roughly 19.5 seconds to render.
While not necessarily an indicator of slow rendering overall page weight is symptom of a poorly performing page. A page with many images but fewer scripts and CSS tends to load faster than a similarly sized page heavy with scripts & CSS. This is due to the critical rendering path issues I highlighted earlier.
While larger pages tend to load slower, the often overlooked issue is how much it costs the customer to load the page. 25% of the US connects to the Internet via cellular only. Coincidently 25% of the US also goes over their monthly data plan limits.
Large pages take longer to load, but cost the customer more money to load. Audi is the super bowl's worst offender at 89/56MB. This means it costs $3.24 in the US.
Fully Loaded Time
Finally, customers typically can't start interacting with your page until it is fully loaded. This is because the browser's UI thread is locked up with rendering tasks. It is loading all those scripts, fonts, images and stylesheets and not letting the user click the button or type their e-mail address. Take too long and the customer goes back to what they were doing or worse the next listing in the search results.
If you don't engage the customer within 3 seconds, half leave the page. Taking more than 3 seconds and you not only wasted your money advertising, but created someone that does not care for your brand. In fact 46% do not return to a slow site.
In our test, super bowl advertisers averaged a 14.6 seconds mobile load time. This means they lost most of the traffic generated by those expensive ad campaigns. 84 Lumber and Audi were the biggest losers.
Reduce your page weight and dependence on assets you are not using to reduce your load time and earn more customers.
Key Issues Hurting Super Bowl Advertisers
There are 4 other key web performance issues we saw super bowl advertisers scoring poorly. Each of these issues can also be quickly resolved, improving customer engagement statistics.
Not Applying Proper Cache Headers
I was shocked to see how many sites were not applying proper cache headers to their assets. In fact only 3 sites passed the static asset caching test.
Companies can see significant performance improvements by applying proper cache headers. Another upside to proper caching is reduced hosting expenses as a request that is never made cost you nothing. It also costs the customer less (see the costs referenced above).
A good Cache-Control header value is what we call a far future expires header. This typically refers to a month or more. We recommend a year for most assets, but you should evaluate your page's needs to apply a proper value.
This is how we set our standard Cache-Control header:
Insane 3rd Party Pollution
The average page loaded 128 files. Of course this is way more than they should. I doubt the web developers in these companies 'shipped' a site that loads anywhere close to that number. What happens is 3rd party script pollution. Simon Hearne has done some excellent research in the area of 3rd party scripts and how they destroy good user experiences.
The main problem is each 3rd party script you install tends to load additional 3rd and 4th party scripts. The process creates a snowball affect that can freeze your page in its tracks, causing customers to panic and leave.
The Budwiser home page had an insane amount of requests, 265. While not the most requests, their rival Busch has more, their speed index was over 19000! The page is a simple age verification and honestly requires about 10-15 files (mostly due to pixel tracking etc). The page should render in about 500-1000ms on desktops. At 19000 they serve as a poster child of unnecessary 3rd party scripts.
We recommend every company audit their use of 3rd party scripts each month. Many companies use tag managers like Google or Tea Leaf to allow marketing department types to blinding add scripts to their sites. There is usually no technical vetting of these scripts and their impact.
To compound the problem further many scripts are added, but never used. I mean they are obviously used, but the data they collect and report are never used. They are sort of like the many 'temporary taxes' created by politicians that never seem to go away. They are added one day to see what they offer, but soon forgotten and never removed.
A good devops team should have a monthly 3rd party script audit. Each script should be matched to the owner and real usage verified. You should also perform static and real user measurement performance tests to see what impact each script has on your user experience.
Let me be frank, most of the time the service offered by most 3rd party scripts offers little to no value to begin with. So think hard before you add their code to your site.
Not Optimizing Images
13 of the 31 sites tested passed the WebPageTest image optimization test. Again this is not unusual, but is easily rectified. Photos taken with cameras or saved from Photoshop and other graphic editors are not optimized for the web. You should have a step in your publishing workflow that automatically optimizes images or removes the unnecessary bytes. This can be done without loss of visual details.
We include an optimization process as part of all of our web sites. Not only does our process optimize images, it also creates a responsive image array. Instead of placing a classic image tag in the markup, our sites automatically include the image srcset and sizes attributes for proper responsive images.
Optimizing images can cut page weight by as much as 50% depending on the page asset composition. This can go a long way to faster rendering times.
From what we could tell the sites did not use HTTP/2. This is a shame because HTTP/2 offers many performance benefits. Most server platforms support HTTP/2 today. However site administrators still need to turn it on.
HTTP/2 advances the HTTP protocol by adding many performance features, like request multiplexing. All modern browsers support HTTP/2 and the protocol is backward compatible with HTTP/1. So there is no reason to not turn HTTP/2 on.
HTTP/2 does require HTTPS, which the sites we tested did a good job implementing. Hopefully they will enable HTTP/2 soon. All Love2Dev sites have HTTP/2 enabled and since doing so was have seen noticeable performance gains.
What Are Super Bowl Advertisers Doing Right
Now that I have covered what the advertisers did wrong, let's talk about what they did right.
Time to First Byte
Despite 84 Lumber's issues, the typical time to first byte was less than 250ms. This represents an ideal time frame to get a page load started. However, since our sites routinely see 100ms or faster response times we feel these large brands could also enjoy that speed. The fast TTFB times mean the sites we tested did a good job configuring their server-side processes.
Using a CDN
We were happy to see the advertisers properly utilizing content delivery networks. A CDN helps cache your web content closer to the visitor. This helps your site load faster as the files do not have as far to travel.
While the Internet is 'fast', speed is still limited by the speed of light (how fast the bytes can travel across the wire). So physics dictate a shorter distance does not require as much time. Even having a server on each US coast can be invaluable. This is why Love2Dev uses a CDN for all web site assets.
No One Uses Flash!!!!
This is huge!!!! Flash is being killed by all platforms. Of course it started when Steve Jobs announced iOS would not support Flash. Today browsers are eliminating Flash because it is a huge security hole and of course eats battery's for breakfast. HTML5 and CSS can now create better, more performant user experiences than Flash can. The one area Flash still has a purpose is live streaming video. But those days are numbered too.
Most of the sites use HTTPS instead of HTTP. In the past HTTPS was slower and recommended only for e-commerce and vital statistics pages (think social security number). Today however this changes. HTTPS causes browsers to turn on modern features. For example you cannot use HTTP/2 unless the size uses HTTPS. You cannot register a service worker and other progressive web application features without a certify certificate.
Browser are going to start marking certain sites as insecure if they are not HTTPS. Search engines are also using HTTPS as a search engine ranking signal. So every site should have SSL implemented. In the past this was expensive, but today you can create certificates for free with some technical know how.
All Love2Dev sites are HTTPS. We also provide professional SSL certificates starting at $69/year.
Winners and Losers
First the losers. We think Audi and 84 Lumber are the biggest losers. Audi has a ridiculously heavy page that renders slowly and costs customers over $3 to load on their mobile devices. 84 Lumber did not properly configure their server infrastructure to serve their home page, for both the Journey84.com and 84Lumber.com sites.
The winners are Mr. Clean, KFC and Buick. Both apply many web performance best practices. As a result their pages load relatively quick on both mobile and desktop. This should go a long way to engaging customers and ensuring they have a positive return on their super bowl ad investments.
Even if you are not investing millions of dollars in super bowl advertising you cannot afford to waste money driving potential customers to a slow web page. You need to convert those visitors into customers and better yet fans for life. Having a fast site that engages those customers quickly is a key part of building your brand relationships.
Love2Dev offers detailed web performance audits for businesses of all sizes. Audits start at $499 and have a 3 business day turn around. You will receive a detailed report covering over 100 web performance metrics. You will know how you fare against the best web sites, but we will also audit some of your closest competition and provide a strategic comparison analysis report. You will know where your performance strengths and weaknesses are.
The audit includes action items you and your team can take to achieve maximum web performance. Many items are what we call 'low hanging fruit' because they are easy to implement and can be done often within minutes.
Customers love fast sites, but so do search engines. Sites that load and render faster rank better in search engines. If you are looking for the extra edge to improve your search engine rankings, spending a few hours improving your site's performance could be all it takes to go from page 5 to a top ranking. This means more business and profits.