The
biggest mistake you can make in UX design is to launch a website or app
and then walk away. Monitoring and improving after launch will benefit
users and your metrics. Nothing will bring bigger gains.
Whether
you’ve launched a redesign of your website or rolled out a new feature
in your app, that is the point where people normally move on to the next
project. But, that is a mistake.
It’s only once a site, app, or
feature goes live that we get to see actual users interacting with it in
a completely natural way. It’s only then that we know if it has
succeeded or failed.
Not that things are ever that black and
white. Even if it does seem successful, there’s always room for
improvement. This is particularly true with conversion rate optimization. Even small tweaks can lead to significant increases in revenue, leads, or other key metrics.
Failing
that, don’t be afraid to use friends and family as in most cases
getting the exact demographic is less important than you might think. As
long as people have comparable physical and cognitive abilities, you
shouldn’t have a problem. The only exception is if the content of your
website or app is highly specialized.
That said, I would avoid
using anybody who works for the organization. They will inevitably be
institutionalized and unable to provide unbiased feedback.
Whatever
approach you use to test your solution, once you’re happy, you can push
that change live for all users. But, your work is still not done.
Rinse And Repeat
Once
you’ve solved one issue, return to your analytics. Find the next
biggest problem. Repeat the whole process. As you fix some problems,
more will become apparent, and so you’ll quickly find yourself with an
ongoing program of improvements that can be made.
The more you
carry out this kind of work, the more the benefits will become obvious.
You will gradually see improvements in metrics like engagement,
conversion, and user satisfaction. You can use these metrics to make the
case to management for ongoing optimization. This is better than the
trap of releasing feature after feature with no regard for their
performance.
Do
perfect Lighthouse scores mean the performance of your website is
perfect? As it turns out, Lighthouse is influenced by a number of things
that can be manipulated and bent to make sites seem more performant
than they really are, as Salma Alam-Naylor demonstrates in several
experiments.
This article has been kindly supported by our dear friends at Sentry.io, who help developers see what actually matters, solve quicker, and learn continuously about their apps. Thank you!
Google
Lighthouse has been one of the most effective ways to gamify and
promote web page performance among developers. Using Lighthouse, we can
assess web pages based on overall performance, accessibility, SEO, and
what Google considers “best practices”, all with the click of a button.
We
might use these tests to evaluate out-of-the-box performance for
front-end frameworks or to celebrate performance improvements gained by
some diligent refactoring. And you know you love sharing screenshots of
your perfect Lighthouse scores on social media. It’s a well-deserved
badge of honor worthy of a confetti celebration.
Just
the fact that Lighthouse gets developers like us talking about
performance is a win. But, whilst I don’t want to be a party pooper, the
truth is that web performance is far more nuanced than this. In this
article, we’ll examine how Google Lighthouse calculates its performance
scores, and, using this information, we will attempt to “hack” those
scores in our favor, all in the name of fun and science
— because in the end, Lighthouse is simply a good, but rough guide for
debugging performance. We’ll have some fun with it and see to what
extent we can “trick” Lighthouse into handing out better scores than we
may deserve.
Local
performance testing is a great way to understand if your website
performance is trending in the right direction, but it won’t paint a
full picture of reality. The World Wide Web is the Wild West, and
collectively, we’ve almost certainly lost track of the variety of device
types, internet connection speeds, screen sizes, browsers, and browser
versions that people are using to access websites — all of which can
have an impact on page performance and user experience.
Web performance is more than a single core web vital metric or Lighthouse performance score. What we’re talking about goes way beyond the type of raw data we’re working with.
Speed
is often the first thing that comes up when talking about web
performance — just how long does a page take to load? This isn’t the
worst thing to measure, but we must bear in mind that speed is probably
influenced heavily by business KPIs and sales targets. Google released a report in 2018
suggesting that the probability of bounces increases by 32% if the page
load time reaches higher than three seconds, and soars to 123% if the
page load time reaches 10 seconds. So, we must conclude that converting
more sales requires reducing bounce rates. And to reduce bounce rates,
we must make our pages load faster.
But what does “load
faster” even mean? At some point, we’re physically incapable of making a
web page load any faster. Humans — and the servers that connect them —
are spread around the globe, and modern internet infrastructure can only
deliver so many bytes at a time.
The bottom line is that page load is not a single moment in time. In an article titled “What is speed?” Google explains that a page load event is:
[…]
“an experience that no single metric can fully capture. There are
multiple moments during the load experience that can affect whether a
user perceives it as ‘fast’, and if you just focus solely on one, you
might miss bad experiences that happen during the rest of the time.”
The key word here is experience. Real web performance is less about numbers and speed than it is about how we experience
page load and page usability as users. And this segues nicely into a
discussion of how Google Lighthouse calculates performance scores. (It’s
much less about pure speed than you might think.)
How Google Lighthouse Performance Scores Are Calculated #
The
Google Lighthouse performance score is calculated using a weighted
combination of scores based on core web vital metrics (i.e., First
Contentful Paint (FCP), Largest Contentful Paint (LCP), Cumulative
Layout Shift (CLS)) and other speed-related metrics (i.e., Speed Index
(SI) and Total Blocking Time (TBT)) that are observable throughout the page load timeline.
The highest weighted metric is Total Blocking Time (TBT), a metric that looks at the total time after the First Contentful Paint (FCP)
to help indicate where the main thread may be blocked long enough to
prevent speedy responses to user input. The main thread is considered
“blocked” any time there’s a JavaScript task running on the main thread
for more than 50ms. Minimizing TBT ensures that a web page responds to
physical user input (e.g., key presses, mouse clicks, and so on).
2. A Web Page Should Load Useful Content With No Unexpected Visual Shifts #
The next most weighted Lighthouse metrics are Largest Contentful Paint (LCP) and Cumulative Layout Shift (CLS). LCP marks the point in the page load timeline when the page’s main content has likely loaded and is therefore useful.
At
the point where the main content has likely loaded, you also want to
maintain visual stability to ensure that users can use the page and are
not affected by unexpected visual shifts (CLS). A good LCP score is
anything less than 2.5 seconds (which is a lot higher than we might have
thought, given we are often trying to make our websites as fast as possible).
The First Contentful Paint (FCP) metric marks the first point in the page load timeline where the user can see something on the screen, and the Speed Index (SI) measures how quickly content is visually displayed during page load over time until the page is “complete”.
Your page is scored based on the speed indices of real websites using performance data from the HTTP Archive.
A good FCP score is less than 1.8 seconds and a good SI score is less
than 3.4 seconds. Both of these thresholds are higher than you might
expect when thinking about speed.
Google Lighthouse’s performance scoring is, without a doubt, less about speed and more about usability.
Your SI and FCP could be super quick, but if your LCP takes too long to
paint, and if CLS is caused by large images or external content taking
some time to load and shifting things visually, then your overall
performance score will be lower than if your page was a little slower to
render the FCP but didn’t cause any CLS. Ultimately, if the page is
unresponsive due to JavaScript blocking the main thread for more than
50ms, your performance score will suffer more than if the page was a
little slow to paint the FCP.
To understand more about how the
weightings of each metric contribute to the final performance score, you
can play about with the sliders on the Lighthouse Scoring Calculator,
and here’s a rudimentary table demonstrating the effect of skewed
individual metric weightings on the overall performance score, proving
that page usability and responsiveness is favored over raw speed.
Description
FCP (ms)
SI (ms)
LCP (ms)
TBT (ms)
CLS
Overall Score
Slow to show something on screen
6000
0
0
0
0
90
Slow to load content over time
0
5000
0
0
0
90
Slow to load the largest part of the page
0
0
6000
0
0
76
Visual shifts occurring during page load
0
0
0
0
0.82
76
Page is unresponsive to user input
0
0
0
2000
0
70
The
overall Google Lighthouse performance score is calculated by converting
each raw metric value into a score from 0 to 100 according to where it
falls on its Lighthouse scoring distribution, which is a log-normal
distribution derived from the performance metrics of real website
performance data from the HTTP Archive. There are two main takeaways
from this mathematically overloaded information:
Your Lighthouse performance score is plotted against real website performance data, not in isolation.
Given
that the scoring uses log-normal distribution, the relationship between
the individual metric values and the overall score is non-linear,
meaning you can make substantial improvements to low-performance scores
quite easily, but it becomes more difficult to improve an already high
score.
I
appreciate Google’s focus on usability over pure speed in the web
performance conversation. It urges developers to think less about aiming
for raw numbers and more about the real experiences we build. That
being said, I’ve wondered whether today in 2024, it’s possible to fool
Google Lighthouse into believing that a bad page in terms of usability and usefulness is actually a great one.
I put on my lab coat and science goggles to investigate. All tests were conducted:
Using the Chromium Lighthouse plugin,
In an incognito window in the Arc browser,
Using the “navigation” and “mobile” settings (apart from where described differently),
By me, in a lab (i.e., no field data).
That
all being said, I fully acknowledge that my controlled test environment
contradicts my advice at the top of this post, but the experiment is an
interesting ride nonetheless. What I hope you’ll take away from this is
that Lighthouse scores are only one piece — and a tiny one at that — of
a very large and complex web performance puzzle. And, without field
data, I’m not sure any of this matters anyway.
TL;DR:
Show the smallest amount of LCP-qualifying content on load to boost the
FCP and LCP scores until the Lighthouse test has likely finished.
FCP marks the first point in the page load timeline where the user can see anything
at all on the screen, while LCP marks the point in the page load
timeline when the main page content (i.e., the largest text or image
element) has likely loaded. A fast LCP helps reassure the user that the page is useful. “Likely” and “useful” are the important words to bear in mind here.
The types of elements on a web page considered by Lighthouse for LCP are:
<img> elements,
<image> elements inside an <svg> element,
<video> elements,
An element with a background image loaded using the url() function, (and not a CSS gradient), and
Block-level elements containing text nodes or other inline-level text elements.
The following elements are excluded from LCP consideration due to the likelihood they do not contain useful content:
Elements with zero opacity (invisible to the user),
Elements that cover the full viewport (likely to be background elements), and
Placeholder images or other images with low entropy (i.e., low informational content, such as a solid-colored image).
However,
the notion of an image or text element being useful is completely
subjective in this case and generally out of the realm of what machine
code can reliably determine. For example, I built a page containing nothing but a <h1> element where, after 10 seconds, JavaScript inserts more descriptive text into the DOM and hides the <h1> element.
Lighthouse
considers the heading element to be the LCP element in this experiment.
At this point, the page load timeline has finished, but the page’s main
content has not loaded, even though Lighthouse thinks it is likely
to have loaded within those 10 seconds. Lighthouse still awards us with
a perfect score of 100 even if the heading is replaced by a single
punctuation mark, such as a full stop, which is even less useful.
This
test suggests that if you need to load page content via client-side
JavaScript, we‘ll want to avoid displaying a skeleton loader screen
since that requires loading more elements on the page. And since we know
the process will take some time — and that we can offload the network
request from the main thread to a web worker so it won’t affect the TBT —
we can use some arbitrary “splash screen” that contains a minimal
viable LCP element (for better FCP scoring). This way, we’re giving
Lighthouse the impression that the page is useful to users quicker than it actually is.
All
we need to do is include a valid LCP element that contains something
that counts as the FCP. While I would never recommend loading your main
page content via client-side JavaScript in 2024 (serve static HTML from a
CDN instead or build as much of the page as you can on a server), I
would definitely not recommend this “hack” for a good user experience,
regardless of what the Lighthouse performance score tells you. This
approach also won’t earn you any favors with search engines indexing
your site, as the robots are unable to discover the main content while
it is absent from the DOM.
I also tried this experiment with a
variety of random images representing the LCP to make the page even less
useful. But given that I used small file sizes — made smaller and
converted into “next-gen” image formats using a third-party image API to
help with page load speed — it seemed that Lighthouse interpreted the
elements as “placeholder images” or images with “low entropy”. As a
result, those images were disqualified as LCP elements, which is a good
thing and makes the LCP slightly less hackable.
View the demo page and use Chromium DevTools in an incognito window to see the results yourself.
This
hack, however, probably won’t hold up in many other use cases. Discord,
for example, uses the “splash screen” approach when you hard-refresh
the app in the browser, and it receives a sad 29 performance score.
Compared
to my DOM-injected demo, the LCP element was calculated as some content
behind the splash screen rather than elements contained within the
splash screen content itself, given there were one or more large images
in the focussed text channel I tested on. One could argue that
Lighthouse scores are less important for apps that are behind
authentication anyway: they don’t need to be indexed by search engines.
There
are likely many other situations where apps serve user-generated
content and you might be unable to control the LCP element entirely,
particularly regarding images.
For example, if you can control the
sizes of all the images on your web pages, you might be able to take
advantage of an interesting hack or “optimization” (in very large quotes) to arbitrarily game the system, as was the case of RentPath. In 2021, developers at RentPath managed to improve their Lighthouse performance score by 17 points when increasing
the size of image thumbnails on a web page. They convinced Lighthouse
to calculate the LCP element as one of the larger thumbnails instead of a
Google Map tile on the page, which takes considerably longer to load
via JavaScript.
The bottom line is that you can gain higher
Lighthouse performance scores if you are aware of your LCP element and
in control of it, whether that’s through a hack like RentPath’s or mine
or a real-deal improvement. That being said, whilst I’ve described the
splash screen approach as a hack in this post, that doesn’t mean this
type of experience couldn’t offer a purposeful and joyful experience.
Performance and user experience are about understanding what’s happening
during page load, and it’s also about intent.
TL;DR: Defer loading content that causes layout shifts until the Lighthouse test haslikelyfinished
to make the test think it has enough data. CSS transforms do not
negatively impact CLS, except if used in conjunction with new elements
added to the DOM.
CLS is measured on a decimal scale; a
good score is less than 0.1, and a poor score is greater than 0.25.
Lighthouse calculates CLS from the largest burst of unexpected layout
shifts that occur during a user’s time on the page based on a
combination of the viewport size and the movement of unstable elements
in the viewport between two rendered frames. Smaller one-off instances
of layout shift may be inconsequential, but a bunch of layout shifts
happening one after the other will negatively impact your score.
If
you know your page contains annoying layout shifts on load, you can
defer them until after the page load event has been completed, thus
fooling Lighthouse into thinking there is no CLS. This demo page I created,
for example, earns a CLS score of 0.143 even though JavaScript
immediately starts adding new text elements to the page, shifting the
original content up. By pausing the JavaScript that adds new nodes to
the DOM by an arbitrary five seconds with a setTimeout(), Lighthouse doesn’t capture the CLS that takes place.
This other demo page
earns a performance score of 100, even though it is arguably less
useful and useable than the last page given that the added elements pop
in seemingly at random without any user interaction.
Whilst
it is possible to defer layout shift events for a page load test, this
hack definitely won’t work for field data and user experience over time
(which is a more important focal point, as we discussed earlier). If we
perform a “time span” test in Lighthouse on the page with deferred
layout shifts, Lighthouse will correctly report a non-green CLS score of
around 0.186.
If
you do want to intentionally create a chaotic experience similar to the
demo, you can use CSS animations and transforms to more purposefully
pop the content into view on the page. In Google’s guide to CLS,
they state that “content that moves gradually and naturally from one
position to another can often help the user better understand what’s
going on and guide them between state changes” — again, highlighting the
importance of user experience in context.
On this next demo page, I’m using CSS transform to scale() the text elements from 0 to 1
and move them around the page. The transforms fail to trigger CLS
because the text nodes are already in the DOM when the page loads. That
said, I did observe in my testing that if the text nodes are added to
the DOM programmatically after the page loads via JavaScript and then animated, Lighthouse will indeed detect CLS and score things accordingly.
The
Speed Index score is based on the visual progress of the page as it
loads. The quicker your content loads nearer the beginning of the page
load timeline, the better.
It is possible to do some hack to trick the Speed Index into thinking a page load timeline is slower
than it is. Conversely, there’s no real way to “fake” loading content
faster than it does. The only way to make your Speed Index score better
is to optimize your web page for loading as much of the page as
possible, as soon as possible. Whilst not entirely realistic in the web
landscape of 2024 (mainly because it would put designers out of a job),
you could go all-in to lower your Speed Index as much as possible by:
Delivering static HTML web pages only (no server-side rendering) straight from a CDN,
Avoiding images on the page,
Minimizing or eliminating CSS, and
Preventing JavaScript or any external dependencies from loading.
TBT
measures the total time after the FCP where the main thread was blocked
by JavaScript tasks for long enough to prevent responses to user input.
A good TBT score is anything lower than 200ms.
JavaScript-heavy
web applications (such as single-page applications) that perform complex
state calculations and DOM manipulation on the client on page load
(rather than on the server before sending rendered HTML) are prone to
suffering poor TBT scores. In this case, you could probably hack your
TBT score by deferring all JavaScript until after the Lighthouse test
has finished. That said, you’d need to provide some kind of placeholder
content or loading screen to satisfy the FCP and LCP and to inform users
that something will happen at some point. Plus, you’d have to
go to extra lengths to hack around the front-end framework you’re using.
(You don’t want to load a placeholder page that, at some point in the
page load timeline, loads a separate React app after an arbitrary amount
of time!)
What’s interesting is that while we’re still doing all
sorts of fancy things with JavaScript in the client, advances in the
modern web ecosystem are helping us all reduce the probability of a
less-than-stellar TBT score. Many front-end frameworks, in partnership
with modern hosting providers, are capable of rendering pages and
processing complex logic on demand without any client-side JavaScript.
While eliminating JavaScript on the client is not the goal, we certainly
have a lot of options to use a lot less of it, thus minimizing the risk of doing too much computation on the main thread on page load.
Bottom Line: Lighthouse Is Still Just A Rough Guide #
Google
Lighthouse can’t detect everything that’s wrong with a particular
website. Whilst Lighthouse performance scores prioritize page usability
in terms of responding to user input, it still can’t detect every
terrible usability or accessibility issue in 2024.
In 2019, Manuel Matuzović published an experiment
where he intentionally created a terrible page that Lighthouse thought
was pretty great. I hypothesized that five years later, Lighthouse might
do better; but it doesn’t.
On this final demo page
I put together, input events are disabled by CSS and JavaScript, making
the page technically unresponsive to user input. After five seconds,
JavaScript flips a switch and allows you to click the button. The page
still scores 100 for both performance and accessibility.
As
with everything in life, there’s always a way to game the system. Here
are some more tried and tested guaranteed hacks to make sure your
Lighthouse performance score artificially knocks everyone else’s out of
the park:
Only run Lighthouse tests using the fastest and highest-spec hardware.
Make sure your internet connection is the fastest it can be; relocate if you need to.
Never
use field data, only lab data, collected using the aforementioned
fastest and highest-spec hardware and super-speed internet connection.
Rerun
the tests in the lab using different conditions and all the special
code hacks I described in this post until you get the result(s) you want
to impress your friends, colleagues, and random people on the internet.
Note: The
best way to learn about web performance and how to optimize your
websites is to do the complete opposite of everything we’ve covered in
this article all of the time. And finally, to seriously level up your
performance skills, use an application monitoring tool like Sentry. Think of Lighthouse as the canary and Sentry as the real-deal production-data-capturing, lean, mean, web vitals machine.
Five-second
testing is a popular method of usability research used in the industry.
It is a quick and effective way to test concepts of visual designs, yet
in essence, its core belief boils down to virtually a superstition. A
notion barely backed up by concrete evidence, yet often taken at face
value, that somehow five seconds are just the right time to gauge first
impressions. In this article, Eduard Kuric looks under the hood of how
first impressions are affected by time, the test participant’s cognitive
abilities and the visual complexity of the tested picture, and how UX
researchers and product owners can ensure that the user’s first steps
can get off on the right foot.
In today’s world of
shortening attention spans and omnipresent hustle, wasting even a second
could mean losing the chance to earn more time from a person you want
to impress. If your interests lie in creating good user experiences,
there is a fair chance you have heard of five-second testing.
Five-second
testing is an established technique of usability research used by UX
researchers, designers, product managers, and in a variety of other
professions, such as marketing or business analysis.
In
short, you show a picture of whatever you are designing (site, app, pair
of socks) to a member of your intended audience for exactly five
seconds. Then, you hide the picture and ask the participant a couple of
questions. The goal is to learn whether the reaction — the
participants’s first impression — is what you wanted to see. Did you get the main message across? Do people remember the company’s name? Sounds like an efficient way to test your product without needing to turn to full-on usability testing, right?
Note: The
word “participant” in this article is used to refer to users involved
in five-second testing or related usability research methods. The word
“user” is used in more general contexts since users form first
impressions all the time, not just when you are testing it.
Why is it five
seconds exactly, though? Are five seconds some magical moment when
everything we see should become clear? And if it does not, does it
automatically mean that a user experience is bad? Or are five seconds
just the right amount of time for first impressions to brew in the
user’s mind so that they’re neither undercooked nor overcooked?
These
are some of the questions that we asked ourselves. Not satisfied with
the answers written by others who covered the topic before us, we kept
drilling and conducted an actual peer-reviewed scientific case study,
exploring the hidden truths behind the testing of first impressions.
The research paper examines the five-second test and discusses the
results.
So strap in and read what science has to say about five-second testing. And then, what the implications are for you
so that you can take practical advantage of this new knowledge to
develop better first impressions of your services or products. But
first, let us delve into what we know about five-second testing and its
caveats so that you see the greater picture of the focal points of our
investigation.
You may be familiar with the well-known statistic that a website has about ten seconds to communicate its key message to the user.
Knowing that waiting only five seconds to ask testing participants
about their first impressions may suddenly seem like an odd choice. If
indeed visitors of a website have about ten seconds to grasp a message,
are five seconds really enough time for users? There is an alleged
justification, as we explain below.
Cutting a bit forward, though,
the factual basis for it is admittedly a bit of a Wild West if you look
for hard data to support it. There is an almost uncanny resemblance to
another not wholly scientific five-second rule that says it’s okay to
eat food off the ground if it’s within five seconds from when it dropped
there.
The
common story is that if something is shown to participants for more
than five seconds, their first impressions will start to deviate from
the actual user’s genuine initial impressions. The participant’s
perspective becomes more analytical and less task-driven. The
five-second test lets you avoid overtly speculative feedback that nobody
would give you under normal circumstances.
Fair enough, that
could potentially be true. But five seconds is still quite a short
period of time. Consider how different people can be when it comes to
their cognitive abilities (and there is nothing wrong
with that). For example, one user’s sharp perception may let them
realistically form first impressions in five seconds or faster, but
another user may barely have the time to blink, much less absorb any
meaningful information–they need a moment to take it in at their own
tempo.
The reasoning starts to fall apart a bit more at its seams once you also consider the visual complexity of the stimulus
(a.k.a., the picture you show to the participants). The nature of the
things you may want to test can range from very simple to very complex.
If the stimulus is simple, it is possible to take even less than five
seconds for participants to form their initial impressions. Would this
mean that there would be the risk of them using the remaining time to
get over-exceedingly analytical?
Conversely, there is the question
of whether five seconds is enough time to let participants
realistically visually scan a more complex stimulus. I can already hear
staunch proponents of five-second testing saying that this last
discrepancy is actually rightfully intentional. It’s a feature, not a
bug, if you will.
After all, if a stimulus is too complex, that is
exactly why you conduct five-second testing. It allows us to find out
about things like complexity. It can help you find out if participants
cannot extract the key information you want to communicate so that you
can fix it.
However, we need to consider that not all user
interfaces are the landing pages of websites. They serve to support
different user tasks, some of which cannot avoid having a certain degree
of complexity.
Five-second testing guides typically avoid directly addressing testing of these types of user interfaces by saying
that the method has the following limitation: it should not be used to
test user interfaces with multiple purposes. If the same stimulus serves
for more than one task, it is alleged that you should probably conduct
full-fledged usability testing, which is technically correct (the best
kind of correct).
Giving up on the five-second testing in the
inherently more complex user interfaces, however, also gives up on its
advantages for measuring and optimizing first impressions. For instance,
the idea that a screenshot or a mockup is all you need to quickly find
usability problems and iterate your designs. This is where five-second
testing really shines.
Usability testing does not tell you
accurately what the actual first impressions are without considerably
interrupting the participant. And even then, you would encounter the
same problem: At what moment from when the participant is exposed to
a design should their first impressions be gauged so that they are
genuine?
As we have discussed so far, there are certainly a
fair number of question marks surrounding five-second testing. The
method still undeniably has a number of merits, as proven by our
experience at UXtweak, where we also provide our own Five Second Test tool. A lack of proper research on the topic is what drove us at UXtweak Research to conduct our very own case study.
Essentially,
what we sought to investigate in our case study are the relations of a
number of key factors that are absolutely crucial for five-second
testing:
What are the cognitive abilities of the participant engaging in the five-second test?
How visually complex is the stimulus shown to the participant?
For how long is the stimulus shown to the participant?
What kinds of questions do we ask the participants afterward?
What is the feedback that participants give you?
As
you may have noticed, time — that iconic yet controversial five-second
threshold — is considered a variable factor. In our experiment, we
investigate the differences in feedback between three separate groups of
participants who are shown pictures for either five (5) or
alternatively two (2) or (10) seconds (so a bit less and a bit more
time, respectively). This means that it would not be correct to refer to
it as just a five-second test anymore, but rather an N-second test (or a
first impression test, if you do not wish to be too pedantic about the
number of seconds).
Each participant first passes not just one but
two cognitive ability tests. Human minds are multifaceted, and there is
not just a single “cognitive ability” metric that would encompass
everything that the mind can do. Among standard tests used by
psychologists, we picked two that are linked to abilities that can be
found as the most relevant to the formation and testing of first
impressions:
Perceptual speed: How quickly you pick up visual information.
Working memory: How much information you can mentally process at the same time.
Working
memory is the appropriate memory ability to focus on since it operates
with information that receives the user’s attention. This distinguishes
it from sensory memory (the memory processing information that our
senses pick up) and long-term memory, where information is stored
persistently for later use.
Working memory governs the capacity of information that receives attention. (Large preview)
For
the first impression test itself, six website screenshots were used as
the stimuli. These screenshots were selected for possessing a broad range of visual complexity,
from the simplest with just a few visual elements to the most complex
with a number of distinct sections that serve different purposes.
Screenshots
of real websites local to Czechia and Slovakia were translated into
English, and their logos were replaced with fictional brand names so
that, for all intents and purposes, the website screenshots would be
authentic yet also unfamiliar to the participants who were recruited in
the UK.
Website screenshots of varied visual complexity which employed during the experiment. (Large preview)
Finally, participants were asked to provide feedback by answering practically a complete portfolio of the various types of questions that can be typically asked during a first impression test. Each type of question tests a different aspect of the first impressions that the participants have formed inside their heads:
Attitudinal questions: Rating a perceived quality of the website (e.g., ugly vs. attractive) on a scale from 1 to 7.
Target identification questions: Questions directed at specific elements or aspects of the stimulus.
Memory dump questions: Asking participants to describe everything that they remember about what they saw.
The
resulting answers were analyzed both quantitatively (with statistics)
and qualitatively (by inspecting the contents of the received answers on
an individual level). With it, a number of conclusions can be reached,
some expected and some rather surprising.
Now that you have a
picture of what our case study was about let’s dive into the actual,
interesting implications for developing the first impressions of your
product.
Note: If you would like to immerse yourself in further details of how our case study was conducted, you can learn more in our scientific paper.
Statistically,
all the variables we experimented with — the time duration of showing
pictures to participants, the participants’ innate cognitive abilities,
and the visual complexity of pictures — had a significant effect on the
first impression answers.
For instance, between the groups that
were shown screenshots of websites for two, five, and ten seconds, the
number of answers that incorrectly identified what the websites were for
dropped as time progressed. Notable is the difference between five and
ten seconds. If the participants were really focusing on inconsequential
details after five seconds, there should not be differences in
recognition of such a key aspect as the website’s entire purpose.
Ability to correctly identify the purpose of a webpage gradually rises within the first ten seconds. (Large preview)
Statistical
differences lay the grounds for further observations on how changing
the conditions of a test can (or cannot) affect its results:
Attitudes crystallize faster than in five seconds. In
attitudinal questions where participants are asked to rate how they
view the picture’s various qualities (e.g., from clear to confusing,
from captivating to dull), answers stay relatively consistent,
regardless of how much time the participant has or how good their
cognitive abilities are. If you are laser-focused on assessing
participants’ attitudes about your product and nothing else, you could
present pictures for two seconds, or possibly even less, as research done by others on a related topic also implies.
Logos are recognized earlier than in five seconds (with one exception). The
target identification questions where participants are asked to recall
the company name from the logo are, on the whole, impacted by time very
little. This is to be expected: when viewing a website, our eyes are
usually drawn to the top left corner to find out where we have found
ourselves. There is an exception to this rule, however. Among
participants with slower perceptual speed, significantly fewer
identified the company name correctly at two seconds when compared to
five seconds. This establishes five seconds as a more inclusive choice
for timing your first impression test if you expect your target audience
to have, on average, lower perceptual speed than the general populace
and if the primary aim is to test contents of the header, such as logo
design or company name identification. Otherwise, two seconds is a safe
bet.
Irrelevant nitpicking? Yes, if visual complexity is low. In
some cases, the popular narrative about five seconds being a good
viewing time for testing first impressions is indeed true. Particularly
for the simplest website screenshots, once five seconds have elapsed,
participants start paying attention to minute details (e.g., the girl’s
shirt color in the hero image). Curiously, though, having more time
does not mean that participants would write longer or more complex
answers. Instead, when participants have ten seconds to view the
screenshots, the higher visual complexity of the screenshots is
reflected in better-quality answers. Participants stay more
on-topic–describing how the site is visually structured or justifying
their criticisms of the page’s design. Different viewing times may be
optimal in different situations. Especially since…
Low working memory warrants longer viewing time. When
asked to reiterate what they saw in their own words, participants with
low and high working memory provided significantly different answers.
With low working memory, answers become shorter, less complex, and
recall fewer concepts overall. However, when the viewing time is
extended to ten seconds, these differences disappear. This implies that
the same information is being processed — memory capacity just dictates
how fast it can happen. Without knowing where each participant’s
memory ability stands, it is difficult to tell what they would actually
recall if we left them to work at their own pace. Consequently,
assessment of working memory before testing first impressions (and
adjusting viewing time accordingly) should be considered a good
practice.
For cognitive powerhouses, five seconds are enough. A
less practical point maybe, but if you are developing an app for people
with reasonably high perceptual speed and working memory — be it the
mentally gifted, hyperproductive hustle enthusiasts, or caffeine addicts
— you could likely show them your screenshots for just two seconds and
get similar results as in a five-second test.
Give participants the proper amount of time to form a first impression. When
the visual stimulus is more visually complex in a first impression
test, the task of mentally processing it becomes more difficult and
time-consuming (just like in any normal scenario). This manifests in
test results. Fewer people correctly identify the purpose of a more
visually complex website, and they recall fewer elements and aspects of
the website. This could be seen as a bit of a paradox since more
complex stimuli mean there is actually more content that participants
could potentially remember and comment on, but only if they had the time
to absorb the information properly. Data shows that when participants
are given ten seconds, the answers do actually normalize, becoming more
similar to stimuli of lower visual complexity.
Keep
in mind there are still aspects of first impression testing that remain
unknown. A reasonable question that you can ask now would be: “Okay, so
how exactly do I time my first impression test?” While we can sum up
our observations into a conceptual framework of how time can be treated
in first-impression test planning, it is not an exact guideline; there
may be other interpretations or exceptions.
How to decide how many seconds to show pictures to participants in first impression testing? (Large preview)
Take
this more as an eye-opener and a call to action. Indeed, in our study,
ten seconds yielded more appropriate results for more complex websites
than five seconds did. But there is nothing to say that for other
websites, the best timing could not be fifteen or even twenty seconds.
Even more so, once you also factor in the influence of the cognitive
ability of each individual participant.
The key takeaway? When you gauge your audience’s first impressions about something, take a more holistic approach.
Try
to adjust the timing in your first impression test to match the
situation. To give an analogy, by blindly following a different
five-second rule and eating off the floor, you could end up getting
sick. Be just as cautious about relying on myths in your usability
research methods. This is not to discount five seconds. As we show, it
is still good timing for first impression tests in plenty of cases, but
it is not the be-all and end-all as far as first impression testing
goes. By broadening your perspective, you can do even better.
“How Long Do Users Stay on Web Pages?” Jakob Nielsen’s article on how long it takes people to leave a website (a fact to consider when testing first impressions).
Having
access to a free reporting tool like Lighthouse in DevTools is a great
start for diagnosing performance issues by identifying bottlenecks on
the main thread. Even better are paid tools like SpeedCurve
to dig deeper into the data for more targeted insights and to produce
visual reports to help make a case for performance improvements for your
team and other stakeholders.
Performance
work is one of those things, as they say, that ought to happen in
development. You know, have a plan for it and write code that’s mindful
about adding extra weight to the page.
But not everything about
performance happens directly at the code level, right? I’d say many — if
not most — sites and apps rely on some number of third-party scripts
where we might not have any influence over the code. Analytics is a good
example. Writing a hand-spun analytics tracking dashboard isn’t what my
clients really want to pay me for, so I’ll drop in the ol’ Google
Analytics script and maybe never think of it again.
That’s one
example and a common one at that. But what’s also common is managing
multiple third-party scripts on a single page. One of my clients is big
into user tracking, so in addition to a script for analytics, they’re
also running third-party scripts for heatmaps, cart abandonments, and
personalized recommendations — typical e-commerce stuff. All of that is
dumped on any given page in one fell swoop courtesy of Google Tag
Manager (GTM), which allows us to deploy and run scripts without having
to go through the pain of re-deploying the entire site.
As a
result, adding and executing scripts is a fairly trivial task. It is so
effortless, in fact, that even non-developers on the team have
contributed their own fair share of scripts, many of which I have no
clue what they do. The boss wants something, and it’s going to happen
one way or another, and GTM facilitates that work without friction
between teams.
All of this adds up to what I often hear described
as a “fight for the main thread.” That’s when I started hearing more
performance-related jargon, like web workers, Core Web Vitals, deferring
scripts, and using pre-connect, among others. But what I’ve
started learning is that these technical terms for performance make up
an arsenal of tools to combat performance bottlenecks.
The
real fight, it seems, is evaluating our needs as developers and
stakeholders against a user’s needs, namely, the need for a fast and
frictionless page load.
Fighting For The Main Thread
We’re
talking about performance in the context of JavaScript, but there are
lots of things that happen during a page load. The HTML is parsed. Same
deal with CSS. Elements are rendered. JavaScript is loaded, and scripts
are executed.
All of this happens on the main thread.
I’ve heard the main thread described as a highway that gets cars from
Point A to Point B; the more cars that are added to the road, the more
crowded it gets and the more time it takes for cars to complete their
trip. That’s accurate, I think, but we can take it a little further
because this particular highway has just one lane, and it only goes in one direction. My mind thinks of San Francisco’s Lombard Street, a twisty one-way path of a tourist trap on a steep decline.
The
main thread may not be that curvy, but you get the point: there’s only
one way to go, and everything that enters it must go through it.
JavaScript operates in much the same way. It’s “single-threaded,” which is how we get the one-way street comparison. I like how Brian Barbour explains it:
“This
means it has one call stack and one memory heap. As expected, it
executes code in order and must finish executing a piece of code before
moving on to the next. It's synchronous, but at times that can be
harmful. For example, if a function takes a while to execute or has to
wait on something, it freezes everything up in the meantime.”
— Brian Barbour
So,
there we have it: a fight for the main thread. Each resource on a page
is a contender vying for a spot on the thread and wants to run first. If
one contender takes its sweet time doing its job, then the contenders
behind it in line just have to wait.
Monitoring The Main Thread
If
you’re like me, I immediately reach for DevTools and open the
Lighthouse tab when I need to look into a site’s performance. It covers a
lot of ground, like reporting stats about a page’s load time that
include Time to First Byte (TTFB), First Contentful Paint (FCP), Largest Contentful Paint (LCP), Cumulative Layout Shift (CLS), and so on.
I
love this stuff! But I also am scared to death of it. I mean, this is
stuff for back-end engineers, right? A measly front-end designer like me
can be blissfully ignorant of all this mumbo-jumbo.
Meh, untrue.
Like accessibility, performance is everyone’s job because everyone’s
work contributes to it. Even the choice to use a particular CSS
framework influences performance.
Total Blocking Time
One
thing I know would be more helpful than a set of Core Web Vitals scores
from Lighthouse is knowing the time it takes to go from the First
Contentful Paint (FCP) to the Time to Interactive (TTI), a metric known
as the Total Blocking Time (TBT). You can see that
Lighthouse does indeed provide that metric. Let’s look at it for a site
that’s much “heavier” than Smashing Magazine.
There
we go. The problem with the Lighthouse report, though, is that I have
no idea what is causing that TBT. We can get a better view if we run the
same test in another service, like SpeedCurve,
which digs deeper into the metric. We can expand the metric to glean
insights into what exactly is causing traffic on the main thread.
That’s
a nice big view and is a good illustration of TBT’s impact on page
speed. The user is forced to wait a whopping 4.1 seconds between the
time the first significant piece of content loads and the time the page
becomes interactive. That’s a lifetime in web seconds, particularly
considering that this test is based on a desktop experience on a
high-speed connection.ne of my favorite charts in SpeedCurve is
this one showing the distribution of Core Web Vitals metrics during
render. You can see the delta between contentful paints and interaction!
]]
Spotting Long Tasks
What I really want to see is JavaScript, which takes more than 50ms to run. These are called long tasks,
and they contribute the most strain on the main thread. If I scroll
down further into the report, all of the long tasks are highlighted in
red.
Another
way I can evaluate scripts is by opening up the Waterfall View. The
default view is helpful to see where a particular event happens in the
timeline.
But
wait! This report can be expanded to see not only what is loaded at the
various points in time but whether they are blocking the thread and by
how much. Most important are the assets that come before the FCP.
I
can see right off the bat that Optimizely is serving a render-blocking
script. SpeedCurve can go even deeper by distinguishing between first-
and third-party scripts.
With
that in place, SpeedCurve actually lets me track all the resources from
a specific third-party source in a custom graph that offers me many
more data points to evaluate. For example, I can dive into scripts that
come from Optimizely with a set of custom filters to compare them with
overall requests and sizes.
This
provides a nice way to compare the impact of different third-party
scripts that represent blocking and long tasks, like how much time those
long tasks represent.
These
are the kinds of tools that allow us to identify bottlenecks and make a
case for optimizing them or removing them altogether. SpeedCurve allows
me to monitor this over time, giving me better insight into the
performance of those assets.
What’s so important about that? Well, FID has been used to measure load responsiveness, which is a fancy way of saying it looks at how fast the browser loads the first user interaction on the page. And by interaction, we mean some action the user takes that triggers an event, such as a click, mousedown, keydown, or pointerdown event. FID looks at the time the user sparks an interaction and how long the browser processes — or responds to — that input.
FID
might easily be overlooked when trying to diagnose long tasks on the
main thread because it looks at the amount of time a user spends waiting
after interacting with the page rather than the time it takes to render
the page itself. It can’t be replicated with lab data because it’s
based on a real user interaction. That said, FID is correlated to TBT in
that the higher the FID, the higher the TBT, and vice versa. So, TBT is
often the go-to metric for identifying long tasks because it can be
measured with lab data as well as real-user monitoring (RUM).
But FID is wrought with limitations, the most significant perhaps being that it’s only a measure of the first
interaction. That’s where INP comes into play. Instead of measuring the
first interaction and only the first interaction, it measures all interactions on a page. Jeremy Wagner has a more articulate explanation:
“The
goal of INP is to ensure the time from when a user initiates an
interaction until the next frame is painted is as short as possible for
all or most interactions the user makes.” — Jeremy Wagner
Some
interactions are naturally going to take longer to respond than others.
So, we might think of FID as merely a first impression of
responsiveness, whereas INP is a more complete picture. And like FID,
the INP score is closely correlated with TBT but even more so, as Annie
Sullivan reports:
Thankfully,
performance tools are already beginning to bake INP into their reports.
SpeedCurve is indeed one of them, and its report shows how its RUM
capabilities can be used to illustrate the correlation between INP and
long tasks on the main thread. This correlation chart illustrates how
INP gets worse as the total long tasks’ time increases.
There
is such a thing as a “good” script. It’s not like I’m some
anti-JavaScript bloke intent on getting scripts off the web. But what
constitutes a “good” one is nuanced.
Some
scripts benefit the organization, and others benefit the user (or
both). The challenge is balancing business needs with user needs.
I
think web fonts are a good example that serves both needs. A font is a
branding consideration as well as a design asset that can enhance the
legibility of a site’s content. Something like that might make loading a
font script or file worth its cost to page performance. That’s a tough
one. So, rather than fully eliminating a font, maybe it can be optimized
instead, perhaps by self-hosting the files rather than connecting to a
third-party domain or only loading a subset of characters.
Analytics
is another difficult choice. I removed analytics from my personal site
long ago because I rarely, if ever, looked at them. And even if I did,
the stats were more of an ego booster than insightful details that
helped me improve the user experience. It’s an easy decision for me, but
not so easy for a site that lives and dies by reports that are used to
identify and scope improvements.
If the script is really being used to benefit the user at the end of the day, then yeah, it’s worth keeping around.
A
script may very well serve a valid purpose and benefit both the
organization and the end user. But does it need to load first before
anything else? That’s the sort of question to ask when a script might be
useful, but can certainly jump out of line to let others run first.
I
think of chat widgets for customer support. Yes, having a persistent
and convenient way for customers to get in touch with support is going
to be important, particularly for e-commerce and SaaS-based services.
But does it need to be available immediately? Probably not.
You’ll probably have a greater case for getting the site to a state that
the user can interact with compared to getting a third-party widget up
front and center. There’s little point in rendering the widget if the
rest of the site is inaccessible anyway. It is better to get things
moving first by prioritizing some scripts ahead of others.
Just
because a script comes from a third party doesn’t mean it has to be
hosted by a third party. The web fonts example from earlier applies. Can
the font files be self-hosted instead rather than needing to establish
another outside connection? It’s worth asking. There are self-hosted
alternatives to Google Analytics, after all. And even GTM can be self-hosted! That’s why grouping first and third-party scripts in SpeedCurve’s reporting is so useful: spot what is being served and where it is coming from and identify possible opportunities.
Loading
one script can bring unexpected visitors along for the ride. I think
the classic case is a third-party script that loads its own assets, like
a stylesheet. Even if you think you’re only loading one stylesheet
&mdahs; your own — it’s very possible that a script loads additional
external stylesheets, all of which need to be downloaded and rendered.
That’s
the goal! We want fewer cars on the road to alleviate traffic on the
main thread. There are a bunch of technical ways to go about it. I’m not
here to write up a definitive guide of technical approaches for
optimizing the main thread, but there is a wealth of material on the
topic.
I’ll break down several different approaches and fill them in with resources that do a great job explaining them in full.
A
web worker, at its most basic, allows us to establish separate threads
that handle tasks off the main thread. Web workers run parallel to the
main thread. There are limitations to them, of course, most notably not
having direct access to the DOM and being unable to share variables with
other threads. But using them can be an effective way to re-route
traffic from the main thread to other streets, so to speak.
The
basic idea is to avoid bundling JavaScript as a monolithic concatenated
file in favor of “code splitting” or splitting the bundle up into
separate, smaller payloads to send only the code that’s needed. This
reduces the amount of JavaScript that needs to be parsed, which improves
traffic along the main thread.
Both are ways to load JavaScript without blocking the DOM. But they are different! Adding the async attribute to a <script> tag will load the script asynchronously, executing it as soon as it’s downloaded. That’s different from the defer attribute, which is also asynchronous but waits until the DOM is fully loaded before it executes.
I guess I could have filed this with async and defer. That’s because preconnect is a value on the rel attribute that’s used on a <link>
tag. It gives the browser a hint that you plan to connect to another
domain. It establishes the connection as soon as possible prior to
actually downloading the resource. The connection is done in advance,
allowing the full script to download later.
While it sounds
excellent — and it is — pre-connecting comes with an unfortunate
downside in that it exposes a user’s IP address to third-party resources
used on the page, which is a breach of GDPR compliance. There was a little uproar over that when it was found out that using a Google Fonts script is prone to that as well.
I often think of a Yiddish proverb I first saw in Malcolm Gladwell’s Outliers; however, many years ago it came out:
To a worm in horseradish, the whole world is horseradish.
It’s
a more pleasing and articulate version of the saying that goes, “To a
carpenter, every problem looks like a nail.” So, too, it is for
developers working on performance. To us, every problem is code that
needs a technical solution. But there are indeed ways to reduce the
amount of work happening on the main thread without having to touch code
directly.
We discussed earlier that performance is not only a
developer’s job; it’s everyone’s responsibility. So, think of these as
strategies that encourage a “culture” of good performance in an
organization.
Nuke Scripts That Lack Purpose
As
I said at the start of this article, there are some scripts on the
projects I work on that I have no idea what they do. It’s not because I
don’t care. It’s because GTM makes it ridiculously easy to inject
scripts on a page, and more than one person can access it across
multiple teams.
So, maybe compile a list of all the third-party
and render-blocking scripts and figure out who owns them. Is it Dave in
DevOps? Marcia in Marketing? Is it someone else entirely? You gotta make
friends with them. That way, there can be an honest evaluation of which
scripts are actually helping and are critical to balance.
Bend Google Tag Manager To Your Will
Or
any tag manager, for that matter. Tag managers have a pretty bad
reputation for adding bloat to a page. It’s true; they can definitely
make the page size balloon as more and more scripts are injected.
But
that reputation is not totally warranted because, like most tools, you
have to use them responsibly. Sure, the beauty of something like GTM is
how easy it makes adding scripts to a page. That’s the “Tag” in Google
Tag Manager. But the real beauty is that convenience, plus the features it provides to manage the scripts. You know, the “Manage” in Google Tag Manager. It’s spelled out right on the tin!
Phew!
Performance is not exactly a straightforward science. There are
objective ways to measure performance, of course, but if I’ve learned
anything about it, it’s that subjectivity is a big part of the process.
Different scripts are of different sizes and consist of different
resources serving different needs that have different priorities for
different organizations and their users.
Having access to a free
reporting tool like Lighthouse in DevTools is a great start for
diagnosing performance issues by identifying bottlenecks on the main
thread. Even better are paid tools like SpeedCurve
to dig deeper into the data for more targeted insights and to produce
visual reports to help make a case for performance improvements for your
team and other stakeholders.
While I wish there were some sort of
silver bullet to guarantee good performance, I’ll gladly take these and
similar tools as a starting point. Most important, though, is having a
performance game plan that is served by the tools. And Vitaly’s front-end performance checklist is an excellent place to start.
