From competitive SEO research and monitoring prices to training AI and
parsing local geographic data, real-time search results power smarter
apps. Tools like SerpApi make it easy to pull, customize, and integrate
this data directly into your app or website.
SerpApi leverages the power of search engine giants, like Google,
DuckDuckGo, Baidu, and more, to put together the most pertinent and
accurate search result data for your users from the comfort of your app
or website. It’s customizable, adaptable, and offers an easy integration
into any project.
What do you want to put together?
Search information on a brand or business for SEO purposes;
Input data to train AI models, such as the Large Language Model, for a customer service chatbot;
Top news and websites to pick from for a subscriber newsletter;
The results from the search API are only a URL request away
for those who want a super quick start. Just add your search details in
the URL parameters. Say you need the search result for “Stone Henge”
from the location “Westminster, England, United Kingdom” in language
“en-GB”, and country of search origin “uk” from the domain
“google.co.uk”. Here’s how simple it is to put the GET request together:
Then
there’s the impressive list of libraries that seamlessly integrate the
APIs into mainstream programming languages and frameworks such as
JavaScript, Ruby, .NET, and more.
Want to give it a spin? Sign up and start for free, or tinker with the SerpApi’s live playground without signing up. The playground
allows you to choose which search engine to target, and you can fill in
the values for all the basic parameters available in the chosen API to
customize your search. On clicking “Search”, you get the search result
page and its extracted JSON data.
Playground search for flights from LGW to MLA airport using SerpApi’s Google Flights API. (Large preview)
If you need to get a feel for the full API first, you can explore their easy-to-grasp web documentation
before making any decision. You have the chance to work with all of the
APIs to your satisfaction before committing to it, and when that time
comes, SerpApi’s multiple price plans tackle anywhere between an economic few hundred searches a month and bulk queries fit for large corporations.
Capture
the global trends, or refine down to more localized particulars by
names of locations or Google’s place identifiers. SerpApi’s optimized
routing of requests ensures accurate retrieval of search result
data from any location worldwide. If locations themselves are the
answers to your queries — say, a cycle trail to be suggested in a
fitness app — those can be extracted and presented as maps using
SerpApi’s Google Maps API.
Although
search engines reveal results in a tidy user interface, deriving data
into your application could cause you to end up with a large data dump
to be sifted through — but not if you’re using SerpApi.
SerpApi pulls data in a well-structured JSON format, even for the popular kinds of enriched search results, such as knowledge graphs, review snippets, sports league stats, ratings, product listings, AI overview, and more.
SerpApi returns data in a JSON format, making it easy to integrate into your application. (Large preview)Various types of search engine results, such as meta related to video, audio, geolocation, questions, and recipes. (Large preview)
SerpApi’s
baseline performance can take care of timely search data for real-time
requirements. But what if you need more? SerpApi’s Ludicrous Speed
option, easily enabled from the dashboard with an upgrade, provides a
super-fast response time. More than twice as fast as usual, thanks to
twice the server power.
There’s also Ludicrous Speed Max,
which allocates four times more server resources for your data
retrieval. Data that is time-sensitive and for monitoring things in
real-time, such as sports scores and tracking product prices, will lose
its value if it is not handled in a timely manner. Ludicrous Speed Max
guarantees no delays, even for a large-scale enterprise haul.
A list of flight prices from Google Flights. (Large preview)
If you don’t need the full depth of the searchAPI, there’s a Light version available for Google Search, Google Images, Google Videos, Google News, and DuckDuckGo Search APIs.
Need the results asynchronously picked up? Want a refined output using advanced searchAPI parameters and a JSON Restrictor? Looking for search outcomes for specific devices? Don’t want auto-corrected query results? There’s no shortage of ways to configure SerpApi to get exactly what you need.
Additionally, if you prefer not to have your search metadata on their servers, simply turn on the “ZeroTrace” mode that’s available for selected plans.
Save
yourself a headache, literally, trying to play match between what you
see on a search result page and its extracted data in JSON. SerpApi’s X-Ray toolshows you where what comes from. It’s available and free in all plans.
If you don’t have the expertise or resources for tackling the validity of scraping search results, here’s what SerpApi says:
“SerpApi,
LLC assumes scraping and parsing liabilities for both domestic and
foreign companies unless your usage is otherwise illegal”.
You
can reach out and have a conversation with them regarding the legal
protections they offer, as well as inquire about anything else you might
want to know about, including SerpApi in your project, such as pricing,
performance expected, on-demand options, and technical support. Just
drop a message at their contact page.
In other words, the SerpApi team has your back with the support and expertise to get the most from your fetched data.
Try SerpApi Free
That’s
right, you can get your hands on SerpApi today and start fetching data
with absolutely no commitment, thanks to a free starter plan that gives
you up to 250 free search queries. Give it a try and then bump up to one
of the reasonably-priced monthly subscription plans with generous
search limits.
Have you heard of the Screen Orientation API? What about the Device
Orientation API, Vibration API, or the Contact Picker API? interested in these under-the-radar web features and
discusses how they can be used to create more usable and robust
progressive web apps if and when they gain broader suppo
A couple of years ago, four JavaScript APIs that landed at the bottom of awareness in the State of JavaScript survey.
I took an interest in those APIs because they have so much potential to
be useful but don’t get the credit they deserve. Even after a quick
search, I was amazed at how many new web APIs have been added to the
ECMAScript specification that aren’t getting their dues and with a lack
of awareness and browser support in browsers.
Most
of these APIs are designed to power progressive web apps (PWA) and
close the gap between web and native apps. Bear in mind that creating a
PWA involves more than just adding a manifest file.
Sure, it’s a PWA by definition, but it functions like a bookmark on
your home screen in practice. In reality, we need several APIs to
achieve a fully native app experience on the web. And the four APIs I’d
like to shed light on are part of that PWA puzzle that brings to the web
what we once thought was only possible in native apps.
The Screen Orientation API
can be used to sniff out the device’s current orientation. Once we know
whether a user is browsing in a portrait or landscape orientation, we
can use it to enhance the UX for mobile devices by changing the UI accordingly. We can also use it to lock the screen in a certain position, which is useful for displaying videos and other full-screen elements that benefit from a wider viewport.
Using the global screen object, you can access various properties the screen uses to render a page, including the screen.orientation object. It has two properties:
type: The current screen orientation. It can be: "portrait-primary", "portrait-secondary", "landscape-primary", or "landscape-secondary".
angle:
The current screen orientation angle. It can be any number from 0 to
360 degrees, but it’s normally set in multiples of 90 degrees (e.g., 0, 90, 180, or 270).
On mobile devices, if the angle is 0 degrees, the type is most often going to evaluate to "portrait" (vertical), but on desktop devices, it is typically "landscape" (horizontal). This makes the type property precise for knowing a device’s true position.
The screen.orientation object also has two methods:
.lock(): This is an async method that takes a type value as an argument to lock the screen.
.unlock(): This method unlocks the screen to its default orientation.
And lastly, screen.orientation counts with an "orientationchange" event to know when the orientation has changed.
To lock the screen, we need to first be in full-screen mode, so we will use another extremely useful feature: the Fullscreen API. Nobody wants a webpage to pop into full-screen mode without their consent, so we need transient activation (i.e., a user click) from a DOM element to work.
The Fullscreen API has two methods:
Document.exitFullscreen() is used from the global document object,
Element.requestFullscreen() makes the specified element and its descendants go full-screen.
We want the entire page to be full-screen so we can invoke the method from the root element at the document.documentElement object:
const fullscreenButton = document.querySelector(".fullscreen-button");
fullscreenButton.addEventListener("click",async()=>{// If it is already in full-screen, exit to normal viewif(document.fullscreenElement){await document.exitFullscreen();}else{await document.documentElement.requestFullscreen();}});
Next, we can lock the screen in its current orientation:
Yes! We can indeed check page orientation via the orientation media feature
in a CSS media query. However, media queries compute the current
orientation by checking if the width is “bigger than the height” for
landscape or “smaller” for portrait. By contrast,
You
may have noticed how PWAs like Instagram and X force the screen to be
in portrait mode even when the native system orientation is unlocked. It
is important to notice that this behavior isn’t achieved through the
Screen Orientation API, but by setting the orientation property on the manifest.json file to the desired orientation type.
Another
API I’d like to poke at is the Device Orientation API. It provides
access to a device’s gyroscope sensors to read the device’s orientation
in space; something used all the time in mobile apps, mainly games. The
API makes this happen with a deviceorientation event that triggers each time the device moves. It has the following properties:
event.alpha: Orientation along the Z-axis, ranging from 0 to 360 degrees.
event.beta: Orientation along the X-axis, ranging from -180 to 180 degrees.
event.gamma: Orientation along the Y-axis, ranging from -90 to 90 degrees.
In
this case, we will make a 3D cube with CSS that can be rotated with
your device! The full instructions I used to make the initial CSS cube
are credited to David DeSandro and can be found in his introduction to 3D transforms.
To
keep this brief, I won’t explain the CSS code here. Just keep in mind
that it provides the necessary styles for the 3D cube, and it can be
rotated through all axes using the CSS rotate() function.
Now, with JavaScript, we listen to the window’s deviceorientation event and access the event orientation data:
Let’s
turn our attention to the Vibration API, which, unsurprisingly, allows
access to a device’s vibrating mechanism. This comes in handy when we
need to alert users with in-app notifications, like when a process is
finished or a message is received. That said, we have to use it
sparingly; no one wants their phone blowing up with notifications.
There’s just one method that the Vibration API gives us, and it’s all we need: navigator.vibrate().
vibrate() is available globally from the navigator
object and takes an argument for how long a vibration lasts in
milliseconds. It can be either a number or an array of numbers
representing a patron of vibrations and pauses.
Let’s
make a quick demo where the user inputs how many milliseconds they want
their device to vibrate and buttons to start and stop the vibration,
starting with the markup:
In
the past, it used to be that only native apps could connect to a
device’s “contacts”. But now we have the fourth and final API I want to
look at: the Contact Picker API.
The API grants web apps access to the device’s contact lists. Specifically, we get the contacts.select() async method available through the navigator object, which takes the following two arguments:
properties: This is an array containing the information we want to fetch from a contact card, e.g., "name", "address", "email", "tel", and "icon".
options: This is an object that can only contain the multiple boolean property to define whether or not the user can select one or multiple contacts at a time.
I’m
afraid that browser support is next to zilch on this one, limited to
Chrome Android, Samsung Internet, and Android’s native web browser at
the time I’m writing this.
There we go, four web APIs that I believe would empower us to build more useful and robust PWAs
but have slipped under the radar for many of us. This is, of course,
due to inconsistent browser support, so I hope this article can bring
awareness to new APIs so we have a better chance to see them in future
browser updates.
Aren’t they interesting? We saw how much control
we have with the orientation of a device and its screen as well as the
level of access we get to access a device’s hardware features, i.e.
vibration, and information from other apps to use in our own UI.
But as I said much earlier, there’s a sort of infinite loop where a lack of awareness begets a lack of browser support.
So, while the four APIs we covered are super interesting, your mileage
will inevitably vary when it comes to using them in a production
environment. Please tread cautiously and refer to Caniuse for the latest support information, or check for your own devices using WebAPI Check.
The
Performance API is a set of standards for measuring and evaluating
performance metrics with JavaScript. Think of it as a box containing all
of the same functionality for reporting on Core Web Vitals and general
performance statistics that you’d get in many performance testing tools.
This article demonstrates how to use the Performance API to generate
performance metrics directly in the DOM to create your own reporting.
This article has been kindly supported by our dear friends at DebugBear, who help optimize web performance to improve user experience. Thank you!
There’s quite a buzz in the performance community with the Interaction to Next Paint (INP) metric becoming an official Core Web Vitals (CWV) metric in a few short weeks. If you haven’t heard, INP is replacing the First Input Delay (FID) metric, something you can read all about here on Smashing Magazine as a guide to prepare for the change.
But
that’s not what I really want to talk about. With performance at the
forefront of my mind, I decided to head over to MDN for a fresh look at
the Performance API.
We can use it to report the load time of elements on the page, even
going so far as to report on Core Web Vitals metrics in real time. Let’s
look at a few ways we can use the API to report some CWV metrics.
Before
we get started, a quick word about browser support. The Performance API
is huge in that it contains a lot of different interfaces, properties,
and methods. While the majority of it is supported by all major
browsers, Chromium-based browsers are the only ones that support all of
the CWV properties. The only other is Firefox, which supports the First
Contentful Paint (FCP) and Largest Contentful Paint (LCP) API
properties.
So, we’re looking at a feature of features, as it
were, where some are well-established, and others are still in the
experimental phase. But as far as Core Web Vitals go, we’re going to
want to work in Chrome for the most part as we go along.
There are two main ways to retrieve the performance metrics we care about:
Using the performance.getEntries() method, or
Using a PerformanceObserver instance.
Using a PerformanceObserver instance offers a few important advantages:
PerformanceObserver observes performance metrics and dispatches them over time. Instead, using performance.getEntries() will always return the entire list of entries since the performance metrics started being recorded.
PerformanceObserver dispatches the metrics asynchronously, which means they don’t have to block what the browser is doing.
The element performance metric type doesn’t work with the performance.getEntries() method anyway.
That all said, let’s create a PerformanceObserver:
For now, we’re passing an empty callback function to the PerformanceObserver
constructor. Later on, we’ll change it so that it actually does
something with the observed performance metrics. For now, let’s start
observing:
The first very important thing in that snippet is the buffered: true property. Setting this to true means that we not only get to observe performance metrics being dispatched after we start observing, but we also want to get the performance metrics that were queued by the browser before we started observing.
The second very important thing to note is that we’re working with the largest-contentful-paint
property. That’s what’s cool about the Performance API: it can be used
to measure very specific things but also supports properties that are
mapped directly to CWV metrics. We’ll start with the LCP metric before
looking at other CWV metrics.
The largest-contentful-paint
property looks at everything on the page, identifying the biggest piece
of content on the initial view and how long it takes to load. In other
words, we’re observing the full page load and getting stats on the
largest piece of content rendered in view.
We already have our Performance Observer and callback:
Let’s fill in that empty callback so that it returns a list of entries once performance measurement starts:
// The Performance Observerconst lcpObserver =newPerformanceObserver(list=>{// Returns the entire list of entriesconst entries = list.getEntries();});// Call the Observer
lcpObserver.observe({ type:"largest-contentful-paint", buffered:true});
Next, we want to know which element is pegged as the LCP. It’s worth noting that the element representing the LCP is always the last element in the ordered list of entries. So, we can look at the list of returned entries and return the last one:
// The Performance Observerconst lcpObserver =newPerformanceObserver(list=>{// Returns the entire list of entriesconst entries = list.getEntries();// The element representing the LCPconst el = entries[entries.length - 1];});// Call the Observer
lcpObserver.observe({ type:"largest-contentful-paint", buffered:true});
The
last thing is to display the results! We could create some sort of
dashboard UI that consumes all the data and renders it in an
aesthetically pleasing way. Let’s simply log the results to the console
rather than switch gears.
// The Performance Observerconst lcpObserver =newPerformanceObserver(list=>{// Returns the entire list of entriesconst entries = list.getEntries();// The element representing the LCPconst el = entries[entries.length -1];// Log the results in the consoleconsole.log(el.element);});// Call the Observer
lcpObserver.observe({ type:"largest-contentful-paint", buffered:true});
There we go!
LCP support is limited to Chrome and Firefox at the time of writing. (Large preview)
It’s
certainly nice knowing which element is the largest. But I’d like to
know more about it, say, how long it took for the LCP to render:
// The Performance Observerconst lcpObserver =newPerformanceObserver(list=>{const entries = list.getEntries();const lcp = entries[entries.length -1];
entries.forEach(entry=>{// Log the results in the console
console.log(`The LCP is:`,
lcp.element,`The time to render was ${entry.startTime} milliseconds.`,);});});// Call the Observer
lcpObserver.observe({ type:"largest-contentful-paint", buffered:true});// The LCP is:// <h2 class="author-post__title mt-5 text-5xl">…</h2>// The time to render was 832.6999999880791 milliseconds.
This
is all about the time it takes for the very first piece of DOM to get
painted on the screen. Faster is better, of course, but the way
Lighthouse reports it, a “passing” score comes in between 0 and 1.8
seconds.
Just like we set the type property to largest-contentful-paint to fetch performance data in the last section, we’re going to set a different type this time around: paint.
When we call paint, we tap into the PerformancePaintTiming interface that opens up reporting on first paint and first contentful paint.
// The Performance Observerconst paintObserver =newPerformanceObserver(list=>{const entries = list.getEntries();
entries.forEach(entry=>{// Log the results in the console.
console.log(`The time to ${entry.name} took ${entry.startTime} milliseconds.`,);});});// Call the Observer.
paintObserver.observe({ type:"paint", buffered:true});// The time to first-paint took 509.29999999981374 milliseconds.// The time to first-contentful-paint took 509.29999999981374 milliseconds.
Notice how paint spits out two results: one for the first-paint and the other for the first-contenful-paint.
I know that a lot happens between the time a user navigates to a page
and stuff starts painting, but I didn’t know there was a difference
between these two metrics.
“The
primary difference between the two metrics is that [First Paint] marks
the first time the browser renders anything for a given document. By
contrast, [First Contentful Paint] marks the time when the browser
renders the first bit of image or text content from the DOM.”
As it turns out, the first paint and FCP data I got back in that last example are identical. Since first paint can be anything that prevents a blank screen,
e.g., a background color, I think that the identical results mean that
whatever content is first painted to the screen just so happens to also
be the first contentful paint.
But there’s apparently a lot more
nuance to it, as Chrome measures FCP differently based on what version
of the browser is in use. Google keeps a full record of the changelog
for reference, so that’s something to keep in mind when evaluating
results, especially if you find yourself with different results from
others on your team.
How
much does the page shift around as elements are painted to it? Of
course, we can get that from the Performance API! Instead of largest-contentful-paint or paint, now we’re turning to the layout-shift type.
As it currently stands, LayoutShift opens up several pieces of information, including a value representing the amount of shifting, as well as the sources
causing it to happen. More than that, we can tell if any user
interactions took place that would affect the CLS value, such as
zooming, changing browser size, or actions like keydown, pointerdown, and mousedown. This is the lastInputTime property, and there’s an accompanying hasRecentInput boolean that returns true if the lastInputTime is less than 500ms.
Got
all that? We can use this to both see how much shifting takes place
during page load and identify the culprits while excluding any shifts
that are the result of user interactions.
const observer =newPerformanceObserver((list)=>{let cumulativeLayoutShift =0;
list.getEntries().forEach((entry)=>{// Don't count if the layout shift is a result of user interaction.if(!entry.hadRecentInput){
cumulativeLayoutShift += entry.value;}
console.log({ entry, cumulativeLayoutShift });});});// Call the Observer.
observer.observe({ type:"layout-shift", buffered:true});
Given the experimental nature of this one, here’s what an entry object looks like when we query it:
Pretty handy, right? Not only are we able to see how much shifting takes place (0.128) and which element is moving around (article.a.main), but we have the exact coordinates of the element’s box from where it starts to where it ends.
This
is the new kid on the block that got my mind wondering about the
Performance API in the first place. It’s been possible for some time now
to measure INP as it transitions to replace First Input Delay as a Core
Web Vitals metric in March 2024. When we’re talking about INP, we’re
talking about measuring the time between a user interacting with the
page and the page responding to that interaction.
We need to hook into the PerformanceEventTiming class
for this one. And there’s so much we can dig into when it comes to user
interactions. Think about it! There’s what type of event happened (entryType and name), when it happened (startTime), what element triggered the interaction (interactionId, experimental), and when processing the interaction starts (processingStart) and ends (processingEnd). There’s also a way to exclude interactions that can be canceled by the user (cancelable).
const observer =newPerformanceObserver((list)=>{
list.getEntries().forEach((entry)=>{// Alias for the total duration.const duration = entry.duration;// Calculate the time before processing starts.const delay = entry.processingStart - entry.startTime;// Calculate the time to process the interaction.const lag = entry.processingStart - entry.startTime;// Don't count interactions that the user can cancel.if(!entry.cancelable){
console.log(`INP Duration: ${duration}`);
console.log(`INP Delay: ${delay}`);
console.log(`Event handler duration: ${lag}`);}});});// Call the Observer.
observer.observe({ type:"event", buffered:true});
Let’s
build off that last one. We can now track INP scores on our website and
break them down into specific components. But what code is actually
running and causing those delays?
The Long Animation Frames API
was developed to help answer that question. It won’t land in Chrome
stable until mid-March 2024, but you can already use it in Chrome
Canary.
A long-animation-frame entry is reported
every time the browser couldn’t render page content immediately as it
was busy with other processing tasks. We get an overall duration for the long frame but also a duration for different scripts involved in the processing.
const observer =newPerformanceObserver((list)=>{
list.getEntries().forEach((entry)=>{if(entry.duration >50){// Log the overall duration of the long frame.
console.log(`Frame took ${entry.duration} ms`)
console.log(`Contributing scripts:`)// Log information on each script in a table.
entry.scripts.forEach(script=>{
console.table({// URL of the script where the processing starts
sourceURL: script.sourceURL,// Total time spent on this sub-task
duration: script.duration,// Name of the handler function
functionName: script.sourceFunctionName,// Why was the handler function called? For example, // a user interaction or a fetch response arriving.
invoker: script.invoker
})})}});});// Call the Observer.
observer.observe({ type:"long-animation-frame", buffered:true});
When
an INP interaction takes place, we can find the closest long animation
frame and investigate what processing delayed the page response.
The
Performance API is so big and so powerful. We could easily spend an
entire bootcamp learning all of the interfaces and what they provide.
There’s network timing, navigation timing, resource timing, and plenty
of custom reporting features available on top of the Core Web Vitals
we’ve looked at.
If CWVs are what you’re really after, then you might consider looking into the web-vitals library to wrap around the browser Performance APIs.
Need a CWV metric? All it takes is a single function.
Boom! That reportAllChanges
property? That’s a way of saying we only want to report data every time
the metric changes instead of only when the metric reaches its final
value. For example, as long as the page is open, there’s always a chance
that the user will encounter an even slower interaction than the
current INP interaction. So, without reportAllChanges, we’d
only see the INP reported when the page is closed (or when it’s hidden,
e.g., if the user switches to a different browser tab).
We can also report purely on the difference between the preliminary results and the resulting changes. From the web-vitals docs:
functionlogDelta({ name, id, delta }){
console.log(`${name} matching ID ${id} changed by ${delta}`);}onCLS(logDelta);onINP(logDelta);onLCP(logDelta);
All
we’ve done here is scratch the surface of the Performance API as far as
programmatically reporting Core Web Vitals metrics. It’s fun to play
with things like this. There’s even a slight feeling of power in being able to tap into this information on demand.
At the end of the day, though, you’re probably just as interested in monitoring performance as you are in measuring it.
We could do a deep dive and detail what a performance dashboard powered
by the Performance API is like, complete with historical records that
indicate changes over time. That’s ultimately the sort of thing we can
build on this — we can build our own real user monitoring (RUM) tool or
perhaps compare Performance API values against historical data from the Chrome User Experience Report(CrUX).
Or
perhaps you want a solution right now without stitching things
together. That’s what you’ll get from a paid commercial service like DebugBear.
All of this is already baked right in with all the metrics, historical
data, and charts you need to gain insights into the overall performance
of a site over time… and in real-time, monitoring real users.
DebugBear
can help you identify why users are having slow experiences on any
given page. If there is slow INP, what page elements are these users
interacting with? What elements often shift around on the page and cause
high CLS? Is the LCP typically an image, a heading, or something else?
And does the type of LCP element impact the LCP score?
To help
explain INP scores, DebugBear also supports the upcoming Long Animation
Frames API we looked at, allowing you to see what code is responsible
for interaction delays.
The Performance API can also report a list of all resource requests on a page. DebugBear uses this information to show a request waterfall chart
that tells you not just when different resources are loaded but also
whether the resources were render-blocking, loaded from the cache or
whether an image resource is used for the LCP element.
In this
screenshot, the blue line shows the FCP, and the red line shows the LCP.
We can see that the LCP happens right after the LCP image request,
marked by the blue “LCP” badge, has finished.
DebugBear offers a 14-day free trial.
See how fast your website is, what’s slowing it down, and how you can
improve your Core Web Vitals. You’ll also get monitoring alerts, so if
there’s a web vitals regression, you’ll find out before it starts
impacting Google search results.
