Building An Offline-Friendly Image Upload System

 

Poor internet connectivity doesn’t have to mean poor UX. With PWA technologies like IndexedDB, service workers, and the Background Sync API, you can build an offline-friendly image upload system that queues uploads and retries them automatically — so your users can upload stress-free, even when offline.

So, you’re filling out an online form, and it asks you to upload a file. You click the input, select a file from your desktop, and are good to go. But something happens. The network drops, the file disappears, and you’re stuck having to re-upload the file. Poor network connectivity can lead you to spend an unreasonable amount of time trying to upload files successfully.

What ruins the user experience stems from having to constantly check network stability and retry the upload several times. While we may not be able to do much about network connectivity, as developers, we can always do something to ease the pain that comes with this problem.

One of the ways we can solve this problem is by tweaking image upload systems in a way that enables users to upload images offline — eliminating the need for a reliable network connection, and then having the system retry the upload process when the network becomes stable, without the user intervening.

This article is going to focus on explaining how to build an offline-friendly image upload system using PWA (progressive web application) technologies such as IndexedDB, service workers, and the Background Sync API. We will also briefly cover tips for improving the user experience for this system.

Planning The Offline Image Upload System

Here’s a flow chart for an offline-friendly image upload system.

Flow chart of an offline-friendly image upload system (Large preview)

As shown in the flow chart, the process unfolds as follows:

1. The user selects an image.
   The process begins by letting the user select their image.
2. The image is stored locally in IndexedDB.
   Next, the system checks for network connectivity. If network connectivity is available, the system uploads the image directly, avoiding unnecessary local storage usage. However, if the network is not available, the image will be stored in IndexedDB.
3. The service worker detects when the network is restored.
   With the image stored in IndexedDB, the system waits to detect when the network connection is restored to continue with the next step.
4. The background sync processes pending uploads.
   The moment the connection is restored, the system will try to upload the image again.
5. The file is successfully uploaded.
   The moment the image is uploaded, the system will remove the local copy stored in IndexedDB.

   Implementing The System

   The first step in the system implementation is allowing the user to select their images. There are different ways you can achieve this:

6. You can use a simple <input type="file"> element;
7. A drag-and-drop interface.

I would advise that you use both. Some users prefer to use the drag-and-drop interface, while others think the only way to upload images is through the <input type="file"> element. Having both options will help improve the user experience. You can also consider allowing users to paste images directly in the browser using the Clipboard API.

Registering The Service Worker

At the heart of this solution is the service worker. Our service worker is going to be responsible for retrieving the image from the IndexedDB store, uploading it when the internet connection is restored, and clearing the IndexedDB store when the image has been uploaded.

To use a service worker, you first have to register one:

if ('serviceWorker' in navigator) {
  navigator.serviceWorker.register('/service-worker.js')
    .then(reg => console.log('Service Worker registered', reg))
    .catch(err => console.error('Service Worker registration failed', err));
}

Checking For Network Connectivity

Remember, the problem we are trying to solve is caused by unreliable network connectivity. If this problem does not exist, there is no point in trying to solve anything. Therefore, once the image is selected, we need to check if the user has a reliable internet connection before registering a sync event and storing the image in IndexedDB.

function uploadImage() {
  if (navigator.onLine) {
    // Upload Image
  } else {
    // register Sync Event
    // Store Images in IndexedDB
  }
}

Note: I’m only using the navigator.onLine property here to demonstrate how the system would work. The navigator.onLine property is unreliable, and I would suggest you come up with a custom solution to check whether the user is connected to the internet or not. One way you can do this is by sending a ping request to a server endpoint you’ve created.

Registering The Sync Event

Once the network test fails, the next step is to register a sync event. The sync event needs to be registered at the point where the system fails to upload the image due to a poor internet connection.

async function registerSyncEvent() {
  if ('SyncManager' in window) {
    const registration = await navigator.serviceWorker.ready;
    await registration.sync.register('uploadImages');
    console.log('Background Sync registered');
  }
}

After registering the sync event, you need to listen for it in the service worker.

self.addEventListener('sync', (event) => {
  if (event.tag === 'uploadImages') {
    event.waitUntil(sendImages());
  }
});

The sendImages function is going to be an asynchronous process that will retrieve the image from IndexedDB and upload it to the server. This is what it’s going to look like:

async function sendImages() {
  try {
    // await image retrieval and upload
  } catch (error) {
    // throw error
  }
}

Opening The Database

The first thing we need to do in order to store our image locally is to open an IndexedDB store. As you can see from the code below, we are creating a global variable to store the database instance. The reason for doing this is that, subsequently, when we want to retrieve our image from IndexedDB, we wouldn’t need to write the code to open the database again.

let database; // Global variable to store the database instance

function openDatabase() {
  return new Promise((resolve, reject) => {
    if (database) return resolve(database); // Return existing database instance 

    const request = indexedDB.open("myDatabase", 1);

    request.onerror = (event) => {
      console.error("Database error:", event.target.error);
      reject(event.target.error); // Reject the promise on error
    };

    request.onupgradeneeded = (event) => {
        const db = event.target.result;
        // Create the "images" object store if it doesn't exist.
        if (!db.objectStoreNames.contains("images")) {
          db.createObjectStore("images", { keyPath: "id" });
        }
        console.log("Database setup complete.");
    };

    request.onsuccess = (event) => {
      database = event.target.result; // Store the database instance globally
      resolve(database); // Resolve the promise with the database instance
    };
  });
}

Storing The Image In IndexedDB

With the IndexedDB store open, we can now store our images.

Now, you may be wondering why an easier solution like localStorage wasn’t used for this purpose.

The reason for that is that IndexedDB operates asynchronously and doesn’t block the main JavaScript thread, whereas localStorage runs synchronously and can block the JavaScript main thread if it is being used.

Here’s how you can store the image in IndexedDB:

async function storeImages(file) {
  // Open the IndexedDB database.
  const db = await openDatabase();
  // Create a transaction with read and write access.
  const transaction = db.transaction("images", "readwrite");
  // Access the "images" object store.
  const store = transaction.objectStore("images");
  // Define the image record to be stored.
  const imageRecord = {
    id: IMAGE_ID,   // a unique ID
    image: file     // Store the image file (Blob)
  };
  // Add the image record to the store.
  const addRequest = store.add(imageRecord);
  // Handle successful addition.
  addRequest.onsuccess = () => console.log("Image added successfully!");
  // Handle errors during insertion.
  addRequest.onerror = (e) => console.error("Error storing image:", e.target.error);
}

With the images stored and the background sync set, the system is ready to upload the image whenever the network connection is restored.

Retrieving And Uploading The Images

Once the network connection is restored, the sync event will fire, and the service worker will retrieve the image from IndexedDB and upload it.

async function retrieveAndUploadImage(IMAGE_ID) {
  try {
    const db = await openDatabase(); // Ensure the database is open
    const transaction = db.transaction("images", "readonly");
    const store = transaction.objectStore("images");
    const request = store.get(IMAGE_ID);
    request.onsuccess = function (event) {
      const image = event.target.result;
      if (image) {
        // upload Image to server here
      } else {
        console.log("No image found with ID:", IMAGE_ID);
      }
    };
    request.onerror = () => {
        console.error("Error retrieving image.");
    };
  } catch (error) {
    console.error("Failed to open database:", error);
  }
}

Deleting The IndexedDB Database

Once the image has been uploaded, the IndexedDB store is no longer needed. Therefore, it should be deleted along with its content to free up storage.

function deleteDatabase() {
  // Check if there's an open connection to the database.
  if (database) {
    database.close(); // Close the database connection
    console.log("Database connection closed.");
  }

  // Request to delete the database named "myDatabase".
  const deleteRequest = indexedDB.deleteDatabase("myDatabase");

  // Handle successful deletion of the database.
  deleteRequest.onsuccess = function () {
    console.log("Database deleted successfully!");
  };

  // Handle errors that occur during the deletion process.
  deleteRequest.onerror = function (event) {
    console.error("Error deleting database:", event.target.error);
  };

  // Handle cases where the deletion is blocked (e.g., if there are still open connections).
  deleteRequest.onblocked = function () {
    console.warn("Database deletion blocked. Close open connections and try again.");
  };
}

With that, the entire process is complete!

Considerations And Limitations

While we’ve done a lot to help improve the experience by supporting offline uploads, the system is not without its limitations. I figured I would specifically call those out because it’s worth knowing where this solution might fall short of your needs.

• No Reliable Internet Connectivity Detection
  JavaScript does not provide a foolproof way to detect online status. For this reason, you need to come up with a custom solution for detecting online status.
• Chromium-Only Solution
  The Background Sync API is currently limited to Chromium-based browsers. As such, this solution is only supported by Chromium browsers. That means you will need a more robust solution if you have the majority of your users on non-Chromium browsers.
• IndexedDB Storage Policies
  Browsers impose storage limitations and eviction policies for IndexedDB. For instance, in Safari, data stored in IndexedDB has a lifespan of seven days if the user doesn’t interact with the website. This is something you should bear in mind if you do come up with an alternative for the background sync API that supports Safari.

Enhancing The User Experience

Since the entire process happens in the background, we need a way to inform the users when images are stored, waiting to be uploaded, or have been successfully uploaded. Implementing certain UI elements for this purpose will indeed enhance the experience for the users. These UI elements may include toast notifications, upload status indicators like spinners (to show active processes), progress bars (to show state progress), network status indicators, or buttons to provide retry and cancel options.

Wrapping Up

Poor internet connectivity can disrupt the user experience of a web application. However, by leveraging PWA technologies such as IndexedDB, service workers, and the Background Sync API, developers can help improve the reliability of web applications for their users, especially those in areas with unreliable internet connectivity.

Databases For Front-End Developers: The Rise Of Serverless Databases

 

As front-end developers, we understand the foundational role data plays in our daily jobs. It may come from an external API, a CMS, or even a spreadsheet. But god forbid we need to talk about setting up databases.

Those days are over. With serverless databases becoming popular by the day, it has never been easier to create a full-stack architecture with both vertical and horizontal scaling, high availability, and bulletproof consistency.

To fully reap the benefits of such an architecture, it’s essential to understand what decisions are made for you. In the same way that the “learn JavaScript, not a framework” mantra became popular, we also ought to understand the concepts behind database architecture in order to use them reliably. So, welcome to the first part of our “Databases for Front-end Developers” series.

This series is not going to make you an expert on distributed systems or capable of jumping into a database admin role, but it will shed some light on the concepts, terms, and acronyms you will face when getting ready to choose your next stack. See it as a primer on (serverless) databases. Hopefully, it will give you a push into the rabbit hole and make you confident in joining conversations to evaluate tradeoffs for different solutions.

Spreadsheets And Content Management Systems

What?! Spreadsheets? Well, yes. The user interface (you and I, or U and I, or UI) is quite similar to that of a database. Spreadsheets give you a table in which to store data. In some cases, they will only allow you to define specific data types per column. The familiarities are there, but spreadsheets find an abrupt end once we pop the hood.

The availability is questionable: spreadsheets are not meant to serve content, only store content. For starters, they will not fuel an app as it scales, and they may not obey certain best practices when it comes to assuring data integrity. Up to very recently, they were the quickest way to get started with some data layer. But now, there is no point for an app not to use a real (serverless) database (more on this later).

A Content Management System (CMS) is another kind of database. “Content” is a special kind of data that the CMS specializes in. It will provide the user (developer) with enough abstractions to facilitate managing such data to a point where the underlying database is not a concern. It will handle the deliverability, availability, and integrity of your data. But the heavier the abstraction is, the higher the tradeoff. The data types are limited to what the CMS will give you, with most even imposing their own architecture for handling relations, queries, types, etc. Of course, there are still significant and viable use cases for CMSs, and they aren’t going anywhere. So, as long as you’re sure that’s your use case, you’ll be fine with one.

Growth Pains

If you choose the simpler, “abstractionful” route of a spreadsheet or a CMS as your source of truth and your data begins to diversify, obstacles will show up. The first issue with a spreadsheet is usually about the underlying API, it’s often not intended for most average-sized apps’ traffic, and then there are the first refactoring conversations.

With a CMS, APIs are usually not the problem, but managing the data can be. As an app grows and data diversifies, some of it ends up not being content anymore and may be more related to application logic.

When data is not content, managing it in a CMS is not ideal. It’s less flexible and often doesn’t fit the owner-team workflow. Now, while it is perfectly possible for other databases and CMSs to coexist, it’s up to the developers to understand the pros and cons of each solution and decide what is best for their app’s delivery and user experience.

More after jump! Continue reading below ↓

Meet Image Optimization, Addy Osmani’s brand new practical guide to optimizing and delivering high-quality images on the web. From formats and compression to delivery and maintenance: everything in one single 528-pages book.

Jump to table of contents ↬

Database Admin Is Hard

As front-end developers, the first time we talk about databases is usually a conversation about “relational vs. non-relational.” From then on, while trying to figure out the differences, we loosely hear a myriad of terms, such as ACID, BASE, and even CAP Theorem. This article will skip a thorough explanation of these differences. We will look better into them in the next part of this series. For now, it is sufficient to say “non-relational” databases impose eventual consistency on an app.

Eventual consistency can also be unwrapped into a longer discussion, but let’s take it as this:

Eventual consistency means that in certain special conditions, the data received is stale.

Like comments in a blog post, they won’t affect your app if a few seconds after a write you still don’t see the latest one. But password updates need to be strongly consistent always, not eventually consistent.

Of course, those are not the only differences. Query performance is different between each type of database. One can imagine being eventually consistent allows for quicker reads because there is less assurance involved.

More Growth Pains

Once the database is decided, the app can grow steadily and smoothly for a while. As an app gets big, data complexity grows, and as data complexity grows, the database becomes slower. At scale, how do we make a database faster?

• Do you add more resources to a single server? (vertical scale)
• How do you replicate data across a cluster of machines?
  • Do you split your database into smaller partitions (shards) instead? (horizontal scale, more about this in part 2)
• Do you add a faster in-memory database in front of it for common queries? (key-value store)

Those are not easy questions to answer. It depends on the user base, the type of data, the amount, frequency, and origin of queries. Is your database read-heavy or write-heavy? And though there is a multitude of factors impacting this decision, there’s also a high cost attached to making the wrong choice.

Additionally, some use cases may even require searching through data easier from user-land. A search engine is not an easy problem to solve and often requires an additional type of database to properly index your data (if sharded, it’s even harder). Having all this around your user’s data also brings a whole set of tools around it just to make it maintainable.

Even more, keeping an eye on our databases (now “data infrastructure” if we’ve got a search engine in the mix) requires a high level of observability and OLAP (Online Analytical Processing). This introduces a whole new level of complexity!

As you may have noticed, very high stakes are associated with creating, maintaining, and growing a database. Decisions that can make or break an app, decisions that are costly to go back on, and that must be made relatively early.

Serverless Databases Are Fun

Because of all the complexity mentioned above, many investors and incubators have their eyes turned to startups creating serverless databases. They are a whole new category of databases. The concepts of traditional ones still apply, but differently.

Serverless Databases

To understand what a “serverless database” really is, we first need to deconstruct the term. It is a common joke that “serverless” is a misnomer. Still, the point of a serverless architecture is to abstract away from the consumer (developer) the complexity of handling site reliability and server maintenance provided by a serverless vendor, such as Netlify, Vercel, Amazon Web Services (AWS), and so many others. I tend to like Xata’s definition of “serverless database”.

A “serverless database” does for databases what serverless does for servers. The complexity is lifted away (to different degrees depending on the chosen platform). Some, like Supabase and Firebase, will offer a multitude of serverless related features to couple with your database; others, like AWS Aurora or PlanetScale, focus on making it easier to use and scale PostgreSQL and MySQL DBs. And finally, there are others that abstract the database entirely, like Xata. They provide you with an ORM-like SDK, keep the database behind an API, and are able to offer a complex set of database features, bending the current limitations of traditional relational and non-relational databases.

Once we get to the next part of this series, we will talk about different kinds of databases. Then you will be ready to pop the hood on any serverless database offering you want and understand the differences for yourself. Meanwhile, let’s keep it superficial.

Batteries Included

Don’t take the “serverless” prefix lightly; these databases are from a different breed. They are able to offer guarantees and performance that “traditional” databases require some effort to reach, sometimes not even so. This is because on serverless databases, the work has been done, just not by your team.

The same way “serverless” means you don’t need to handle your server, “serverless database” means you don’t need to handle your database. The platform will handle it for you.

“

Because of this, the decisions about scalability and deliverability are often made external to your team. What your team gets is the assurance that any request will receive a response in a timely manner and that data will respect the consistency guarantees. Again, different solutions have different tradeoffs. It’s important to check what each offering imposes before jumping in.

See You On The Next One 

Hopefully, this has been enough to spark your curiosity. This is the first article of a 3-part series. In the next ones, we will cover more in-depth information about what databases actually are. Specifically, we’ll look into:

• Schemas,
• Theorems and models,
• Types of databases,
• whatever you suggest in the comments below!

All that necessary knowledge will enable you to choose the best solution for your app. Understanding the tradeoffs of different serverless solutions and surrounding yourself with the right kind of help is crucial to setting your app up for success. Reach out to me if you need anything meanwhile. Otherwise, see you in a few days!

Databases For Front-End Developers: The Concepts Under The Hood

 we talked about the hurdles and traps of scaling and maintaining your databases. We went from simpler and specialized alternatives like Content Management Systems and spreadsheets to self-hosted databases and, finally, to Serverless Databases.

Today, we go deeper into the rabbit hole. We will explore concepts to equip you to have your own opinions about which kinds of databases suit your specific needs. And this is important to stress up front: there is no right answer. Each database carries its own set of tradeoffs and advantages. If something looks like a “one-size-fits-all” solution, be careful: you might be missing something!

Anatomy Of A Database

Before we begin, it’s important to highlight that what we loosely refer to as “databases” are actually “Database Management Systems (DBMS).” A DBMS is a piece of software that enables the user to more ergonomically write, read, delete, or update information in a given set of data. For this series, we will focus mainly on Relational and Non-Relational DBMSs. There are many other types, all categorized by their data structures, but relational and non-relational are the most common for web development by any measure.

Both Relational (R) and Non-Relational (NR) DBMS have different terms for the parts that compose them. Such components are almost interchangeable in definition, and that’s why you can commonly hear a developer referring to a Document (NR term) as “Table,” which is its Relational equivalent structure. Don’t be afraid of confusing them; they appear often enough for this cognitive overload to disappear quickly with usage. Additionally, once you get more familiar with the differences of each data structure, you will realize they probably shouldn’t be used interchangeably because there are differences among them. But for now, and for the sake of simplicity, let’s focus on the similarities:

• Schema
  It’s the description of the database (like a blueprint), describing the landscape of the database in a supported language. This is required by relational databases, and though not required by non-relational DBMS, many interfaces offer a possibility to define it too.
• Tables (R/NR), Collections (NR)
  It’s the logical data structure, unsorted. A relatable example of this would be a spreadsheet table or a group (collection) of JSON objects.
• Databases (R/NR)
  It’s the logical grouping of data. You group your tables in databases (user database, invoices databases) and your documents in indexes based on how you intend to query them.

Keys And Columns

To use a more familiar example, let’s take a JSON object as an example:

{
  "name": "Atila"
  "role": "DX Engineer at Xata"
}

Given that JSON, a column would be represented by the key-value pair (column name has value “Atila”), and the key follows the same meaning as the JSON spec, key name will access the value “Atila.”

A table has columns, and each column’s name determines the key with which you can access a value in a record.

In addition to the above definition, there are special kinds of keys. Such keys play a special role in the schema of your database and how you will interact with your data. Define these wisely: any minimal change to them can be considered a breaking change to your data layer.

• Unique Keys (UK)
  The keys that are unique between records on a table. They also accept null.
• Primary Keys (PK)
  It’s a special kind of Unique Key. There can only be one Primary Key per table, and it can never be null (Primary Keys are always considered required). Primary Keys are also indexed by default, helping with queries that want to filter on the value.
• Foreign Keys (FK)
  When there is a relation between tables, the keys are represented as foreign keys at the extraneous table.

Example of Foreign Key: if each author in an authors table has posts (and posts is another table), post.title will be a FK at authors. And the junction between authors and posts is called a relation.

Mapping Your DBMS

Once you look at the different data structures and choose your DBMS, you are ready to draw the first connection from your data layer to your application layer. And suddenly, you noticed it isn’t quite straightforward to bring data from the database to your client-side (or even to your server-side API in some cases).

Here comes ORM (Object Relational Mapping) and ODM (Object Document Mapping) to assist your developer experience. Prisma is probably the most widely used ORM at the moment, while Mongoose has the largest ODM user base. It’s important to note they are not a requirement for connecting to databases. Still, if you keep an eye on how they build your queries (some specific cases can present performance issues on the account of the abstraction), they tend to make your life easier and fetching or writing data much more ergonomic.

When it comes to serverless databases, the need for them becomes a bit more questionable. And this is because many of these databases provide the users with an officially supported Software Development Kit (SDK). Your mileage will vary depending on the SDK, but they tend to have a big feature overlap with ORMs and ODMs, especially on databases that will keep the data layer behind an API (Xata, for example). This way, you won’t need to worry about translating your queries, and you can demand equivalent ergonomics between the SDK and an ORM.

Common Concepts

In this part of our series, we are learning what to look for when choosing the stack for our data layers. It is essential to understand common concepts around maintaining and choosing a DBMS (from here on out, we’re back into referring to them as “databases” to remain consistent with the rest of the world).

The next sections will not go so deep that you jump out and find a job as Database Administrator (DBA), but hopefully, it will offer you enough ammo to engage in conversation with experts and identify the best solution for your use cases. These concepts are common for every kind of data layer, from a spreadsheet to a self-hosted database and even to a serverless database. What will vary is how each solution will balance the variables intertwined in these paradigms.

As Thanos (from Marvel: Infinity Wars) would say: “perfectly balanced, like all things should be.”

The most important concepts, to begin with, are Consistency, Availability, and Partition Tolerance. They’re better understood if presented together because the balance between them will guide how predictable your data is in different contexts.

CAP Theorem

This theorem describes the relationship between 3 components in a distributed system: Consistency, Availability, and Partition tolerance (C.A.P). The overall conclusion is easy to summarize: any system is only able to contemplate 2 of these components at the same time. Though just a simple sentence, this idea requires a bit of unpacking.

Consistency (C)

Within the CAP Theorem constraints, “consistency” refers directly to the data. When different clients make the same request, they will get the same response. When a written request is accepted and confirmed, all users will have access to this updated information at the same time.

Availability (A)

Every request will receive a response with data. No errors. However, this comes with no promises on whether the data is up to date or not. Depending on whether this is paired with consistency © or partition tolerance (P), you will get different behavior.

Partition Tolerance (P)

A “partition” is when the connection between two nodes in a system is broken. The CAP Theorem essentially describes how a system will tolerate the partition: enforcing availability (AP system) or enforcing consistency (CP system). Regardless of how small a system is, partitions can always happen. Therefore there is no such thing as a CA system.

To provide a more graphical example of each system, we can consider:

• AP System
  Another node has a copy of the data. The user will get information back but without promises of it being 100% correct (up-to-date). It’s commonly implemented with DynamoDB, Cassandra, and so on.
• CP System
  To return an error and accept the transaction is impossible. It’s commonly implemented with traditional sharded DBs, Citus, for example.

Though popular and very often referred to, the CAP Theorem can be considered incomplete because it doesn’t consider situations beyond the network partition. Especially today, with high-availability content delivery networks (CDN), and extreme connectivity, it’s crucial to take into consideration latency and deeper aspects of consistency (linearizability and serializability).

Consistency Confusion

It’s funny how “consistency” is a term that switches meanings based on the context. So, in the CAP Theorem, as we just saw, it’s all about data and how reliably a user will get the same response to the same request regardless of which partition they reach. Once we take a step away from our own system, a new perspective makes us ask: “how consistently will we handle concurrent operations?”. And just like that, the CAP Theorem does not sufficiently describe the intricacies of operating at scale.

There is a third meaning of “consistency,” which we will save for later; it’s part of yet another acronym (ACID). If the last paragraph left you scratching your head, I can’t recommend enough “Inconsistent Thoughts on Database Consistency” by Alex DeBrie.

PACELC Theorem

The first three letters are the same as CAP, just reordered. The “consistency” in the PACELC Theorem goes deeper than in the CAP Theorem. It follows the Consistency Model, which determines the contract between a data store and a system. To make things less complicated, consider the PACELC an extension of the CAP Theorem.

Beyond asking the developer to strategize for the event of a network partition, the PACELC also considers what happens when the system has no partitions (healthy network).

ELC stands for Else: Latency or Consistency?

• Latency: Can you accept an occasional stale response for the sake of performance?
• Consistency:
  • Linearizability: Will you accept a higher latency to sync data in all nodes of a network before considering the transaction done?
  • Serializability: In the event of concurrent transactions on the same data, will you handle them in parallel or in a queue?

Because of this, I consider the PACELC carries a better mental model for this new era of Serverless Databases. When healthy, it’s not a scalar classification “AP” or “CP”; it accepts a spectrum because latency can be high or not, and consistency can have different levels as well.

Once we start talking about the types of databases and their data structures and which guarantees each can give, we will also talk about a bit of system architecture and how your architecture can reduce the tradeoffs when even by enforcing consistency, you can strive for a low latency scenario.

See You Next Time

With that, I believe we are ready to start narrowing down our discussions on the differences between each database type. Moreover, it’s possible to discuss and level expectations on each solution taken and analyze architectures individually. From now on, we will focus on Relational and Non-Relational databases.

In a few days, on our season finale, we will cover the differences between NoSQL and SQL: what guarantees to expect, what that means to your data, and how that affects development workflow. Then we will be ready to jump right into Serverless Databases and what to expect going forward. I can’t wait!

Demystifying The New Gatsby Framework

 Gatsby 4 came out in October 2021 with a lot of new features. It may seem as if Gatsby 4 didn’t cause a big stir among developers, but it brought several game-changing features. Let’s see what v4 brings to the table.

After the release of Gatsby 4, the Gatsby team saw the biggest rise in signups on Gatsby Cloud. According to Gatsby co-founder Kyle Mathews:

“Gatsby 4 is the most powerful version of Gatsby yet. We’ve made Gatsby 10x faster to build and deploy by opening up two new rendering modes, adding support for Parallel Query Processing, and making a host of other optimizations and improvements.”

— Kyle Mathews

Until a few months ago, Gatsby.js was best suited to make not-so-big Jamstack websites, but with the release of Gatsby 4, how well does Gatsby work with big and rapidly-changing pages? And how will it compare to other frameworks like Next.js? We will hopefully answer these questions in the next article!

New Gatsby 4 Features #

Gatsby 3 had a lot of useful features, and it stood out among other pre-rendering frameworks due to being extremely fast when using Static Site Generation. However, it lacked the tools of other frameworks — mainly Next.js — to render pages on request using Server-Side Rendering. Gatsby heard its community, and besides a lot of optimization features, Gatsby 4 brought two more rendering options: Deferred Static Generation (DSG) and the good old Server-Side Rendering (SSR).

Deferred Static Generation (DSG) #

One of the problems that a lot of people had with Gatsby was its very long build times. When using SSG, building the pages and all the heavy lifting had to be done on the server, which led to longer build times as pages in your site grew. To solve this, Gatsby started to use incremental building only to build new or changed pages and keep the other ones in the cache. However, the first build without cache always took a long time to finish. The incremental building is very helpful, but it didn’t tackle the roots of the problem, and that’s where Deferred Static Generation comes in.

Deferred Static Generations is very similar to what SSG achieves, but it is more efficient and has a different method of delivering pages. In DSG, you can choose pages to “delay,” which means not to build with the rest of the pages but once a user requests them. After the first request, the page will be available in the CDN and will be the same as any other page generated with SSG. This is very useful when a site has old and unvisited pages that aren’t worth building on each production build.

If you are worried that DSG can tank your page’s SEO, remember that pages built using DSG only behave as a Server-Side Rendered the first time they are requested, and then they act just like any other SSG page. So, according to Gatsby, your page will act as an SSG 99.9% of the time, and Google will analyze its SEO that way.

As you may notice, DSG requires a running server to listen for user requests and build the requested page for every other user onwards. It is a game-changing feature in Gatsby since Gatsby was known for being a server-less SSG framework, but now you can combine SSG with other rendering options involving a server. Even though it changes the backend structure — only if you use DSG or SSR — it gives developers more flexibility to work with Gatsby.

Server-Side Rendering (SSR) #

If DSG uses a server to work, does that mean that Gatsby can use a server to render all pages on request time? Absolutely yes. This was the most long-awaited feature in Gatsby and the only one that made a lot of developers choose other pre-rendering frameworks over Gatsby. In case you don’t know, Server-Side Rendering consists in building a page each time a user requests it. And it’s very useful when the page content depends on the user that requested it, so we can deliver a static and customized page to each user.

SSR is a well-known feature used daily in frameworks like Next.js, which Gatsby finally implemented. Now developers using Gatsby have an enormous range of options to deliver pages to users, either with Static Site Generation, Deferred Site Generation, Server-Side Rendering, or even — if you are brave enough — with Client-Side Rendering.

Parallel Query Running #

Besides the new rendering options, Gatsby still brings a lot of other features to improve performance and developer experience. As previously mentioned, one of the major drawbacks JAMstack and static sites had was their long building times. In addition to incremental building and DSG, Gatsby 4 introduces Parallel Query Running to diminish build time even more.

As you may know, Gatsby uses a data layer to group external data and then query it onto your site. Sourcing the data from your content sources into the data layer is relatively easy and fast. However, the process of querying the data from the data layer and querying it into the pages is pretty time-consuming. This step is called query running, which happens in the middle of the build and is the longest by far, taking around 40% of the build time.

Query running only was able to use one of the cores of the CPU, so all the queries had to be done one at a time. But in Gatsby 4, the entire way data was handled was changed since it migrated from Redux to its own in-memory data store and switched to lmdb-js.

“Lmdb-js is an ultra-fast NodeJS and Deno interface to LMDB; probably the fastest and most efficient key-value/database interface that exists for storage and retrieval of structured JS data.”

With this new architecture, all queries can be done across all available cores, which leads to immensely faster build times.

Build Time Optimization #

Lastly, both a little and big change that may pass unnoticed between all the prior features is all the configurations and tweaks made to improve build times. One of the major reasons build times and even developer server starts were slow was due to an incorrect configuration of the local development environment or when Gatsby’s sites were deployed anywhere other than Gatsby Cloud.

The Gatsby team has changed several critical parts of Gatsby’s infrastructure across Gatsby 3 minor updates and is now in Gatsby 4 to reduce build and development times. I consider reducing development times a key change to enhance developer experience since it becomes tedious to restart and wait for the developer server each time you make a change that requires a restart to take effect.

More after jump! Continue reading below ↓

Meet Smashing Email Newsletter with useful tips on front-end, design & UX. Subscribe and get “Smart Interface Design Checklists” — a free PDF deck with 150+ questions to ask yourself when designing and building almost anything.

Your (smashing) email

Once a week. Useful tips on front-end & UX. Trusted by 190.000 friendly folks.

Gatsby Cloud #

What’s Gatsby Cloud #

Even though Gatsby is known for being a framework, the Gatsby team offers other great products like Gatsby Cloud. Gatsby Cloud is a deployment service with an edge network dedicated uniquely to deploying Gatsby sites in a matter of minutes. It has been around for a long time and is very similar to Netlify. The service has been optimized only to build and host Gatsby sites, which means you can’t host Next.js, Vue, Angular, or Vanilla JS sites, but in exchange, Gatsby Cloud offers greatly faster build times for Gatsby sites.

Configuration #

The main advantage of using Gatsby Cloud to host your Gatsby project is the little to no configuration you need to start hosting it. It can host and run a Gatsby Project without configuration and supports all Gatsby’s new rendering features and more out of the box. As long as they are declared in your code, the server won’t have problems understanding and hosting your project. So, if you want to migrate from Gatsby 3 to 4, you may consider switching to Gatsby Cloud for your migration.

Build And Deployment Time #

As said before, a lot of people didn’t prefer Gatsby due to its long build times. But Gatsby Cloud brings even more features to make build times faster, and it has been configured to use all optimization features like incremental buildings, parallel querying, caching, and support pages using DSG to the point where a CMS update from Contentful on Gatsby’s 10,000-page site took around 10 seconds on average.

Besides build time, another aspect that has been drastically improved with Gatsby Cloud is deployment time. Deployment time is the time it takes to push your site onto the edge of the CDN, and it accounts for the final time between triggering a build and releasing your site to the world. It can take as much time as build time, to the point that deployment times for a medium-size static site of around 10,000 pages can take 2 to 10 minutes. But the same medium-size site takes, on average, a surprising 2-5 seconds in Gatsby Cloud.

Workflow #

Similar to Netlify, Gatsby Cloud workflow is based on Github, Gitlab, or BitBucket repositories. Gatsby Cloud fetches a Gatsby project from a repository, builds the site, and/or runs the server. And like many other services, you can enable automatic deployment when a branch changes in your repository. In case external data from a source like a CMS changes, Gatsby Cloud offers a build webhook to trigger a production build.

Other small workflow features I really liked were:

• Previewing
  It lets you create a preview site with all your changes and a shareable URL.
• Lighthouse Support
  It gives you a Lighthouse report every time a build is finished.

Gatsby Functions #

Lastly, we will see Gatsby’s serverless functions in action. They aren’t completely part of Gatsby Cloud since they can be deployed in Netlify too, but deploying Gatsby Functions is more straightforward in Gatsby Cloud. Now, what are Gatsby’s Functions? In simpler terms, they let you add an express-like backend to your Gatsby project to create an API, authenticate users, or submit forms.

Why would I want to create a backend in my Gatsby project? Most Gatsby use cases don’t require creating a backend or API. Still, in cases where there is more than one frontend application using the same data, it is a very good idea to decouple your data from your main frontend and make it accessible through an API, so it is accessed the same way with the same logic through all frontends. The power of Gatsby Functions is to have your backend hosted in the same place as your frontend, but they communicate headlessly through an API.

Gatsby Cloud vs. Netlify #

After seeing all the features offered by Gatsby Cloud, there isn’t a decisive factor in choosing one over another. Both of them offer pretty similar services and functionalities, and they come in a free tier, too. The main difference is that you can only host a Gatsby project in Gatsby Cloud, but with Netlify, you can host projects using almost any library or framework you want, such as Next.js, create-react-app, Vue, Angular, etc. So, in case you want to host a Gatsby project, I would recommend Gatsby Cloud since it is amazingly optimized and guarantees good results.

• 
  100+ fully native Angular components trusted by over 3M developers.
  Start free Trial

Gatsby 4 vs. Next.js #

In the SSR ecosystem, the clear dominant player has been Next.js, and many comparisons between Gatsby and Next.js ended by saying that Next.js offered SSR and Gatsby did not. And even though Gatsby was still extremely useful and effective, the fact that it didn’t support SSR was a decisive factor when choosing a framework to use. But the introduction of Gatsby 4 changed the game completely.

Since this article is dedicated to Gatsby’s new features, I will try not to focus too much on Next.js features. Still, I will say that Next.js and Gatsby currently offer similar functionality, and we can achieve almost the same results with both frameworks. However, there are still a lot of factors and features to consider when choosing one framework or the other. Gatsby offers features that Next.js doesn’t, like Deferred Static Generation, a data layer, and an enormous plugin library. On the same note, Next.js offers Incremental Static Regeneration and Server Components which is currently on alpha.

You can achieve the same user experience with both frameworks, and they will help you make beautiful websites. So, to choose a framework, I strongly believe the developer experience is the decisive factor. I feel Gatsby offers a better DX due to its awesome data layer that lets developers source their data and keeps it all in the same place, so it can be queried the same way. Another reason is its plugin library, which lets you add out-of-the-box support to CMS, internalization, e-commerce, analytics, offline support, et cetera. These are life-saving features that have helped me quickly start working on a new project.

Conclusion #

After seeing all the new features of Gatsby 4, we can confidently say that the future of Gatsby is already looking great! It is clear that the Gatsby team has been pretty self-critical and has heard its community since all Gatsby 4 features were wanted by the community for a long time. Therefore, I am pretty confident that Gatsby will keep following this path, and we will have many other cool features in the future.

Even though I might sound like a Gatsby salesman, I don’t consider it the best framework, just one that is completely worth experimenting with. Next.js is vastly more used among the SSR ecosystem, and the community believes it gives a better developer experience than Gatsby, but with Gatsby 4 this trend may change a little. It isn’t putting Next.js out of business any time soon, but thanks to its new rendering options, Gatsby 4 will clearly be on your list of frameworks to try out on your next project.

The Case For Prisma In The Jamstack

  Jamstack has surged in popularity over the past few years as an approach to building websites. As developers are building Jamstack sites, finding a well-integrated method to interact with a database can be a major stumbling block. In this article, Sam Poder explores how Prisma integrates with the Jamstack and why it’s a great solution for Serverless databases in JavaScript or TypeScript-based projects.

The Jamstack approach originated from a speech given by Netlify’s CEO Matt Biilmann at Smashing Magazine’s very own Smashing Conf in 2016.

Jamstack sites serve static pre-rendered content through a CDN and generate dynamic content through microservices, APIs & serverless functions. They are commonly created using JavaScript frameworks, such as Next.js or Gatsby, and static site generators — Hugo or Jekyll, for example. Jamstack sites often use a Git-based deployment workflow through tools, such as Vercel and Netlify. These deployment services can be used in tandem with a headless CMS, such as Strapi.

The goal of using Jamstack to build a site is to create a site that is high performant and economical to run. These sites achieve high speeds by pre-rendering as much content as possible and by caching responses on “the edge” (A.K.A. executing on servers as close to the user as possible, e.g. serving a Mumbai-based user from a server in Singapore instead of San Francisco).

Jamstack sites are more economical to run, as they don’t require using a dedicated server as a host. Instead, they can provision usage from cloud services (PAASs) / hosts / CDNs for a lower price. These services are also set up to scale in a cost-efficient manner, without developers changing their infrastructure and reducing their workload.

The other tool that makes up this combination is Prisma — an open source ORM (object relational mapping) built for TypeScript & JavaScript.

Prisma is a JavaScript / TypeScript tool that interpretes a schema written in Prisma’s standards and generates a type-safe module that provides methods to create records, read records, update records, and delete records (CRUD).

Prisma handles connections to the database (including pooling) and database migrations. It can connect with databases that use PostgreSQL, MySQL, SQL Server or SQLite (additionally MongoDB support is in preview).

To help you get a sense of Prisma, here’s the some basic example code to handle the CRUD of users:

import { PrismaClient } from '@prisma/client'

const prisma = new PrismaClient()

const user = await prisma.user.create({
  data: {
    name: Sam,
    email: 'sam@sampoder.com',
  },
})

const users = await prisma.user.findMany()

const updateUser = await prisma.user.update({
  where: {
    email: 'sam@sampoder.com',
  },
  data: {
    email: 'deleteme@sampoder.com',
  },
})

const deleteUser = await prisma.user.delete({
  where: {
    email: 'deleteme@sampoder.com',
  },
})

The associated project’s Prisma schema would look like:

datasource db {
  url      = env("DATABASE_URL")
  provider = "postgresql"
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id        Int      @id @default(autoincrement())
  email     String   @unique
  name      String?
}

The Use Cases for Prisma

Armed with a knowledge of how Prisma operates, let’s now explore where we can use it within Jamstack projects. Data is important in two aspects of the Jamstack: whilst pre-rendering static pages and on API routes. These are tasks often achieved using JavaScript tools, such as Next.js for static pages and Cloudfare Workers for API routes. Admitally, these aren’t always achieved with JavaScript — Jekyll, for example, uses Ruby! So, maybe I should amend the title for the case of Prisma in JavaScript-based Jamstack. Anyhow, onwards!

A very common use-case for the Jamstack is a blog, where Prisma will come in handy for a blog to create a reactions system. You’d use it in API routes with one that would fetch and return the reaction count and another that could register a new reaction. To achieve this, you could use the create and findMany methods of Prisma!

Another common use-case for the Jamstack is a landing page, and there’s nothing better than a landing with some awesome stats! In the Jamstack, we can pre-render these pages with stats pulled from our databases which we can achieve using Prisma’s reading methods.

Sometimes, however, Prisma can be slightly overkill for certain tasks. I’d recommend avoiding using Prisma and relational databases in general for solutions that need only a single database table, as it adds additional and often unnecessary development complexity in these cases. For example, it’d be overkill to use Prisma for an email newsletter signup box or a contact form.

Alternatives to Prisma #

So, we could use Prisma for these tasks, but we could use a plethora of other tools to achieve them. So, why Prisma? Let’s go through three Prisma alternatives, and I’ll try to convince you that Prisma is preferable.

Cloud Databases / Services

Services like Airtable are incredibly popular in the Jamstack space (I myself have used it a ton), they provide you with a database (like platform) that you can access through a REST API. They’re good fun to use and prototype with, however, Prisma is arguably a better choice for Jamstack projects.

Firstly, with cost being a major factor in Jamstack’s appeal, you may want to avoid some of these services. For example, at Hack Club, we spent $671.54 on an Airtable Pro subscription last month for our small team (yikes!).

On the other hand, hosting an equivalent PostgreSQL database on Heroku’s platform costs $9 a month. There certainly is an argument to make for these cloud services based on their UI and API, but I would respond by pointing you to Prisma’s Studio and aforementioned JavaScript / TypeScript client.

Cloud services also suffer from a performance-issue, especially considering that you, as the user, have no ability to change / improve the performance. The cloud services providing the database put a middleman in between your program and the database they’re using, slowing down how fast you can get to the database. However, with Prisma you’re making direct calls to your database from your program which reduces the time to query / modify the database.

Writing Pure SQL

So, if we’re going to access our PostgreSQL database directly, why not just use the node-postgres module or — for many other databases — their equivalent drivers? I’d argue that the developer experience of using Prisma’s client makes it worth the slightly increased load.

Where Prisma shines is with its typings. The module generated for you by Prisma is fully type-safe — it interprets the types from your Prisma schema — which helps you prevent type errors with your database. Furthermore, for projects using TypeScript, Prisma auto-generates type definitions that reflect the structure of your model. Prisma uses these types to validate database queries at compile-time to ensure they are type-safe.

Even if you aren’t using TypeScript, Prisma also offers autocomplete / Intelli-sense, linting, and formatting through its Visual Studio Code extension. There are also community built / maintained plugins for Emacs (emacs-prisma-mode), neovim (coc-prisma), Jetbrains IDE (Prisma Support), and nova (the Prisma plugin) that implement the Prisma Language Server to achieve code validation. Syntax highlighting is also available for a wide array of editors through plugins.

Other ORMs

Prisma is, of course, not the only ORM available for JavaScript / TypeScript. For example, TypeORM is another high quality ORM for JavaScript projects. And in this case, it is going to come down to personal preference, and I encourage you to try a range of ORMs to find your favourite. I personally choose Prisma to use for my project for three reasons: the extensive documentation (especially this CRUD page, which is a lifesaver), the additional tooling within the Prisma ecosystem (e.g. Prisma Migrate and Prisma Studio), and the active community around the tool (e.g. Prisma Day and the Prisma Slack).

Using Prisma in Jamstack Projects

So, if I’m looking to use Prisma in a Jamstack project, how do I do that?

Next.js #

Next.js is growing to be a very popular framework in the Jamstack space, and Prisma is a perfect fit for it. The examples below will serve as pretty standard examples that you can transfer into other projects using different JavaScript / TypeScript Jamstack tools.

The main rule of using Prisma within Next.js is that it must be used in a server-side setting, this means that it can be used in getStaticProps, getServerSideProps, and in API routes (e.g. api/emojis.js).

In code, it looks like this (example taken from a demo app I made for a talk at Prisma Day 2021 which was a virtual sticker wall):

import prisma from '../../../lib/prisma'
import { getSession } from 'next-auth/client'

function getRandomNum(min, max) {
  return Math.random() * (max - min) + min
}

export async function getRedemptions(username) {
  let allRedemptions = await prisma.user.findMany({
    where: {
      name: username,
    },
    select: {
      Redemptions: {
        select: {
          id: true,
          Stickers: {
            select: { nickname: true, imageurl: true, infourl: true },
          },
        },
        distinct: ['stickerId'],
      },
    },
  })
  allRedemptions = allRedemptions[0].Redemptions.map(x => ({
    number: getRandomNum(-30, 30),
    ...x.Stickers,
  }))
  return allRedemptions
}

export default async function RedeemCodeReq(req, res) {
  let data = await getRedemptions(req.query.username)
  res.send(data)
}

As you can see, it integrates really well into a Next.js project. But you may notice something interesting: '../../../lib/prisma'. Previously, we imported Prisma like this:

import { PrismaClient } from '@prisma/client'

const prisma = new PrismaClient()

Unfortunately, this is due to a quirk in Next.js’ live refresh system. So, Prisma recommends you paste this code snippet into a file and import the code into each file.

Redwood

Redwood is a bit of an anomaly in this section, as it isn’t necessarily a Jamstack framework. It began under the banner of bringing full stack to the Jamstack but has transitioned to being inspired by Jamstack. I’ve chosen to include it here, however, as it takes an interesting approach of including Prisma within the framework.

It starts, as always, with creating a Prisma schema, this time in api/db/schema.prisma (Redwood adds this to every new project). However, to query and modify the database, you don’t use Prisma’s default client. Instead, in Redwood, GraphQL mutations and queries are used. For example, in Redwood’s example todo app, this is the GraphQL mutation used to create a new todo:

const CREATE_TODO = gql`
  mutation AddTodo_CreateTodo($body: String!) {
    createTodo(body: $body) {
      id
      __typename
      body
      status
    }
  }
`

And in this case, the Prisma model for a todo is:

model Todo {
  id     Int    @id @default(autoincrement())
  body   String
  status String @default("off")
}

To trigger the GraphQL mutation, we use the useMutation function which is based on Apollo’s GraphQL client imported from @redwoodjs/web:

const [createTodo] = useMutation(CREATE_TODO, {
    //  Updates Apollo's cache, re-rendering affected components
    update: (cache, { data: { createTodo } }) => {
      const { todos } = cache.readQuery({ query: TODOS })
      cache.writeQuery({
        query: TODOS,
        data: { todos: todos.concat([createTodo]) },
      })
    },
  })

  const submitTodo = (body) => {
    createTodo({
      variables: { body },
      optimisticResponse: {
        __typename: 'Mutation',
        createTodo: { __typename: 'Todo', id: 0, body, status: 'loading' },
      },
    })
  }

With Redwood, you don’t need to worry about setting up the GraphQL schema / SDLs after creating your Prisma schema, as you can use Redwood’s scaffold command to convert the Prisma schema into GraphQL SDLs and services — yarn rw g sdl Todo, for example.

Cloudfare Workers

Cloudfare Workers is a popular platform for hosting Jamstack APIs, as it puts your code on the “edge”. However, the platform has its limitations, including a lack of TCP support, which the traditional Prisma Client uses. Though now, through Prisma Data Proxy, it is possible.

To use it, you’ll need a Prisma Cloud Platform account which is currently free. Once you’ve followed the setup process (make sure to enable Prisma Data Proxy), you’ll be provided with a connection string that begins with prisma://. You can use that Prisma connection string in your .env file in place of the traditional database URL:

DATABASE_URL="prisma://aws-us-east-1.prisma-data.com/?api_key=•••••••••••••••••"

And then, instead of using npx prisma generate, use this command to generate a Prisma client:

PRISMA_CLIENT_ENGINE_TYPE=dataproxy npx prisma generate

Your database requests will be proxied through, and you can use the Prisma Client as usual. It isn’t a perfect set-up, but for those looking for database connections on Cloudfare Workers, it’s a relatively good solution.

Conclusion

To wrap up, if you’re looking for a way to connect Jamstack applications with a database, I wouldn’t look further than Prisma. Its developer experience, extensive tooling, and performance make it the perfect choice. Next.js, Redwood, and Cloudfare Workers — each of them has a unique way of using Prisma, but it still works very well in all of them.

I hope you’ve enjoyed exploring Prisma with me. Thank you!