Wrapping Things with HTML5 Local Storage

HTML5 is here to turn the web from a web of hacks into a web of applications – and we are well on the way to this goal. The coming year will be totally and utterly awesome if you are excited about web technologies.
This year the HTML5 revolution started and there is no stopping it. For the first time all the browser vendors are rallying together to make a technology work. The new browser war is fought over implementation of the HTML5 standard and not over random additions. We live in exciting times.

Starting with a bang

As with every revolution there is a lot of noise with bangs and explosions, and that’s the stage we’re at right now. HTML5 showcases are often CSS3 showcases, web font playgrounds, or video and canvas examples.
This is great, as it gets people excited and it gives the media something to show. There is much more to HTML5, though. Let’s take a look at one of the less sexy, but amazingly useful features of HTML5 (it was in the HTML5 specs, but grew at such an alarming rate that it warranted its own spec): storing information on the client-side.

Why store data on the client-side?

Storing information in people’s browsers affords us a few options that every application should have:

• You can retain the state of an application – when the user comes back after closing the browser, everything will be as she left it. That’s how ‘real’ applications work and this is how the web ones should, too.
• You can cache data – if something doesn’t change then there is no point in loading it over the Internet if local access is so much faster
• You can store user preferences – without needing to keep that data on your server at all.

In the past, storing local data wasn’t much fun.

The pain of hacky browser solutions

In the past, all we had were cookies. I don’t mean the yummy things you get with your coffee, endorsed by the blue, furry junkie in Sesame Street, but the other, digital ones. Cookies suck – it isn’t fun to have an unencrypted HTTP overhead on every server request for storing four kilobytes of data in a cryptic format. It was OK for 1994, but really neither an easy nor a beautiful solution for the task of storing data on the client.
Then came a plethora of solutions by different vendors – from Microsoft’s userdata to Flash’s LSO, and from Silverlight isolated storage to Google’s Gears. If you want to know just how many crazy and convoluted ways there are to store a bit of information, check out Samy’s evercookie.
Clearly, we needed an easier and standardised way of storing local data.

Keeping it simple – local storage

And, lo and behold, we have one. The local storage API (or session storage, with the only difference being that session data is lost when the window is closed) is ridiculously easy to use. All you do is call a few methods on the window.localStorage object – or even just set the properties directly using the square bracket notation:

1. if('localStorage' in window && window['localStorage'] !== null){
2. var store = window.localStorage;
3.  
4. // valid, API way
5. store.setItem('cow','moo');
6. console.log(
7. store.getItem('cow')
8. ); // => 'moo'
9.  
10. // shorthand, breaks at keys with spaces
11. store.sheep = 'baa'
12. console.log(
13. store.sheep
14. ); // 'baa'
15.  
16. // shorthand for all
17. store['dog'] = 'bark'
18. console.log(
19. store['dog']
20. ); // => 'bark'
21.  
22. }

Browser support is actually pretty good: Chrome 4+; Firefox 3.5+; IE8+; Opera 10.5+; Safari 4+; plus iPhone 2.0+; and Android 2.0+. That should cover most of your needs. Of course, you should check for support first (or use a wrapper library like YUI Storage Utility or YUI Storage Lite).
The data is stored on a per domain basis and you can store up to five megabytes of data in localStorage for each domain.

Strings attached

By default, localStorage only supports strings as storage formats. You can’t store results of JavaScript computations that are arrays or objects, and every number is stored as a string. This means that long, floating point numbers eat into the available memory much more quickly than if they were stored as numbers.

1. var cowdesc = "the cow is of the bovine ilk, "+
2. "one end is for the moo, the "+
3. "other for the milk";
4.  
5. var cowdef = {
6. "ilk":"bovine",
7. "legs":4,
8. "udders":4,
9. "purposes":{
10. "front":"moo",
11. "end":"milk"
12. }
13. };
14.  
15. window.localStorage.setItem('describecow',cowdesc);
16. console.log(
17. window.localStorage.getItem('describecow')
18. ); // => the cow is of the bovine...
19.  
20. window.localStorage.setItem('definecow',cowdef);
21. console.log(
22. window.localStorage.getItem('definecow')
23. ); // => [object Object] = bad!

This limits what you can store quite heavily, which is why it makes sense to use JSON to encode and decode the data you store:

1. var cowdef = {
2. "ilk":"bovine",
3. "legs":4,
4. "udders":4,
5. "purposes":{
6. "front":"moo",
7. "end":"milk"
8. }
9. };
10.  
11. window.localStorage.setItem('describecow',JSON.stringify(cowdef));
12. console.log(
13. JSON.parse(
14. window.localStorage.getItem('describecow')
15. )
16. ); // => Object { ilk="bovine", more...}

You can also come up with your own formatting solutions like CSV, or pipe | or tilde ~ separated formats, but JSON is very terse and has native browser support.

Some use case examples

The simplest use of localStorage is, of course, storing some data: the current state of a game; how far through a multi-form sign-up process a user is; and other things we traditionally stored in cookies. Using JSON, though, we can do cooler things.

Speeding up web service use and avoiding exceeding the quota

A lot of web services only allow you a certain amount of hits per hour or day, and can be very slow. By using localStorage with a time stamp, you can cache results of web services locally and only access them after a certain time to refresh the data.
I used this technique in my An Event Apart 10K entry, World Info, to only load the massive dataset of all the world information once, and allow for much faster subsequent visits to the site. The following screencast shows the difference:
For use with YQL (remember last year’s 24 ways entry?), I’ve built a small script called YQL localcache that wraps localStorage around the YQL data call. An example would be the following:

1. yqlcache.get({
2. yql: 'select * from flickr.photos.search where text="santa"',
3. id: 'myphotos',
4. cacheage: ( 60*60*1000 ),
5. callback: function(data) {
6. console.log(data);
7. }
8. });

This loads photos of Santa from Flickr and stores them for an hour in the key myphotos of localStorage. If you call the function at various times, you receive an object back with the YQL results in a data property and a type property which defines where the data came from – live is live data, cached means it comes from cache, and freshcache indicates that it was called for the first time and a new cache was primed. The cache will work for an hour (60×60×1,000 milliseconds) and then be refreshed. So, instead of hitting the YQL endpoint over and over again, you hit it once per hour.

Caching a full interface

Another use case I found was to retain the state of a whole interface of an application by caching the innerHTML once it has been rendered. I use this in the Yahoo Firehose search interface, and you can get the full story about local storage and how it is used in this screencast:
The stripped down code is incredibly simple (JavaScript with PHP embed):

1. // test for localStorage support
2. if(('localStorage' in window) && window['localStorage'] !== null){
3.  
4. var f = document.getElementById('mainform');
5. // test with PHP if the form was sent (the submit button has the name "sent")
7.  
8. // get the HTML of the form and cache it in the property "state"
9. localStorage.setItem('state',f.innerHTML);
11. // if the form hasn't been sent...
13.  
14. // check if a state property exists and write back the HTML cache
15. if('state' in localStorage){
16. f.innerHTML = localStorage.getItem('state');
17. }
18.  
20.  
21. }

Other ideas

In essence, you can use local storage every time you need to speed up access. For example, you could store image sprites in base-64 encoded datasets instead of loading them from a server. Or you could store CSS and JavaScript libraries on the client. Anything goes – have a play.

Issues with local and session storage

Of course, not all is rainbows and unicorns with the localStorage API. There are a few niggles that need ironing out. As with anything, this needs people to use the technology and raise issues. Here are some of the problems:

• Inadequate information about storage quota – if you try to add more content to an already full store, you get a QUOTA_EXCEEDED_ERR and that’s it. There’s a great explanation and test suite for localStorage quota available.
• Lack of automatically expiring storage – a feature that cookies came with. Pamela Fox has a solution (also available as source code)
• Lack of encrypted storage – right now, everything is stored in readable strings in the browser.

Bigger, better, faster, more!

As cool as the local and session storage APIs are, they are not quite ready for extensive adoption – the storage limits might get in your way, and if you really want to go to town with accessing, filtering and sorting data, real databases are what you’ll need. And, as we live in a world of client-side development, people are moving from heavy server-side databases like MySQL to NoSQL environments.
On the web, there is also a lot of work going on, with Ian Hickson of Google proposing the Web SQL database, and Nikunj Mehta, Jonas Sicking (Mozilla), Eliot Graff (Microsoft) and Andrei Popescu (Google) taking the idea beyond simply replicating MySQL and instead offering Indexed DB as an even faster alternative.
On the mobile front, a really important feature is to be able to store data to use when you are offline (mobile coverage and roaming data plans anybody?) and you can use the Offline Webapps API for that.
As I mentioned at the beginning, we have a very exciting time ahead – let’s make this web work faster and more reliably by using what browsers offer us. For more on local storage, check out the chapter on Dive into HTML5.

Local Storage And How To Use It On Websites

Storing information locally on a user’s computer is a powerful strategy for a developer who is creating something for the Web. In this article, we’ll look at how easy it is to store information on a computer to read later and explain what you can use that for.

Adding State To The Web: The “Why” Of Local Storage

The main problem with HTTP as the main transport layer of the Web is that it is stateless. This means that when you use an application and then close it, its state will be reset the next time you open it. If you close an application on your desktop and re-open it, its most recent state is restored.
This is why, as a developer, you need to store the state of your interface somewhere. Normally, this is done server-side, and you would check the user name to know which state to revert to. But what if you don’t want to force people to sign up?
This is where local storage comes in. You would keep a key on the user’s computer and read it out when the user returns.

C Is For Cookie. Is That Good Enough For Me?

The classic way to do this is by using a cookie. A cookie is a text file hosted on the user’s computer and connected to the domain that your website runs on. You can store information in them, read them out and delete them. Cookies have a few limitations though:

• They add to the load of every document accessed on the domain.
• They allow up to only 4 KB of data storage.
• Because cookies have been used to spy on people’s surfing behavior, security-conscious people and companies turn them off or request to be asked every time whether a cookie should be set.

To work around the issue of local storage — with cookies being a rather dated solution to the problem — the WHATWG and W3C came up with a few local storage specs, which were originally a part of HTML5 but then put aside because HTML5 was already big enough.

Using Local Storage In HTML5-Capable Browsers

Using local storage in modern browsers is ridiculously easy. All you have to do is modify the localStorage object in JavaScript. You can do that directly or (and this is probably cleaner) use the setItem() and getItem() method:

localStorage.setItem('favoriteflavor','vanilla');

If you read out the favoriteflavor key, you will get back “vanilla”:

var taste = localStorage.getItem('favoriteflavor');
// -> "vanilla"

To remove the item, you can use — can you guess? — the removeItem() method:

localStorage.removeItem('favoriteflavor');
var taste = localStorage.getItem('favoriteflavor');
// -> null

That’s it! You can also use sessionStorage instead of localStorage if you want the data to be maintained only until the browser window closes.

Working Around The “Strings Only” Issue

One annoying shortcoming of local storage is that you can only store strings in the different keys. This means that when you have an object, it will not be stored the right way.
You can see this when you try the following code:

var car = {};
car.wheels = 4;
car.doors = 2;
car.sound = 'vroom';
car.name = 'Lightning McQueen';
console.log( car );
localStorage.setItem( 'car', car );
console.log( localStorage.getItem( 'car' ) );

Trying this out in the console shows that the data is stored as [object Object] and not the real object information:

You can work around this by using the native JSON.stringify() and JSON.parse() methods:

var car = {};
car.wheels = 4;
car.doors = 2;
car.sound = 'vroom';
car.name = 'Lightning McQueen';
console.log( car );
localStorage.setItem( 'car', JSON.stringify(car) );
console.log( JSON.parse( localStorage.getItem( 'car' ) ) );

Where To Find Local Storage Data And How To Remove It

During development, you might sometimes get stuck and wonder what is going on. Of course, you can always access the data using the right methods, but sometimes you just want to clear the plate. In Opera, you can do this by going to Preferences → Advanced → Storage, where you will see which domains have local data and how much:

Large view
Doing this in Chrome is a bit more problematic, which is why we made a screencast:
Mozilla has no menu access so far, but will in future. For now, you can go to the Firebug console and delete storage manually easily enough.
So, that’s how you use local storage. But what can you use it for?

Use Case #1: Local Storage Of Web Service Data

One of the first uses for local storage that I discovered was caching data from the Web when it takes a long time to get it. My World Info entry for the Event Apart 10K challenge shows what I mean by that.
When you call the demo the first time, you have to wait up to 20 seconds to load the names and geographical locations of all the countries in the world from the Yahoo GeoPlanet Web service. If you call the demo a second time, there is no waiting whatsoever because — you guessed it — I’ve cached it on your computer using local storage.
The following code (which uses jQuery) provides the main functionality for this. If local storage is supported and there is a key called thewholefrigginworld, then call the render() method, which displays the information. Otherwise, show a loading message and make the call to the Geo API using getJSON(). Once the data has loaded, store it in thewholefrigginworld and call render() with the same data:

if(localStorage && localStorage.getItem('thewholefrigginworld')){
  render(JSON.parse(localStorage.getItem('thewholefrigginworld')));
} else {
  $('#list').html('
'+loading+'

');
  var query = 'select centroid,woeid,name,boundingBox'+
              ' from geo.places.children(0)'+
              ' where parent_woeid=1 and placetype="country"'+
              ' | sort(field="name")';
  var YQL = 'http://query.yahooapis.com/v1/public/yql?q='+
             encodeURIComponent(query)+'&diagnostics=false&format=json';
  $.getJSON(YQL,function(data){
    if(localStorage){
      localStorage.setItem('thewholefrigginworld',JSON.stringify(data));
    }
    render(data);
  });
}

You can see the difference in loading times in the following screencast:
The code for the world info is available on GitHub.
This can be extremely powerful. If a Web service allows you only a certain number of calls per hour but the data doesn’t change all that often, you could store the information in local storage and thus keep users from using up your quota. A photo badge, for example, could pull new images every six hours, rather than every minute.
This is very common when using Web services server-side. Local caching keeps you from being banned from services, and it also means that when a call to the API fails for some reason, you will still have information to display.
getJSON() in jQuery is especially egregious in accessing services and breaking their cache, as explained from the YQL team. Because the request to the service using getJSON() creates a unique URL every time, the service does not deliver its cached version but rather fully accesses the system and databases every time you read data from it. This is not efficient, which is why you should cache locally and use ajax() instead.

Use Case #2: Maintaining The State Of An Interface The Simple Way

Another use case is to store the state of interfaces. This could be as crude as storing the entire HTML or as clever as maintaining an object with the state of all of your widgets. One instance where I am using local storage to cache the HTML of an interface is the Yahoo Firehose research interface (source on GitHub):
The code is very simple — using YUI3 and a test for local storage around the local storage call:

YUI().use('node', function(Y) {
  if(('localStorage' in window) && window['localStorage'] !== null){
    var key = 'lastyahoofirehose';
  
    localStorage.setItem(key,Y.one('form').get('innerHTML'));
  
  if(key in localStorage){
      Y.one('#mainform').set('innerHTML',localStorage.getItem(key));
      Y.one('#hd').append('
Notice: We restored your last search for you - not live data');
  }
  
  }
});

You don’t need YUI at all; it only makes it easier. The logic to generically cache interfaces in local storage is always the same: check if a “Submit” button has been activated (in PHP, Python, Ruby or whatever) and, if so, store the innerHTML of the entire form; otherwise, just read from local storage and override the innerHTML of the form.

The Dark Side Of Local Storage

Of course, any powerful technology comes with the danger of people abusing it for darker purposes. Samy, the man behind the “Samy is my hero” MySpace worm, recently released a rather scary demo called Evercookie, which shows how to exploit all kind of techniques, including local storage, to store information of a user on their computer even when cookies are turned off. This code could be used in all kinds of ways, and to date there is no way around it.
Research like this shows that we need to look at HTML5′s features and add-ons from a security perspective very soon to make sure that people can’t record user actions and information without the user’s knowledge. An opt-in for local storage, much like you have to opt in to share your geographic location, might be in order; but from a UX perspective this is considered clunky and intrusive. Got any good ideas?