OpenReplay
Navigate back to the homepage
BLOG
Browse Repo
Back

Getting Started with IndexedDB for Big Data Storage

Craig Buckler
June 3rd, 2021 · 6 min read

It’s not always necessary to send a user’s data to the server: you could choose to store some information in the browser. It’s ideal for device-specific settings such as interface configurations (e.g. light/dark modes) or private data which should not be transmitted across the network (e.g. an encryption key).

Consider the Performance API for timing network and page events. You could upload all data the moment it’s available but that’s a considerable volume of information which, ironically, would hit page performance. A better option would be to store it locally, process statistics using a Web Worker, and upload the results when the browser is less busy.

Two cross-browser client-side storage APIs are available:

  1. Web Storage

    Synchronous name-value pair storage saved permanently (localStore) or for the current session (sessionStore). Browsers permit up to 5MB of Web Storage per domain.

  2. IndexedDB

    An asynchronous NoSQL-like name-value store which can save data, files, and blobs. At least 1GB should be available per domain and it can reach up to 60% of the remaining disk space.

Another storage option, WebSQL, is available in some editions of Chrome and Safari. However, it has a 5MB limit, is inconsistent, and was deprecated in 2010.

In this tutorial we’ll store timing data for the page and all assets. Web Storage space can be slow and too limiting so IndexedDB is the best option.

All example code is available on Github should you want to try IndexedDB on your own site.

What is IndexedDB?

IndexedDB was first implemented in 2011 and became a W3C standard in January 2015. It has good browser support although its callback and events-based API seems clunky now we have ES2015+. This article demonstrates how to write a Promise-based wrapper so you can use chaining and async/await.

Note the following IndexedDB terms:

  • database — a top-level store. A domain can create any number of IndexedDB databases but it’s unusual to see more than one. Only pages within the same domain can access the database.

  • object store — a name/value store for related data items. It’s similar to a collection in MongoDB or tables in a relational database.

  • key — a unique name used to reference every record (value) in the object store. It can be generated using an autoIncrement number or set to any unique value within the record.

  • index — another way to organize data in an object store. Search queries can only examine the key or an index.

  • schema — the definition of object stores, keys, and indexes.

  • version — a version number (integer) assigned to a schema. IndexedDB offers automated versioning so you can update databases to the latest schema.

  • operation — database activities such as creating, reading, updating, or deleting records.

  • transaction — a set of one or more operations. A transaction guarantees all its operations either succeed or fail. It cannot fail some and not others.

  • cursor — a way to iterate over records without having to load all into memory at once.

Developing and debugging a database

In this tutorial, you will create an IndexedDB database named performance. It contains two object stores:

1. navigation

This stores page navigation timings information (redirects, DNS lookups, page loading, file sizes, load events, etc). A date will be added to use as the key.

2. resource

This stores resource timings information (timings for other resources such as images, stylesheets, scripts, Ajax calls etc.) A date will be added, but two or more assets could load at the same time so an auto-incrementing ID will be used as the key. Indexes will be created for the date and the name (the resource’s URL).

All Chrome-based browsers have an Application tab where you can examine storage space, artificially limit the capacity, and wipe all data. The IndexedDB entry in the Storage tree allows you to view, update, and delete object stores, indexes, and individual records. Firefox’s panel is named Storage.

You can also run your application in incognito mode so all data is deleted once the browser window is closed.

Connecting to an IndexedDB database

A wrapper class created in indexeddb.js checks for IndexedDB support using:

1if ('indexedDB' in window) // ...

It then opens a database connection using indexedDB.open() by passing:

  1. the database name, and
  2. an optional version integer.
1const dbOpen = indexedDB.open('performance', 1);

Three important event handler functions must be defined:

  1. dbOpen.onerror runs when an IndexedDB connection cannot be established.

  2. dbOpen.onupgradeneeded runs when the version required (1) is greater than the current version (0 when the database is not defined). A handler function must run IndexedDB methods such as createObjectStore() and createIndex() to create the storage structures.

  3. dbOpen.onsuccess runs when the connection has been established and any upgrades have completed. The connection object in dbOpen.result is used in all subsequent data operations. It is assigned to this.db in the wrapper class.

The wrapper constructor code:

1// IndexedDB wrapper class: indexeddb.js
2export class IndexedDB {
3
4 // connect to IndexedDB database
5 constructor(dbName, dbVersion, dbUpgrade) {
6
7 return new Promise((resolve, reject) => {
8
9 // connection object
10 this.db = null;
11
12 // no support
13 if (!('indexedDB' in window)) reject('not supported');
14
15 // open database
16 const dbOpen = indexedDB.open(dbName, dbVersion);
17
18 if (dbUpgrade) {
19
20 // database upgrade event
21 dbOpen.onupgradeneeded = e => {
22 dbUpgrade(dbOpen.result, e.oldVersion, e.newVersion);
23 };
24
25 }
26
27 // success event handler
28 dbOpen.onsuccess = () => {
29 this.db = dbOpen.result;
30 resolve( this );
31 };
32
33 // failure event handler
34 dbOpen.onerror = e => {
35 reject(`IndexedDB error: ${ e.target.errorCode }`);
36 };
37
38 });
39
40 }
41
42 // more methods coming later...
43
44}

A performance.js script loads this module and instantiates a new IndexedDB object named perfDB after the page has loaded. It passes the database name (performance), version (1), and an upgrade function. The indexeddb.js constructor calls the upgrade function with the database connection object, the current database version, and the new version:

1// performance.js
2import { IndexedDB } from './indexeddb.js';
3
4window.addEventListener('load', async () => {
5
6 // IndexedDB connection
7 const perfDB = await new IndexedDB(
8 'performance',
9 1,
10 (db, oldVersion, newVersion) => {
11
12 console.log(`upgrading database from ${ oldVersion } to ${ newVersion }`);
13
14 switch (oldVersion) {
15
16 case 0: {
17
18 const
19 navigation = db.createObjectStore('navigation', { keyPath: 'date' }),
20 resource = db.createObjectStore('resource', { keyPath: 'id', autoIncrement: true });
21
22 resource.createIndex('dateIdx', 'date', { unique: false });
23 resource.createIndex('nameIdx', 'name', { unique: false });
24
25 }
26
27
28 }
29
30 });
31
32 // more code coming later...
33
34});

At some point it will become necessary to change the database schema — perhaps to add new object stores, indexes, or data updates. In that situation, you must increment the version (from 1 to 2). The next page load will trigger the upgrade handler again so you can add a further block to the switch statement, e.g. to create an index named durationIdx on the duration property in the resource object store:

1case 1: {
2 const resource = db.transaction.objectStore('resource');
3 resource.createIndex('durationIdx', 'duration', { unique: false });
4}

The usual break at the end of each case block is omitted. When someone accesses the application for the first time, the case 0 block will run followed by case 1 and all subsequent blocks. Anyone already on version 1 would run the updates starting at the case 1. IndexedDB schema update methods include:

Everyone who loads the page will be on the same version — unless they have the app running in two or more tabs. To avoid conflicts, the database connection onversionchange handler can be added to indexeddb.js which prompts the user to reload the page:

1// version change handler
2dbOpen.onversionchange = () => {
3
4 dbOpen.close();
5 alert('Database upgrade required - reloading...');
6 location.reload();
7
8};

You can now add the performance.js script to a page and run it to check that object stores and indexes are created (DevTools Application or Storage panels):

1<script type="module" src="./performance.js"></script>

Record performance statistics

All IndexedDB operations are wrapped in a transaction. The following process is used:

  1. Create a database transaction object. This defines one or more object stores (single string or array of strings) and the access type: "readonly" for fetching data, or "readwrite" for inserts and updates.

  2. Create a reference to an objectStore() within the scope of the transaction.

  3. Run any number of add() (inserts only) or put() methods (inserts and updates).

Add a new update() method to the IndexedDB class in indexeddb.js:

1// store item
2 update(storeName, value, overwrite = false) {
3
4 return new Promise((resolve, reject) => {
5
6 // new transaction
7 const
8 transaction = this.db.transaction(storeName, 'readwrite'),
9 store = transaction.objectStore(storeName);
10
11 // ensure values are in array
12 value = Array.isArray(value) ? value : [ value ];
13
14 // write all values
15 value.forEach(v => {
16 if (overwrite) store.put(v);
17 else store.add(v);
18 });
19
20 transaction.oncomplete = () => {
21 resolve(true); // success
22 };
23
24 transaction.onerror = () => {
25 reject(transaction.error); // failure
26 };
27
28 });
29
30 }

This adds or updates (if the overwrite parameter is true) one or more values in the named store and wraps the whole transaction in a Promise. The transaction.oncomplete event handler runs when the transaction auto-commits at the end of the function and all database operations are complete. A transaction.onerror handler reports errors.

IndexedDB events bubble up from the operation to the transaction, to the store, and to the database. You could create a single onerror handler on the database which receives all errors. Like DOM events, propagation can be stopped with event.stopPropagation().

The performance.js script can now report page navigation metrics:

1// record page navigation information
2 const
3 date = new Date(),
4
5 nav = Object.assign(
6 { date },
7 performance.getEntriesByType('navigation')[0].toJSON()
8 );
9
10 await perfDB.update('navigation', nav);

and resource metrics:

1const res = performance.getEntriesByType('resource').map(
2 r => Object.assign({ date }, r.toJSON())
3 );
4
5 await perfDB.update('resource', res);

In both cases, a date property is added to the cloned timing objects so it becomes possible to search for data within a specific period.

Frontend Monitoring

OpenReplay is a developer-oriented, open-source session replay alternative to FullStory and LogRocket. It’s like having your browser’s DevTools open while looking over your user’s shoulder.

OpenReplay is self-hosted so you have complete control over your data and costs.

OpenReplay

Happy debugging, for modern frontend teams - Start monitoring your web app for free.

Reading performance records

IndexedDB searching is rudimentary compared to other databases. You can only fetch records by their key or an indexed value. You cannot use equivalents of SQL JOIN or functions such as AVERAGE() and SUM(). All record processing must be handled with JavaScript code; a background Web Worker thread could be a practical option.

You can retrieve a single record by passing its key to an object store or index’s .get() method and defining an onsuccess handler:

1// EXAMPLE CODE
2const
3
4 // new readonly transaction
5 transaction = db.transaction('resource', 'readonly'),
6
7 // get resource object store
8 resource = transaction.objectStore('resource'),
9
10 // fetch record 1
11 request = resource.get(1);
12
13// request complete
14request.onsuccess = () => {
15 console.log('result:', request.result);
16};
17
18// request failed
19request.onerror = () => {
20 console.log('failed:', request.error);
21};

Similar methods include:

The query can also be a KeyRange argument to find records within a range, e.g. IDBKeyRange.bound(1, 10) returns all records with key between 1 and 10 inclusive:

1request = resource.getAll( IDBKeyRange.bound(1, 10) );

KeyRange options:

The lower, upper, and bound methods have an optional exclusive flag, e.g. IDBKeyRange.bound(1, 10, true, false) — keys greater than 1 (but not 1 itself) and less than or equal to 10.

Reading a whole dataset into an array becomes impossible as the database grows larger. IndexedDB provides cursors which can iterate through each record one at a time. The .openCursor() method is passed a KeyRange and optional direction string ("next", "nextunique", "prev", or "preunique").

Add a new fetch() method to the IndexedDB class in indexeddb.js to search an object store or index with upper and lower bounds with a callback function that is passed the cursor. Two further methods are also required:

  1. index(storeName, indexName) — returns either an object store or index on that store, and
  2. bound(lowerBound, upperBound) — returns an appropriate KeyRange object.
1// get items using cursor
2 fetch(storeName, indexName, lowerBound = null, upperBound = null, callback) {
3
4 const
5 request = this.index(storeName, indexName)
6 .openCursor( this.bound(lowerBound, upperBound) );
7
8 // pass cursor to callback function
9 request.onsuccess = () => {
10 if (callback) callback(request.result);
11 };
12
13 request.onerror = () => {
14 return(request.error); // failure
15 };
16
17 }
18
19
20 // start a new read transaction on object store or index
21 index(storeName, indexName) {
22
23 const
24 transaction = this.db.transaction(storeName),
25 store = transaction.objectStore(storeName);
26
27 return indexName ? store.index(indexName) : store;
28
29 }
30
31
32 // get bounding object
33 bound(lowerBound, upperBound) {
34
35 if (lowerBound && upperBound) return IDBKeyRange.bound(lowerBound, upperBound);
36 else if (lowerBound) return IDBKeyRange.lowerBound(lowerBound);
37 else if (upperBound) return IDBKeyRange.upperBound(upperBound);
38
39 }

The performance.js script can now retrieve page navigation metrics, e.g. return all domContentLoadedEventEnd during June 2021:

1// fetch page navigation objects in June 2021
2 perfDB.fetch(
3 'navigation',
4 null, // not an index
5 new Date(2021,5,1,10,40,0,0), // lower
6 new Date(2021,6,1,10,40,0,0), // upper
7 cursor => { // callback function
8
9 if (cursor) {
10 console.log(cursor.value.domContentLoadedEventEnd);
11 cursor.continue();
12 }
13
14 }
15 );

Similarly, you can calculate the average download time for a specific file and report it back to OpenReplay:

1// calculate average download time using index
2 let
3 filename = 'http://mysite.com/main.css',
4 count = 0,
5 total = 0;
6
7 perfDB.fetch(
8 'resource', // object store
9 'nameIdx', // index
10 filename, // matching file
11 filename,
12 cursor => { // callback
13
14 if (cursor) {
15
16 count++;
17 total += cursor.value.duration;
18 cursor.continue();
19
20 }
21 else {
22
23 // all records processed
24 if (count) {
25
26 const avgDuration = total / count;
27
28 console.log(`average duration for ${ filename }: ${ avgDuration } ms`);
29
30 // report to OpenReplay
31 if (asayer) asayer.event(`${ filename }`, { avgDuration });
32
33 }
34
35 }
36
37 });

In both cases, the cursor object is passed to the callback function where it can:

  1. obtain the record value with cursor.value
  2. advance to the next record with cursor.continue()
  3. move forward N records with cursor.advance(N)
  4. update the record with cursor.update(data), or
  5. delete the record with cursor.delete()

cursor is null when all matching records have been processed.

Check Remaining Storage Space

Browsers allocate a significant volume of storage to IndexedDB but it will eventually run out. The new Promise-based StorageManager API can calculate the space remaining for the domain:

1(async () => {
2
3 if (!navigator.storage) return;
4
5 const
6 estimate = await navigator.storage.estimate(),
7
8 // calculate remaining storage in MB
9 available = Math.floor((estimate.quota - estimate.usage) / 1024 / 1024);
10
11 console.log(`${ available } MB remaining`);
12
13})();

The API is not supported in IE or Safari. As the limit is approached, you could choose to remove older records.

Conclusion

IndexedDB is one of the older and more complex browser APIs but you can add wrapper methods to adopt Promises and async/await. Pre-built libraries such as idb can help if you’d rather not do that yourself.

Despite its drawbacks and some unusual design decisions, IndexedDB remains the fastest and largest browser-based data store.

More articles from OpenReplay Blog

Discovering Vue Composition API with examples

Learn about the Composition API with these practical examples

May 25th, 2021 · 6 min read

Learn how Mapping Works In VueX

Learn how mapgetters, mapactions, mapmutations and mapstate work in VueX

May 24th, 2021 · 3 min read
© 2021 OpenReplay Blog
Link to $https://twitter.com/OpenReplayHQLink to $https://github.com/openreplay/openreplayLink to $https://www.linkedin.com/company/18257552