Firestore Data Migration

Supabase provides several tools to convert data from a Firebase Firestore database to a Supabase PostgreSQL database. The process copies the entire contents of a single Firestore collection to a single PostgreSQL table.

The Firestore collection is "flattened" and converted to a table with basic columns of one of the following types: text, numeric, boolean, or jsonb. If your structure is more complex, you can write a program to split the newly-created json file into multiple, related tables before you import your json file(s) to Supabase.

Set up the migration tool #

Clone the firebase-to-supabase repository:

1git clone https://github.com/supabase-community/firebase-to-supabase.git

In the /firestore directory, create a file named supabase-service.json with the following contents:

{
  "host": "database.server.com",
  "password": "secretpassword",
  "user": "postgres",
  "database": "postgres",
  "port": 5432
}

Go to the Database settings for your project in the Supabase Dashboard.
Under Connection Info, copy the Host string and replace the entry in your supabase-service.json file.
Enter the password you used when you created your Supabase project in the password entry in the supabase-service.json file.

Generate a Firebase private key #

Log in to your Firebase Console and open your project.
Click the gear icon next to Project Overview in the sidebar and select Project Settings.
Click Service Accounts and select Firebase Admin SDK.
Click Generate new private key.
Rename the downloaded file to firebase-service.json.

Command line options#

List all Firestore collections#

node collections.js

Dump Firestore collection to JSON file#

node firestore2json.js <collectionName> [<batchSize>] [<limit>]

batchSize (optional) defaults to 1000
output filename is <collectionName>.json
limit (optional) defaults to 0 (no limit)

Customize the JSON file with hooks

You can customize the way your JSON file is written using a custom hook. A common use for this is to "flatten" the JSON file, or to split nested data into separate, related database tables. For example, you could take a Firestore document that looks like this:

[{ "user": "mark", "score": 100, "items": ["hammer", "nail", "glue"] }]

And split it into two files (one table for users and one table for items):

[{ "user": "mark", "score": 100 }]

[
  { "user": "mark", "item": "hammer" },
  { "user": "mark", "item": "nail" },
  { "user": "mark", "item": "glue" }
]

Import JSON file to Supabase (PostgreSQL) #

node json2supabase.js <path_to_json_file> [<primary_key_strategy>] [<primary_key_name>]

<path_to_json_file> The full path of the file you created in the previous step (Dump Firestore collection to JSON file ), such as ./my_collection.json
[<primary_key_strategy>] (optional) Is one of:
- none (default) No primary key is added to the table.
- smallserial Creates a key using (id SMALLSERIAL PRIMARY KEY) (autoincrementing 2-byte integer).
- serial Creates a key using (id SERIAL PRIMARY KEY) (autoincrementing 4-byte integer).
- bigserial Creates a key using (id BIGSERIAL PRIMARY KEY) (autoincrementing 8-byte integer).
- uuid Creates a key using (id UUID PRIMARY KEY DEFAULT uuid_generate_v4()) (randomly generated UUID).
- firestore_id Creates a key using (id TEXT PRIMARY KEY) (uses existing firestore_id random text as key).
[<primary_key_name>] (optional) Name of primary key. Defaults to "id".

Custom hooks#

Hooks are used to customize the process of exporting a collection of Firestore documents to JSON. They can be used for:

Customizing or modifying keys
Calculating data
Flattening nested documents into related SQL tables

Write a custom hook#

Create a .js file for your collection

If your Firestore collection is called users, create a file called users.js in the current folder.

Construct your .js file

The basic format of a hook file looks like this:

module.exports = (collectionName, doc, recordCounters, writeRecord) => {
  // modify the doc here
  return doc
}

Parameters

collectionName: The name of the collection you are processing.
doc: The current document (JSON object) being processed.
recordCounters: An internal object that keeps track of how many records have been processed in each collection.
writeRecord: This function automatically handles the process of writing data to other JSON files (useful for "flatting" your document into separate JSON files to be written to separate database tables). writeRecord takes the following parameters:
- name: Name of the JSON file to write to.
- doc: The document to write to the file.
- recordCounters: The same recordCounters object that was passed to this hook (just passes it on).

Examples#

Add a new (unique) numeric key to a collection

module.exports = (collectionName, doc, recordCounters, writeRecord) => {
  doc.unique_key = recordCounter[collectionName] + 1
  return doc
}

Add a timestamp of when this record was dumped from Firestore

module.exports = (collectionName, doc, recordCounters, writeRecord) => {
  doc.dump_time = new Date().toISOString()
  return doc
}

Flatten JSON into separate files

Flatten the users collection into separate files:

[
  {
    "uid": "abc123",
    "name": "mark",
    "score": 100,
    "weapons": ["toothpick", "needle", "rock"]
  },
  {
    "uid": "xyz789",
    "name": "chuck",
    "score": 9999999,
    "weapons": ["hand", "foot", "head"]
  }
]

The users.js hook file:

module.exports = (collectionName, doc, recordCounters, writeRecord) => {
  for (let i = 0; i < doc.weapons.length; i++) {
    const weapon = {
      uid: doc.uid,
      weapon: doc.weapons[i],
    }
    writeRecord('weapons', weapon, recordCounters)
  }
  delete doc.weapons // moved to separate file
  return doc
}

The result is two separate JSON files:

[
  { "uid": "abc123", "name": "mark", "score": 100 },
  { "uid": "xyz789", "name": "chuck", "score": 9999999 }
]

[
  { "uid": "abc123", "weapon": "toothpick" },
  { "uid": "abc123", "weapon": "needle" },
  { "uid": "abc123", "weapon": "rock" },
  { "uid": "xyz789", "weapon": "hand" },
  { "uid": "xyz789", "weapon": "foot" },
  { "uid": "xyz789", "weapon": "head" }
]

Resources#

Edit this page on GitHub