Migrating from Marple Files

To migrate your data from Marple Files to Marple DB, you can use this python script:

7KB

migrate_marple.py

Open

The migration process consists of a few different steps.

Install dependencies

pip install click dotenv pandas requests tqdm;

Create an API token in Marple Files and in Marple DB, place them in a .env file, alternatively you can specify them in the CLI commands later on.

FILES_TOKEN=...
DB_TOKEN=...

Create an overview of all your datasets on Marple Files by running

python migrate_marple.py create-overview

This will create a datasets_overview.csv file with all available information associated to each dataset. The metadata you specified in Marple Files is added as extra columns. Two other columns are added to assist uploading to Marple DB:

• STATUS: by default set to "SKIP", set to "TODO" to mark this dataset for migration. Once a dataset is migrated, it will get status "DONE"

• STREAM LABEL: by default, this label is set to the file extension. You can change this label to specify which Datastream Marple DB should use to import this dataset.

Any other column you add to the CSV will be added as metadata to the datasets in Marple DB.

This is a perfect opportunity to add clear and uniform metadata to all your datasets. The better your metadata, the easier it is to find insights in your data.

Once the overview CSV is finalized, you can start the actual migration. First, create a new Datastream in Marple DB for each STREAM LABEL . Use the id of this newly created Datastream in the following command:

python migrate_marple.py migrate --stream-label [LABEL] --stream-id [ID]

This will migrate the datasets with the selected stream-label to the Datastream with the specified stream-id. By default only 10 datasets are moved, this gives you time to verify the Datastream settings. When you're confident the datasets import well in Marple DB, you can increase this number by specifying the --number-datasets option.