Ingestion pipeline

From file to processed data

To ingest data from your file format into structured cold/hot storage in the database, configure your pipeline in the following steps:

1. Define dataset metadata. Specify required and optional metadata for the dataset.

2. Configure plugin settings. Select the dataset type and set its specific parameters.

3. Calibrate signals. Set signal priority, apply scaling, and add units and descriptions.

4. Configure signal aliasing. Define aliasing rules to create a unified namespace across all your datastreams.

5. Add scripts. Enhance your datasets with custom Python scripts to add metadata, signals, KPIs, and more.

You can configure your datastream pipeline under Datastream → Configure → Pipeline.