iModel Transformation and Data Exchange

The @bentley/imodeljs-backend package provides some classes that implement Extract, Transform, and Load (ETL) functionality:

• IModelExporter and IModelExportHandler are the base classes that implement the extract (or export) part of ETL functionality.
• IModelTransformer is the base class that implements the transform part of ETL functionality.
• IModelImporter is the base class that implements the load (or import) part of ETL functionality.

The above classes contain the lower-level functionality required to implement transformation and data exchange services. These classes should be considered a framework and not confused with the actual packaged and deployed services that use the framework.

IModelExporter

The IModelExporter and IModelExportHandler base classes are used when the source data in an ETL workflow is contained within an iModel.

While it is possible to export data from an iModel using the standard [IModelDb] API, the IModelExporter and IModelExportHandler base classes offer the following capabilities:

• An implementation of a visitor pattern that makes it easy to iterate the iModel in a prescribed order that attempts to visit dependencies/prerequisites before dependents.
• Visit the entire iModel using IModelExporter.exportAll
• Visit only changed entities using IModelExporter.exportChanges
• Visit a subset of the iModel using IModelExporter.exportModel, IModelExporter.exportModelContents, or IModelExporter.exportElement
• Easily exclude certain entity types to filter the export content using IModelExporter.excludeElementCategory, IModelExporter.excludeElementClass, or IModelExporter.excludeElementAspectClass
• Integration with IModelTransformer

Below is an example of using IModelExporter and IModelExportHandler to export all Code values from an iModel:

import { CodeSpec } from "@bentley/imodeljs-common";
import { Element, IModelDb, IModelExporter, IModelExportHandler, IModelJsFs as fs } from "@bentley/imodeljs-backend";

/** CodeExporter creates a CSV output file containing all Codes from the specified iModel. */
class CodeExporter extends IModelExportHandler {
  public outputFileName: string;

  /** Initiate the export of codes. */
  public static exportCodes(iModelDb: IModelDb, outputFileName: string): void {
    const exporter = new IModelExporter(iModelDb);
    const exportHandler = new CodeExporter(outputFileName);
    exporter.registerHandler(exportHandler);
    exporter.exportAll();
  }

  /** Construct a new CodeExporter */
  private constructor(outputFileName: string) {
    super();
    this.outputFileName = outputFileName;
  }

  /** Override of IModelExportHandler.onExportElement that outputs a line of a CSV file when the Element has a Code. */
  protected onExportElement(element: Element, isUpdate: boolean | undefined): void {
    const codeValue: string = element.code.getValue();
    if ("" !== codeValue) { // only output when Element has a Code
      const codeSpec: CodeSpec = element.iModel.codeSpecs.getById(element.code.spec);
      fs.appendFileSync(this.outputFileName, `${element.id}, ${codeSpec.name}, ${codeValue}\n`);
    }
    super.onExportElement(element, isUpdate);
  }
}

IModelImporter

The IModelImporter base class is used when the target in an ETL workflow is an iModel.

While it is possible to import data into an iModel using the standard [IModelDb] API, the IModelImporter class offers the following capabilities:

• Callbacks whenever IModelImporter is used to insert, update, or delete entities. Simply override one of the protected onInsert*, onUpdate*, or onDelete* methods.
• Optionally auto-extend IModel.projectExtents during import
• Integration with IModelTransformer

IModelCloneContext

The IModelCloneContext class provides the core cloning capability required for iModel transformation. It also maintains the sourceId --> targetId mapping which is required to successfully clone Entity instances from the source iModel into the target iModel. iModel entities are highly related to each other. Therefore, cloning an entity means copying a graph of objects and remapping their source references (Ids) to other target entities.

IModelTransformer

The IModelTransformer base class is used when the source and target in an ETL workflow are both/different iModels and some sort of data transformation is needed in the middle. An instance of IModelTransformer holds instances of IModelExporter, IModelImporter, and IModelCloneContext. This means that customization is possible at the export stage, the transformation stage, and the import stage of the overall ETL process.

Potential transformations include:

• Cloning - copying an entity from the source and remapping its internal Ids for the target
• Filtering - excluding data from the target that is contained in the source
• Augmenting - generating data during transformation for the target that is not part of the source
• Schema Mapping - mapping classes and properties to a new schema during transformation
• Change Squashing - each iModel has its own change timeline, so multiple changesets to the source could be squashed into a single changeset to the target

Logging

With batch processes like iModel transformation and data exchange, logging is often the only way to figure out what is actually happening. The following logger categories are provided for use with the Logger:

• BackendLoggerCategory.IModelExporter
• BackendLoggerCategory.IModelTransformer
• BackendLoggerCategory.IModelImporter

Last Updated: 13 June, 2024