Before you import a packaged flow into a Production environment, you may need to apply import rules to remap objects and locations from the source instance to the new instance. Import mapping rules are not required when importing into the same environment, although they may be helpful in some cases.

• For more information on creating an export package, see Export Flow.
• For more information on how to import, see Import Flow.

You can apply the following types of remappings:

Import Rules

When a flow is imported, references in the flow definition that apply in the source instance may not apply in the target instance. For example, the location paths to the source datasets may need to be rewritten to point to a different location in the target instance. 

Before you import your flow definition, you need to define rules for any value or object remapping that must be done in the target environment. 

Notes on import rules

1. Value and object remapping rules should be completed before you import the flow. The flow may be non-functional until the rules are applied.
   

   Tip: After you create your import rules, you can perform via API a dry run of the import. Any errors are reported in the response. Details are provided below.

   
   

2. Value and object remapping rules are applied at the time of import.  If you add new rules, they are not retroactively applied to release packages that have already been imported.
3. When changing rules:
   1. Any previously applied rules to the same import object are deleted.
   2. You can apply multiple rules in the same change. 
   3. Rules are applied in the order in which they are listed in the request. Rules listed later in the request must be compatible with expected changes applied by the earlier rules. 
4. Value and object remapping must be completed via API. API usage is considered a developer-level skill. Examples are provided below.

Import Rule Requirements

Examples

The following are some example import rules to address specific uses.

Example - Replace a connection

In this following example, you must remap the connection from the source instance of the platform to the corresponding connection in the instance where you are importing. 

First, you must be able to uniquely identify the connection from the source that you wish to remap. 

• While the connection Id may work in a limited scope, that identifier is unlikely to be unique within your environment.
• If you do know the connect Id from the source system, you can skip the first step below.

In the API response in a connection definition, you can acquire the uuid value for the connection, which is a unique identifier for the connection object across all instances of the platform:

/v4/connections

GET

{
    "data": [
        {
            "connectParams": {
                "vendor": "redshift",
                "vendorName": "redshift",
                "host": "redshift.example.com",
                "port": "5439",
                "extraLoadParams": "BLANKSASNULL EMPTYASNULL TRIMBLANKS TRUNCATECOLUMNS",
                "defaultDatabase": "test"
            },
            "id": 2,
            "host": "redshift.example.com",
            "port": 5439,
            "vendor": "redshift",
            "params": {
                "extraLoadParams": "BLANKSASNULL EMPTYASNULL TRIMBLANKS TRUNCATECOLUMNS",
                "defaultDatabase": "test"
            },
            "ssl": false,
            "vendorName": "redshift",
            "name": "redshift",
            "description": null,
            "type": "jdbc",
            "isGlobal": true,
            "credentialType": "custom",
            "credentialsShared": true,
            "uuid": "097c2300-2f6a-11e9-a585-57562e0d9cd6",
            "disableTypeInference": false,
            "createdAt": "2019-02-13T08:33:28.368Z",
            "updatedAt": "2019-02-13T08:33:28.381Z",
            "credentials": [
                {
                    "iamRoleArn": "arn:aws:iam:something",
                    "username": "UserName"
                }
            ],
            "creator": {
                "id": 1
            },
            "updater": {
                "id": 1
            },
            "workspace": {
                "id": 1
            }
        },
        {
            "connectParams": {
                "vendor": "hive",
                "vendorName": "hive",
                "host": "hadoop",
                "port": "10000",
                "jdbc": "hive2",
                "defaultDatabase": "default"
            },
            "id": 1,
            "host": "hadoop",
            "port": 10000,
            "vendor": "hive",
            "params": {
                "jdbc": "hive2",
                "connectStringOptions": "",
                "defaultDatabase": "default"
            },
            "ssl": false,
            "vendorName": "hive",
            "name": "hive",
            "description": null,
            "type": "jdbc",
            "isGlobal": true,
            "credentialType": "conf",
            "credentialsShared": true,
            "uuid": "08a1a180-2f6a-11e9-b2b2-85d2b0b67f5e",
            "disableTypeInference": false,
            "createdAt": "2019-02-13T08:33:26.936Z",
            "updatedAt": "2019-02-13T08:33:26.952Z",
            "credentials": [],
            "creator": {
                "id": 1
            },
            "updater": {
                "id": 1
            },
            "workspace": {
                "id": 1
            }
        }
    ],
    "count": 2
}
 

operation/getConnection

In the above, you identify that the connection used for the exported flow is the Redshift one. This object has the following unique identifier:

"uuid": "097c2300-2f6a-11e9-a585-57562e0d9cd6"

In the target system, you must now create a rule in the deployment into which you are importing that searches for this unique value. In the following example:

• The deploymentId is known to be 4.
• The connectionId for the equivalent Redshift connection in the target system is 1.

The uuid field in the import package is searched for the matching string. If it is found, the connection in the import package is replaced with the connection in the target system with an Id of 1:

/v4/deployments/4/objectImportRules

[
  {
    "tableName": "connections",
    "onCondition": {
      "uuid": "097c2300-2f6a-11e9-a585-57562e0d9cd6"
    },
    "withCondition": {
      "id": 1
    }
  }
]

{
    "deleted": {
        "data": []
    }
}

operation/updateObjectImportRules

To test your rule, perform a dry run of the import. See below.

Example - Remap an HDFS location

In this example, your import rule must remap the path to the source from your Dev paths to your Prod paths. Suppose the pattern looks like this:

Note the differences:

• The /dev/ part of the path has been replaced by /prod/.
• The filename is different.

You can use the following value import rules to change the path values. In the following example, the rules are applied separately.

Request:

/v4/deployments/4/valueImportRules

Request Body:

[
  {"type":"fileLocation","on":"/\/dev\//","with":"/prod/"},
  {"type":"fileLocation","on":"/\/([a-zA-Z0-9_]*).csv/","with":"$1-Prod.csv"}
]

Response:

{
    "deleted": {
        "data": []
    }
}

operation/updateValueImportRules

To test your rule, perform a dry run of the import. See below.

Example - Remap an S3 location

For S3 sources, you can apply remapping rules including changing to a new S3 bucket. 

In this example, your import rule must remap the path to the source from your Dev paths to your Prod paths. Suppose the pattern looks like this:

You can use the following value import rules to change the bucket name and path values.

s3Bucket name rule: This rule replaces the name of the S3 bucket to use with the new one: wrangle-prod.

fileLocation rule: This rule uses regular expressions to match each segment of the path in the source bucket's paths. 

• Files are located at a consistent depth in the source bucket.
• Path segments and filename use only alphanumeric values and underscores (_). 
• The replacement path is shortened to contain only the parent name ($2) and the filename ($4) in the path.

Request:

/v4/deployments/4/valueImportRules

Request Body:

[
  {"type":"s3Bucket","on":"wrangle-dev","with":"wrangle-prod"},
  {"type":"fileLocation","on":"/\/([a-zA-Z0-9_]*)\/([a-zA-Z0-9_]*)\/([a-zA-Z0-9_]*)\/([a-zA-Z0-9_]*).csv/","with":"/$2/$4.csv"}
]

Response:

{
    "deleted": {
        "data": []
    }
}

operation/updateValueImportRules

To test your rule, perform a dry run of the import. See below.

Example - Remap a WASB location

For WASB sources, you can apply remapping rules during import. 

In this example, your import rule must remap the blob host, container, and file location:

You can use the following value import rules to change the blobhost, container, and file paths.

host rule: This rule replaces the blobhost name to use with the new one: storage-wasb-account-prod.blob.core.windows.net.

userinfo rule: This rule replaces the container name to use with the new one: container-prod.

fileLocation rule: This rule performs a text substitution to replace the file path. This rule applies to both input and output object file paths.

Request:

/v4/deployments/4/valueImportRules

Request Body:

[
  {"type":"host","on":"storage-wasb-account-dev.blob.core.windows.net","with":"storage-wasb-account-prod.blob.core.windows.net"},
  {"type":"userinfo","on":"container-dev","with":"container-prod"},
  {"type":"fileLocation","on":"/projs/work/orders.csv","with":"/2003/transactions/orders.csv"}
]

Response:

{
    "deleted": {
        "data": []
    }
}

operation/updateValueImportRules

To test your rule, perform a dry run of the import. See below.

Example - Remap an ADLS Gen2 location

For ADLs Gen2 sources, you can apply remapping rules during import. 

In this example, your import rule must remap the storage account, filesystem, and file location:

You can use the following value import rules to change the storage account, filesystem, and file paths.

host rule: This rule replaces the storage account name to use with the new one: storage-adlsgen2-account-prod.blob.core.windows.net.

userinfo rule: This rule replaces the filesystem name to use with the new one: filesystem-prod.

fileLocation rule: This rule performs a text substitution to replace the file path. This rule applies to both input and output object file paths.

Request:

/v4/deployments/4/valueImportRules

Request Body:

[
  {"type":"host","on":"storage-adlsgen2-account-dev.blob.core.windows.net","with":"storage-adlsgen2-account-prod.blob.core.windows.net"},
  {"type":"userinfo","on":"filesystem-dev","with":"filesystem-prod"},
  {"type":"fileLocation","on":"/projs/work/orders.csv","with":"/2003/transactions/orders.csv"}
]

Response:

{
    "deleted": {
        "data": []
    }
}

operation/updateValueImportRules

To test your rule, perform a dry run of the import. See below.

Example - Remap an ADLS Gen1 location

For ADLS Gen1 sources, you can apply remapping rules during import. 

In this example, your import rule must remap the Azure data lake store and file location:

You can use the following value import rules to change the datastore and file paths.

host rule: This rule replaces the datastore name to use with the new one: storage-adlsgen1-account-prod.azuredatalakestore.net.

fileLocation rule: This rule performs a text substitution to replace the file path. This rule applies to both input and output object file paths.

Request:

/v4/deployments/4/valueImportRules

Request Body:

[
  {"type":"host","on":"storage-adlsgen1-account-dev.azuredatalakestore.net","with":"storage-adlsgen1-account-prod.azuredatalakestore.net"},
  {"type":"fileLocation","on":"/projs/work/orders.csv","with":"/2003/transactions/orders.csv"}
]

Response:

{
    "deleted": {
        "data": []
    }
}

operation/updateValueImportRules

To test your rule, perform a dry run of the import. See below.

Example - Remap a relational datasource

When you migrate a relational source from a Dev instance to a Prod instance, you may need to remap your flow to use the production database and table.

In this example, you are replacing the input and output source databases and tables with the corresponding production DB values.

In a single request, you can apply the rules changes to map the above Dev values to the Prod values.

The on parameter accepts regular expressions. In the following example request, the on parameter has been configured to use a regular expression, under the assumption that all current and future imports will respect the current pattern or database paths and table names.

dbTableName rule: This rule replaces the name of the table to use.

dbPath rule: This rule replaces the path value to database table. 

Request:

/v4/deployments/4/valueImportRules

Request Body:

[
  {"type":"dbTableName","on":"/dev_([a-zA-Z0-9_]*)/","with":"prod_$1"},
  {"type":"dbPath","on":"/dev_([a-zA-Z0-9_]*)_src/","with":"prod_$1_out"}
]

Response:

{
    "deleted": {
        "data": []
    }
}

operation/updateValueImportRules

To test your rule, perform a dry run of the import. See below.

Import Dry-Run

After you have specified a set of import rules, you can perform a dry-run of an import of an import package. This dry-run does not perform the actual import but does report any permissions errors or other issues in the response. 

In this example, the flow2import.zip file contains the package to import into deployment 4. 

Request:

/v4/deployments/4/releases/dryRun

Response:

operation/importPackageForDeploymentDryRun

After the above dry-run has been executed, the import package can be imported and is automatically connected to the appropriate connection. See 

operation/importPackageForDeployment