CLI Migration to APIs

Support for the Alteryx Command Line Interface (CLI) has been deprecated. This means:

• A version of the CLI that is compatible with the release will no longer be available for use. 
• Old versions of the CLI will not work with the new version of the platform.

Before you upgrade to the next  Designer Cloud Enterprise Edition release, you must migrate any scripts or other automation projects that currently use the CLI to use the v4 versions of the APIs. This section provides information on how to manage that migration. 

General Differences

API authentication

In CLI usage, you pass authentication username/password with each command. 

In API usage, you must pass some form of authentication as a header in each request. 

For more information, see API Authentication.

Terminology

Depending on the version you are using, please use the following mapping: 

Use internal identifiers

Parameters passed to the CLI are often user-friendly text values. The CLI tool then queries the appropriate REST API endpoint and converts those values to internal identifiers.

When using the APIs, you must reference the internal identifiers directly.

Below is some information on how you can acquire the appropriate internal identifiers for each type of operation supported by the CLI.

Object Identifiers

Get primary object identifiers

For each CLI command, there is an associated object identifier, which is used to uniquely reference the object. To reference the object through the APIs, you must use the API unique id.

Example - Get User Id

The following example steps through the process of acquiring user ids so that you can use the APIs. 

CLI - Get list of usernames:

The CLI references users via their platform usernames.

• If your CLI scripts contain references to individual users, search them for:
  
  

  --username <user_email_address>

• If you want to acquire the list of all available usernames, it's easier to do that via the APIs.

API - Get list of users:

Use the following API endpoint to get the list of all users, including deleted and disabled users. 

Parsing the JSON: 

In the JSON response body, here are the key values for each user entry:

Important notes on CLI packages

Unlike connection, job, or user objects, a CLI script package does not contain any references to platform objects by design. These independent, self-contained objects can be used to run a script snapshot as a job at any time.

After you download, however, the script package is no longer aware of any changes that have occurred to the source objects on the platform, which has the following implications:

1. If source objects, such as the source recipe, have changed, those changes are not present in the CLI package.
   1. The above does not apply to data sources. In the downloaded CLI package, sources are referenced by URL, which means that they should be using the latest data. 
   2. If the data source URL has changed, however, that is not reflected in any previously downloaded CLI packages. 
2. There is no object identifier in the CLI that directly corresponds to a unique identifier in the platform. 

If you download the CLI package each time that you wish to run a job:

• You can acquire the recipe identifier from the Flow View page or Transformer page where you download the CLI package. 

• In Flow View page, select the recipe icon. The URL is the following:
  
  

  http://example.com:3005/flows/11?recipe=39&tab=recipe

  The recipe id is 39.

• In the Transformer page, the URL is the following:
  
  

  http://example.com:3005/data/11/39

  The recipe id is 39.

• This value corresponds to the value to look for in the API WrangledDatasets Get List v4 output.

If you wish to use the latest version of the recipe for running a job:

• Use the Transformer page. See previous.

If you have downloaded and saved off a set of CLI packages for execution as recipe snapshots:

• This use case may be problematic, as there may be differences between the platform version of the recipe and the version in the CLI package. Currently, the platform does not support importing individual recipes or CLI packages. 

Options:

1. Make a copy of the closest approximation to your downloaded recipe. Modify the recipe steps in the application to change the copy to match your local CLI package. 
2. Create a new flow and rebuild the individual steps in the recipe to match your local CLI package.

Reference - CLI for Jobs

In this section, you can review the commands available in the CLI, followed by their equivalent commands using the v4 APIs. 

CLI Docs: CLI for Jobs

Run Job

You can issue commands to the CLI to execute jobs using the local package downloaded from the Recipe panel.

Below are the three files in the package and their API equivalents:

CLI example:

./trifacta_cli.py run_job --user_name <trifacta_user> --password <trifacta_password> --job_type spark 
--output_format json --data redshift-test/datasources.tsv --script redshift-test/script.cli 
--cli_output_path ./job_info.out --profiler on --output_path hdfs://localhost:8020/trifacta/queryResults/foo@trifacta.com/MyDataset/42/cleaned_table_1.json

API v4 example - REST client: 

Default settings: After you have captured the wrangledDataset identifier, you can launch a new job using default settings:

{
  "wrangledDataset" {
    "id", <wrangledDatasetId>
  }
}

Specify job overrides: The above request contains only the wrangledDataset identifier. All default output settings are used. 

If needed, you can override these default settings by specifying values as part of the request body. In the following example, the relevant parameters from the CLI have been added as elements of the JSON body of the request.

Through the APIs, you can also override the default files, formats, and locations where you output results in the writesettings block. 

{
  "wrangledDataset": {
    "id": <wrangled_dataset_id>
  },
  "overrides": {
    "execution": "spark",
    "profiler": true,
    "writesettings": [
      {
        "path": "hdfs://hadoop:50070/trifacta/queryResults/foo@trifacta.com/MyDataset/42/cleaned_table_1.json",
        "action": "create",
        "format": "json",
        "compression": "none",
        "header": false,
        "asSingleFile": false
      }
    ]
  },
  "ranfrom": "cli"
}

Reference Docs:

• API WrangledDatasets Get List v4
• API JobGroups Create v4
  • For more information on modifying output objects and write settings, see:
    • API OutputObjects Create v4
    • API WriteSettings Create v4
    • API Workflow - Manage Outputs

File Publishing Options

You can specify publication options as part of your run_job command. In the following, a single CSV file with headers is written to a new file with each job execution.

Example (All one command):

./trifacta_cli.py run_job --user_name <trifacta_user> --password <trifacta_password> --job_type spark 
--output_format csv --data redshift-test/datasources.tsv --script redshift-test/script.cli 
--publish_action create --header true --single_file true
--cli_output_path ./job_info.out --profiler on --output_path hdfs://localhost:8020/trifacta/queryResults/foo@trifacta.com/MyDataset/43/cleaned_table_1.csv

CLI example:

./trifacta_cli.py run_job --user_name <trifacta_user> --password <trifacta_password> --job_type spark 
--output_format csv --data redshift-test/datasources.tsv --script redshift-test/script.cli 
--publish_action create --header true --single_file true
--cli_output_path ./job_info.out --profiler on --output_path hdfs://localhost:8020/trifacta/queryResults/foo@trifacta.com/MyDataset/43/cleaned_table_1.csv

API v4 example - REST client:

For more information, see API Workflow - Manage Outputs.

Reference Docs:

See API WriteSettings Create v4.

Get Job Status

After you queue a job through the CLI, you can review the status of the job through the application or through the CLI. 

CLI example:

./trifacta_cli.py get_job_status --user_name <trifacta_user> --password <trifacta_password> --job_id 42 
--cli_output_path ./job_info.out

API v4 example - REST client:

Using the jobGroup identifier value, you can query the status of any job that has been launched.

Reference Docs:

See API JobGroups Get v4.

Publish

After a job has successfully completed, you can publish the results to another datastore with which the platform is integrated.

CLI example:

The following command publishes the results of jobId 42 through connectionId 1 to the dev database. Let's assume that this is a Hive database.

./trifacta_cli.py publish --user_name <trifacta_user> --password <trifacta_password> --job_id 42
--database dev --table table_job_42 --conn_name 1 --publish_format avro
--cli_output_path ./publish_info.out

API v4 example - REST client:

You can publish results for a specified jobId to an output that can be specified through properties in the request (see below). 

{  "connection": {
    "id": 1
  },
  "path": ["dev"],
  "table": "table_job_42",
  "action": "create",
  "inputFormat": "avro",
  "flowNodeId": 27
}

{    "jobgroupId": 42,
    "reason": "JobStarted",
    "sessionId": "f6c5f350-2102-11e9-bb80-9faf7b15f235"
}

Reference Docs:

See API JobGroups Put Publish v4.

Get Publications

You can retrieve a JSON list of all publications that have been executed for a specific job.

• A publication is an object that corresponds to the delivery of a job's results to an external datastore.
• In the Designer Cloud application, publications are executed through the Publishing Dialog, which is available through the Job Details page. See Publishing Dialog.

CLI example:

./trifacta_cli.py get_publications --user_name <trifacta_user> --password <trifacta_password> --job_id 42
--cli_output_path ./publications.out --publish_format avro

API v4 example - REST client:

You can use the job Id to retrieve the list of publications that have been executed for that job.

Reference Docs:

See API JobGroups Get Publications v4.

Load Data into Table

You can load data into pre-existing Redshift tables.

• Data is appended after any existing rows.

• If the table does not exist, the job fails.

CLI example:

In the following example, the results of jobId 42 are loaded into a Redshift table called table_42 using connectionId 2. 

./trifacta_cli.py load_data --user_name <trifacta_user> --password <trifacta_password> --job_id 42 
--database dev --table table_42 --conn_id 2 --publish_format avro 
--cli_output_path ./load_info.out

API v4 example - REST client:

In the request body, note that the action parameter is set to load.

{  "connection": {
    "id": 2
  },
  "path": ["dev"],
  "table": "table_42",
  "action": "load",
  "inputFormat": "avro",
  "flowNodeId": 27
}

{    "jobgroupId": 42,
    "reason": "JobStarted",
    "sessionId": "f6c5f350-2102-11e9-bb80-9faf7b15g574"
}

Reference Docs:

API JobGroups Put Publish v4

Truncate and Load

For existing tables, you can clear them and load them with results from a job. If the table does not exist, a new one is created and populated.

CLI example:

./trifacta_cli.py truncate_and_load --user_name <trifacta_user> --password <trifacta_password> --job_id 10 
--database dev --table table_43 --conn_name aSQLServerConnection --publish_format avro 
--cli_output_path ./load_and_trunc_info.out

API v4 example - REST client:

In the request body, note that the action parameter is set to truncateAndLoad.

{  "connection": {
    "id": 2
  },
  "path": ["dev"],
  "table": "table_43",
  "action": "truncateAndLoad",
  "inputFormat": "avro",
  "flowNodeId": 27
}

{    "jobgroupId": 10,
    "reason": "JobStarted",
    "sessionId": "f6c5f350-2102-11e9-bb80-9faf7b15v291"
}

Reference Docs:

API JobGroups Put Publish v4

Reference - CLI for Connections

You can use the CLI for basic management of your connections.

CLI Docs: CLI for Connections

Create Connection

To create a connection, you specify the connection parameters as part of your command line command.

CLI example:

./trifacta_cli.py create_connection --user_name <trifacta_user> --password <trifacta_password>
--conn_type microsoft_sqlserver --conn_name aSQLServerConnection
--conn_description "This is my connection."
--conn_host example.com --conn_port 1234
--conn_credential_type basic
--conn_credential_location ~/.trifacta/config_conn.json
--conn_params_location ~/.trifacta/p.json
--cli_output_path ./conn_create.out

API v4 example - REST client:

{
    "connectParams": {
        "vendor": "sqlserver",
        "vendorName": "sqlserver",
        "host": "example.com",
        "port": "1234"
    },
    "host": "example.com",
    "port": 1234,
    "vendor": "sqlserver",
    "params": {
        "connectStrOpts": ""
    },
    "ssl": false,
    "vendorName": "sqlserver",
    "name": "aSQLServerConnection",
    "description": "",
    "type": "jdbc",
    "isGlobal": false,
    "credentialType": "basic",
    "credentialsShared": true,
    "disableTypeInference": false,
    "credentials": [
        {
            "username": "<username>",
            "password": "<password>"
        }
    ]
}

{
    "connectParams": {
        "vendor": "sqlserver",
        "vendorName": "sqlserver",
        "host": "example.com",
        "port": "1234"
    },
    "id": 26,
    "host": "example.com",
    "port": 1234,
    "vendor": "sqlserver",
    "params": {
        "connectStrOpts": ""
    },
    "ssl": false,
    "vendorName": "sqlserver",
    "name": "aSQLServerConnection",
    "description": "",
    "type": "jdbc",
    "isGlobal": false,
    "credentialType": "basic",
    "credentialsShared": true,
    "uuid": "fa7e06c0-0143-11e8-8faf-27c0392328c5",
    "disableTypeInference": false,
    "createdAt": "2018-01-24T20:20:11.181Z",
    "updatedAt": "2018-01-24T20:20:11.181Z",
    "credentials": [
        {
            "username": "<username>"
        }
    ],
    "creator": {
        "id": 1
    },
    "updater": {
        "id": 1
    }
}

Reference Docs:

API Connections Create v4

Edit Connection

In the CLI, you use the edit_connection action to pass in modifications to a connection that is specified using the conn_name command line parameter. 

CLI example:

In the following example, the description, host, and port number are being changed for the aSQLServerConnection.

./trifacta_cli.py edit_connection --user_name <trifacta_user> --password <trifacta_password>
--conn_name aSQLServerConnection
--conn_description "This is my connection."
--conn_host mynewhost.com --conn_port 1234
--conn_credential_type basic --conn_credential_location ~/.trifacta/config_conn.json
--cli_output_path ./conn_edit.out

API v4 example - REST client:

When using the APIs, you use the internal identifier for the connection to modify.

In the body of the request, you should include only the parameters that you are modifying for the connection. In this example, the connectionId is 8.

{
  "description": "This is my connection.",
  "host": "mynewhost.com",
  "port": 1234
}

{
    "id": 8,
    "updater": {
        "id": 1
    },
    "updatedAt": "2019-01-25T23:19:27.648Z"
}

Reference Docs:

See API Connections Patch v4.

List Connections

The CLI command list_connections dumps the JSON objects for all connections to a local file.

CLI example:

./trifacta_cli.py list_connections --host dev.redshift.example.com
--user_name <trifacta_user> --password <trifacta_password>
--cli_output_path ./conn_list.out

API v4 example - REST client:

The following API endpoint can be used to retrieve the JSON objects for all connections to which the authenticating user has access in the body of the response. 

Reference Docs:

See API Connections Get List v4.

Delete Connection

For the CLI, you use the delete_connection command to remove connections that are specified by conn_name.

CLI example:

./trifacta_cli.py delete_connection --user_name <trifacta_user> --password <trifacta_password>
--conn_name aSQLServerConnection --cli_output_path ./conn_delete.out

API v4 example - REST client:

Use the internal identifier for the connection to delete it. In the following example, the connectionId is 4.

Reference Docs:

See API Connections Delete v4.

Reference - CLI for User Admin

You can use the CLI for handling of some elements of user management. 

CLI Docs: CLI for User Admin

Create User

CLI example: 

./trifacta_admin_cli.py --admin_username <trifacta_admin_username> 
--admin_password <trifacta_admin_password>  --verbose create_user 
--user_name joe@example.com --password user_pwd 
--name "<user_display_name>"

API v4 example - REST client:

The request body below contains the minimum set of required parameters to create a new user. 

• The accept parameter must be set to accept for every new user.

{
  "accept": "accept",
  "password": "Hello2U",
  "password2": "Hello2U",
  "email": "joe@example.com",
  "name": "Joe"
}

{
    "isDisabled": false,
    "forcePasswordChange": false,
    "state": "active",
    "id": 9,
    "email": "joe@example.com",
    "name": "Joe",
    "ssoPrincipal": null,
    "hadoopPrincipal": null,
    "isAdmin": false,
    "updatedAt": "2019-01-09T20:23:31.560Z",
    "createdAt": "2019-01-09T20:23:31.560Z",
    "outputHomeDir": "/trifacta/queryResults/joe@example.com",
    "lastStateChange": null,
    "fileUploadPath": "/trifacta/uploads",
    "awsConfig": null
}

Reference Docs:

API People Create v4

Show User

You can gather a specific user object using the username through the CLI.

CLI example:

./trifacta_admin_cli.py --admin_username <trifacta_admin_user> --admin_password <trifacta_admin_password> 
show_user --user_name joe@example.com 

API v4 example - REST client:

Through the APIs, you can retrieve individual users through the internal userId. In the following example, the user corresponding to userId 4 is retrieved.

{
    "id": 4,
    "email": "joe2@example.com",
    "name": "Joe2",
    "ssoPrincipal": null,
    "hadoopPrincipal": null,
    "isAdmin": false,
    "isDisabled": false,
    "forcePasswordChange": false,
    "state": "active",
    "lastStateChange": null,
    "createdAt": "2019-02-20T20:05:49.882Z",
    "updatedAt": "2019-02-20T20:05:49.882Z",
    "outputHomeDir": "/trifacta/queryResults/joe@example.com",
    "fileUploadPath": "/trifacta/uploads",
    "awsConfig": null
}

Reference Docs:

API People Get v4.

Edit User

You can edit some properties through the CLI edit_user command.

CLI example:

In this example, the ssoPrincipal for the user is being changed.

./trifacta_admin_cli.py --admin_username <trifacta_admin_user> --admin_password <trifacta_admin_password> 
edit_user --user_name joe2@example.com --ssoPrincipal my_principal

API v4 example - REST client:

Using the APIs, you reference the user to modify by userId. In the following example, the userId is 4. 

Include only the parameters in the request that are being modified.

{
  "ssoPrincipal": "my_principal"
}

{  "id": 4,
  "updatedAt": "2018-01-24T23:49:08.199Z"
}

Reference Docs:

API People Patch v4

Generate Password Reset URL

Through the CLI, admins can generate password reset emails to be sent to specific users.

CLI example:

./trifacta_admin_cli.py --admin_username <trifacta_admin_user> --admin_password <trifacta_admin_password> 
edit_user --user_name joe2@example.com --disable

API v4 example - REST client:

Use the following endpoint to generate a password reset code for the specified user (accountId). 

{
  "accountId": 6,
  "email": "joe@example.com",
  "orginURL": "http://loginpage.example.com:3005/"
}

{
    "code": "<AccountResetCode>",
    "email": "joe@example.com"
}

The above must be built into a URL in the following format:

<http|https>://<host_name>:<port_number>/password-reset?email=<user_id>&code=<AccountResetCode>

<http|https>

<host_name>

loginpage.example.com

<port_number>

<user_id>

<AccountResetCode>

Reference Docs:

API Password Reset Request Create v4

Disable User

Through the CLI, you can disable individual users by adding the disable flag as part of an edit_user directive.

CLI example:

./trifacta_admin_cli.py --admin_username <trifacta_admin_user> --admin_password <trifacta_admin_password> edit_user --user_name joe@example.com --disable

API v4 example - REST client:

In the APIs, you disable a specified user by patching the user object with the disable flag.

{
  "isDisabled": false
}

{  "id": 4,
  "updatedAt": "2018-01-24T23:56:32.834Z"
}

Reference Docs:

API People Patch v4

Delete User

CLI example:

In the following example, the user is deleted by username, and the user's assets are transferred to another user.

./trifacta_admin_cli.py --admin_username <trifacta_admin_user> --admin_password <trifacta_admin_password> 
delete_user --user_name joe@example.com --transfer_assets_to jim@example.com

API v4 example - REST client:

Via the APIs, this transfer of assets and deletion of the user must be accomplished in two steps.

Here is the mapping of example userIds between CLI and API:

Transfer of assets:

The following endpoint call transfers assets from userId 4 to userId 7.

[
    [
        1,
        [
            0,
            [
                {
                    "connectionId": 7,
                    "personId": 7,
                    "role": "owner",
                    "createdAt": "2019-02-21T19:52:22.993Z",
                    "updatedAt": "2019-02-21T19:52:22.993Z"
                }
            ]
        ]
    ]
]

Delete user:

After assets have been transferred, users can be deleted by userId (4).

Reference Docs:

API People Delete v4