Overview of Deployment Manager
You can deploy flows that you have created into a separate, production environment where jobs for those flows can be executed on a periodic or scheduled basis. In this manner, you can create separation between your development and production environments and their flows. The Deployment Manager includes the tools to migrate your software between environments, manage releases of it, and separately control access to development and production flows.
Deployment Manager enables the transfer of flows between development and production instances of the platform. A customer may have one or more instances of the platform.
For managing user access to flows within the same development instance, you can use sharing. See Overview of Sharing.
This feature was formerly known as, "deployment management."
Key Features:
Development environment:
Export of flows and all dependent objects
Import back into Development deployments for further development
Production environment:
Import of flows
Import global and object-level mapping rules
Manage releases of flows
Rollback to previous versions as needed
APIs to manage deployments
Dev/Test and Prod Deployments
In a typical environment, deployments may be segmented between Development (Dev), Testing (Test), and Production (Prod) environments. With respect to the Designer Cloud Powered by Trifacta platform, these deployments break down into the following:
Note
In some cases, Dev and Test may be the same instance.
Note
Multiple browser tabs or windows open to different versions of the product is not supported.
Platform instance | Description |
|---|---|
Development (Dev) | New flows and recipes are created in a Development instance of the platform. Experiments can be undertaken without concern that production use of the recipe or flow is affected. Tip You should do all of your recipe development and testing in Dev/Test. Avoid making changes in a Prod environment. Rules should be established on how flows, datasets, and recipes are organized and structured. Where are these assets stored? Where are shared versions of them made available? What are the rules by which items in Dev can be moved to Test/Prod? |
Testing (Test) | In the Testing deployment, the objects in development are subjected to various stress tests. In the Designer Cloud Powered by Trifacta platform, this testing can include load testing, malformed inputs, and changes to any parameters affecting the use of the object. For example, scheduled executions of flows should be thoroughly tested in this deployment. When errors are detected, they can be corrected in Dev or Test. Ideally, they are first applied in Test to address the issue at hand. Changes should then be applied back into the Dev deployment, so that future versions can consume the fix. |
Production (Prod) | In the Production deployment, flows and their objects are presumed to be ready for regular, read-only use. After imported flows are reconfigured for the environment, they are ready for immediate use and require no further modification.
When errors are detected, you can:
|
Implementation in the platform
In the Designer Cloud Powered by Trifacta platform, deployment management can be addressed in either of the following ways.
Implementation type | Description |
|---|---|
Separate environments: Multiple instances of the platform | Dev, Test, or both environments are separate instances of the Designer Cloud Powered by Trifacta platform from the Production environment. Flows are migrated between environments using the export/import mechanisms. Note Each platform instance is configured to be either a Dev instance or a Prod instance. |
All-in-one: Single instance of the platform, separate roles | Dev, Test, and Prod are contained in a single instance of the Designer Cloud Powered by Trifacta platform. This scenario can apply to cloud-based environments as well. A user can access either Dev/Test or Prod, but not both at the same time. In this scenario, a user can access Production deployments by having the Deployment account role. |
Tip
Access to the Production environments should be tightly controlled to prevent inadvertant changes to Production jobs.
Terminology
Production environment terminology changes
A Prod environment focuses on management of the following objects. Differences between how these objects are used in a Dev environment are noted below.
Note
Some objects are available only in the Production environment. These objects are described later.
Object | Differences |
|---|---|
Flows | In a Prod environment, you can review a flow through Flow View. Note Avoid making changes to your flows in a Prod environment. Any changes in the Prod version should be exported and then imported to the Dev version. Otherwise, when the next release is imported as a package into the Prod environment, those changes are lost. |
Jobs | In the Prod environment, you can execute jobs against Prod flows. For the version of the flow that is active, you trigger a job for its overall deployment. Details are below. These jobs are accessible through an interface that is very similar to a Dev environment. |
The following flow objects from the Dev environment must be replaced in the Prod environment:
Dev Object | Replacement |
|---|---|
Connections | Any connections used in the Dev system must be recreated or replaced with connections in the Production system. |
Output Objects | Output objects from the Dev flow must be recreated or replaced in Flow View in the Prod environment. |
Imported Datasets | If the Prod environment is not using the same sources as the Dev environment, you must create import rules to remap the point the flow to use imported datasets that are stored in a different location for the Prod environment. |
Deployment Objects
In a Prod environment, you can explore the following objects, which are organized in a hierarchy:
Level | Item | Description |
|---|---|---|
1 | deployment | A deployment is a versioned set of releases that have been uploaded to the Prod instance for use. You can think of it as a production instance of your primary flow and its dependencies. |
2 | release | A release is a specific instance of a package that has been imported to the Prod instance. Each time you import, you create a new release within the deployment where you imported. A release is created whenever you import a package into a deployment. A package is a ZIP file containing a flow definition that has been exported from an instance of the Designer Cloud Powered by Trifacta platform. |
3 | flows | Within a release, you can explore the primary flow and any upstream flows that were included in the package. Each flow can be explored through a version of the Flow View page.
|
Enable Deployment
This feature must be enabled through configuration. When enabled, the user experience of the product changes significantly, and a number of features are no longer available, including the Transformer page and its ability to modify recipes.
Tip
When you initially set up a platform instance, you should decide whether it is a Dev instance, a Prod instance or both.
User management
For more information on how to configure user accounts for Deployment Manager, see Configure Deployment Manager.
Import/Export
To transfer your flows between instances, you must export the flow from one instance of the platform and import it into the other instance of the platform.
Note
If Dev and Prod are in the same instance, you must export the flow and import it into a deployment. These are separate processes.
Note
As part of the import process, you must define rules for how objects and values contained in the imported flow definition are remapped in the Prod environment. See below.
Exported Flows
Through Flow View or the Flows page, you can export the flow through the context menu. The export is a ZIP file called a package.
Note
You must be the owner of a flow to export it.
A package ZIP contains all objects required to reconstruct and use the flow in a new environment.
It includes the exported flow and any flows on which it depends.
It does not include data, samples, or jobs.
If the outputs of an exported flow require imported datasets or recipes from another flow, that entire flow is included as part of the export package. This package includes objects that may not be required to run the primary exported flow.
Connections
In the target instance, connections must be created prior to import. You may need to create import mapping rules to use this connections. See Connections Page.
Import
How a flow is imported depends on the environment into which you are importing it and how you intend to use it.
Note
If a flow is imported into an instance that is different from the instance where it was created, you must first create remapping rules for values and objects contained in the flow definition. More information is provided below.
For more information, see Import Flow.
Value and Object Mapping Rules
When objects are moved between environments, paths and other object-related references may require updating to point to the new environment.
Note
Import mapping rules do not work for parameterized datasets. If the imported dataset with parameters is still accessible, you should be able to run jobs from it.
For example, a dataset in the Dev environment may be pointing to the following location:
hdfs:///mydata-dev/1/00005a1a-81b0-4e4d-9c9b-f42ce55e1dde/Open_Order.csv
For the Prod version, the flow may need to be changed to the following:
hdfs:///mydata-prod/1/11115z4a-92f5-9f91-7v7f-g22fk99f2rru/Open_Order.csv
To support this kind of remapping, you can specify import rules at the level of individual deployments.
Note
For each deployment that you create, you must define new import remapping rules.
These rules can be specified using literal values, Alteryx patterns, or regular expressions. For more information, see API Task - Define Deployment Import Mappings.
Production Environment
When a user accesses a Production environment, the UI is changed to include only the following pages:
Note
You cannot modify recipes within a Prod instance because the Transformer page is not available. The Prod flow must be exported and re-imported into a Dev instance.
Page | Description |
|---|---|
On this page, you create deployments, for which you manage import of packages, activation of releases, and rollback to previous release as needed. For more information on the deployment objects, see below. | |
Within a specific release, you can review and update the flow definition, including specification of outputs and schedules. Flow View for a Prod instance has some restrictions. Note Use of scheduling through Flow View of a Prod instance is not supported. When a new release of a flow is imported, the schedule still points to the older release and is orphaned until the old release is reactivated or the schedule or release is removed. | |
Same as Dev instance. No changes. | |
Connections that have been included as part of imported packages are available for review through the Production environment. | |
Same as Dev instance. No changes to the interface. Note In a multi-instance environment, some settings do not apply to the Prod environment. |
Version Management
When you explore a deployment, you can see the list of releases pertaining to the deployment, with the active release listed at the top of the list. The active release is the one that is triggered for execution when a job is run.
You can roll back to using previous releases. Select Activate from the context menu for the desired release.
Note
Do not use scheduling features available through the user interface in a Production instance. If you have defined schedules through Flow View in the Prod instance and then add a new release, the schedules in the previous release are still available. You must remove them to prevent scheduled executions of outdated flows.
Flow View Page
In a Prod instance, you can drill into a release to review its flows through Flow View page.
Note
Avoid making modifications to the flow in a Prod instance.
Example Task
In this example, your environment contains separate Dev and Prod instances, each of which has a different set of users.
Item | Dev | Prod |
|---|---|---|
Environment | http://wrangle-dev.example.com:3005 | http://wrangle-prod.example.com:3005 |
User | User1 Note User1 has no access to Prod. | Admin2 |
Source DB | devWrangleDB | prodWrangleDB |
Source Table | Dev-Orders | Prod-Orders |
Connection Name | Dev Redshift Conn | Prod Redshift Conn |
Example Flow:
User1 is creating a flow, which is used to wrangle weekly batches of orders for the enterprise. The flow contains:
A single imported dataset that is created from a Redshift database table.
A single recipe that modifies the imported dataset.
A single output to a JSON file.
Production data is hosted in a different Redshift database. So, the Prod connection is different from the Dev connection.
Steps:
Build in Dev instance: User1 creates the flow and its steps.
Export: When User1 is ready to push the flow to production, User1 exports the flow from the Flows page and delivers the export package ZIP to Admin2. See Export Flow.
Deploy to Prod instance:
Admin2 creates a new deployment in the Prod instance. See Deployment Manager Page.
Admin2 creates a new connection (Prod Redshift Conn) to the Redshift database ProdWrangleDB. See Create Connection Window.
Admin2 creates an import rule to map the old connection (Dev Redshift Conn) to the new one (Prod Redshift Conn). See API Task - Define Deployment Import Mappings.
Admin2 uploads the export ZIP package provided by User1. See Import Flow.
The deployment now contains a single release.
Test deployment:
Through Flow View in the Prod instance, Admin2 runs a job.
In reviewing the profile results of the job, Admin2 discovers a problem with the recipe. One column contains a number of mismatched values.
Admin2 chooses to fix in Dev and re-import into Prod.
Note
Any changes made in Production that must appear in future releases must be applied back in the Dev environment, too. You can either 1) export the flow from Prod and import back into Dev, or 2) manually apply all Prod changes back to the Dev environment and export/import into Prod when ready.
Fix in development: Back in the Dev environment, Admin2 opens the recipe for the flow.
Admin2 adds a step to the recipe to delete the rows containing mismatched values for the column.
Admin2 runs a job and verifies that the problem is fixed. In the visual profile for the dataset, the mismatched rows are removed from the dataset.
Deploy again: Admin2 exports the flow and imports it again as a new release in the deployment.
Since import rules have already been created for this deployment, the connection is automatically re-mapped for this second import.
Admin2 runs a job. The results look fine.
Admin2 removes profiling from the output object, since profiling takes time and is unnecessary in this production environment.
Set schedule: Using cron, Admin2 sets a schedule to run the active release for this deployment once per week.
Each week, the Prod-Orders table must be refreshed with data.
The dataset is now operational in the Prod environment.
Recommended Practices
If possible, you should maintain separate instances of the platform for Dev and Prod.
If you must use the All-in-One method of managing Dev and Prod instances, you should maintain a small number of non-admin accounts that are specifically used for Deployment Manager.
Avoid scheduling Prod executions through Flow View. While possible, these schedules continue to exist even if the version of the flow has been replaced by another. Consequently, schedules that were specified through the application continue to execute, even though the flow itself is outdated. Instead, scheduled executions should be specified at the command line through cron jobs pointing at the latest release of each at all times.
Do not modify Flow View settings through a Prod instance. These settings are not applied back to the Dev version and are lost when the next release package is imported.
Job Execution
On-demand jobs
You can configure jobs on-demand through the Flow View page of a Production instance. See Flow View Page.
Scheduled jobs
In Dev:
When your flow is exported from a Dev instance, all scheduling-related data is removed from the export package.
In Prod:
In a Prod instance, an imported flow contains no schedules. You must configure schedules through the REST APIs to execute on the currently active release for each deployment.
Note
Do not schedule executions through Flow View in a Prod instance.
Schedules defined in Flow View are applied to Active and Non-Active releases in Production environments.
If the scheduled release is deactivated, the schedule still exists, and the jobs are executed on an flow that is now out-of-date.
Automation
Automation of Deployment Manager is supported through the APIs.
Note
When you run a deployment, you run the primary flow in the active release for that deployment. Running the flow generates the output objects for all recipes in the flow.
Note
Scheduled execution of jobs in a deployment environment must be managed through external tools such as cron. For more information on the endpoint to schedule, see https://api.trifacta.com/ee/9.7/index.html#operation/runDeployment
For more information on the APIs for Deployment Manager, see API Reference.
For more information on an API-based method for deploying flows, see API Task - Deploy a Flow.