Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info

NOTE: In some cases, Dev and Test may be the same instance.


Info

NOTE: Multiple browser tabs or windows open to different versions of the product is not supported.


Platform instanceDescription
Development (Dev)

New flows and recipes are created in a Development instance of the platform. Experiments can be undertaken without concern that production use of the recipe or flow is affected.

Tip

Tip: You should do all of your recipe development and testing in Dev/Test. Avoid making changes in a Prod environment.

 

Rules should be established on how flows, datasets, and recipes are organized and structured. Where are these assets stored? Where are shared versions of them made available? What are the rules by which items in Dev can be moved to Test/Prod?

Testing (Test)

In the Testing deployment, the objects in development are subjected to various stress tests. In the

D s platform
, this testing can include load testing, malformed inputs, and changes to any parameters affecting the use of the object.For example, scheduled executions of flows should be thoroughly tested in this deployment.When errors are detected, they can be corrected in Dev or Test. Ideally, they are first applied in Test to address the issue at hand. Changes should then be applied back into the Dev deployment, so that future versions can consume the fix.

Production (Prod)

In the Production deployment, flows and their objects are presumed to be ready for regular, read-only use. After imported flows are reconfigured for the environment, they are ready for immediate use and require no further modification.

  • Management of flows and jobs is typically handled via API.
  • The UI should be used for checking and modifying settings and perform on-demand job executions to verify operations.

When errors are detected, you can:

  • Revert to a previous version of the flow
  • Apply any fixes in the Dev/Test instance for refinement and eventual updating back to the Prod instance.

...

When objects are moved between environments, paths and other object-related references may require updating to point to the new environment. For example, a dataset in the Dev environment may be pointing to the following location: 

Code Block
hdfs:///mydata-dev/1/00005a1a-81b0-4e4d-9c9b-f42ce55e1dde/Open_Order.csv

For the Prod version, the flow may need to be changed to the following:


Code Block
hdfs:///mydata-prod/1/11115z4a-92f5-9f91-7v7f-g22fk99f2rru/Open_Order.csv


 

 

To support this kind of remapping, you can specify import rules at the level of individual deployments.


Info

NOTE: For each deployment that you create, you must define new import remapping rules.

These rules can be specified using literal values,

D s item
itempatterns
, or regular expressions. For more information, see Define Import Mapping Rules.

...

  • If possible, you should maintain separate instances of the platform for Dev and Prod. 
    • If you must use the All-in-One method of managing Dev and Prod instances, you should maintain a small number of non-admin accounts that are specifically used for Deployment Manager.
  • Avoid scheduling Prod executions through Flow View. While possible, these schedules continue to exist even if the version of the flow has been replaced by another. Consequently, schedules that were specified through the application continue to execute, even though the flow itself is outdated. Instead, scheduled executions should be specified at the command line through cron jobs pointing at the latest release of each at all times.
  • Do not modify Flow View settings through a Prod instance. These settings are not applied back to the Dev version and are lost when the next release package is imported.

Job Execution

Info

NOTE: Running a package containing more than 5 concurrent jobs is not supported.

On-demand jobs

You can configure jobs on-demand through the Flow View page of a Production instance. See Flow View Page.

...