Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info

NOTE: In some cases, Dev and Test may be the same instance.


Info

NOTE: Multiple browser tabs or windows open to different versions of the product is not supported.



Platform instanceDescription
Development (Dev)

New flows and recipes are created in a Development instance of the platform. Experiments can be undertaken without concern that production use of the recipe or flow is affected.

Tip

Tip: You should do all of your recipe development and testing in Dev/Test. Avoid making changes in a Prod environment.

 

Rules should be established on how flows, datasets, and recipes are organized and structured. Where are these assets stored? Where are shared versions of them made available? What are the rules by which items in Dev can be moved to Test/Prod?

Testing (Test)

In the Testing deployment, the objects in development are subjected to various stress tests. In the

D s platform
, this testing can include load testing, malformed inputs, and changes to any parameters affecting the use of the object.For example, scheduled executions of flows should be thoroughly tested in this deployment.

When errors are detected, they can be corrected in Dev or Test. Ideally, they are first applied in Test to address the issue at hand. Changes should then be applied back into the Dev deployment, so that future versions can consume the fix.
Production (Prod)

In the Production deployment, flows and their objects are presumed to be ready for regular, read-only use. After imported flows are reconfigured for the environment, they are ready for immediate use and require no further modification.

  • Management of flows and jobs is typically handled via API.
  • The UI should be used for checking and modifying settings and perform on-demand job executions to verify operations.

When errors are detected, you can:

  • Revert to a previous version of the flow
  • Apply any fixes in the Dev/Test instance for refinement and eventual updating back to the Prod instance.

...

When objects are moved between environments, paths and other object-related references may require updating to point to the new environment.  

Info

NOTE: Import mapping rules do not work for parameterized datasets. If the imported dataset with parameters is still accessible, you should be able to run jobs from it.

For example, a dataset in the Dev environment may be pointing to the following location:

Code Block
hdfs:///mydata-dev/1/00005a1a-81b0-4e4d-9c9b-f42ce55e1dde/Open_Order.csv

...

Code Block
hdfs:///mydata-prod/1/11115z4a-92f5-9f91-7v7f-g22fk99f2rru/Open_Order.csv

To support this kind of remapping, you can specify import rules at the level of individual deployments.


Info

NOTE: For each deployment that you create, you must define new import remapping rules.

These rules can be specified using literal values,

D s item
itempatterns
, or regular expressions. For more information, see Define Import Mapping Rules.

...