Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info

NOTE: Avoid creating datasets with parameters where individual files or tables have differing schemas. Either import these sources separately and then correct in the application before performing a union on the datasets, or make corrections in the source application to standardize the schemas.

 


When working with datasets with parameters, it may be useful to do the following if you expect the underlying datasets to be less than 100% consistent with each other. 

...

  1. In the Import Data page, navigate your environment to locate one of the files or tables that you wish to parameterize.  
  2. Click Create Dataset with Parameters

     

    D caption
    Create Dataset with Parameters


  3. Within the Define Parameterized Path, select a segment of text. Then select one of the following options:

    Tip

    Tip: For best results when parameterizing directories in your file path, include the trailing slash (/ ) as part of your parameterized value.


    1. Add Datetime Parameter
    2. Add Variable
    3. Add Pattern Parameter - wildcards and patterns
    4. If you need to navigate elsewhere, select Browse.
  4. Specify the parameter. Click Save.
  5. Click Update matches. Verify that all of your preferred datasets are matching.

    Info

    NOTE: If you are matching with more datasets than you wish, you should review your parameters.


  6. Click Create.

  7. The parameterized dataset is loaded. 

...

For example, suppose you have two environments: Dev and Prod. You can create an environment parameter called env.sourceBucketName to store the name of the bucket from which all data in the workspace or project is imported.

Environment NameSource Bucket NameEnvironment Parameter Value
DevMyCo_Dev


Code Block
$env.sourceBucketName = 'MyCo_Dev'


ProdMyCo_Prod


Code Block
$env.sourceBucketName = 'MyCo_Prod'


For more information, see Environment Parameters Page.

...