Overview

After you have created a flow, imported a dataset, and created a recipe for that dataset, you may need to swap in a different dataset and run the recipe against that one. This workflow steps through that process via the APIs.

This workflow utilizes the following methods:

1. Creating an imported dataset. After the new file has been added to the backend datastore, you can import into  as an imported dataset.

2. Swap dataset. Using the ID of the imported dataset you created, you can now assign the dataset to the recipe in your flow. 
3. Run a job. Run the job against the dataset. 
4. Monitor progress. Monitor the progress of the job until it is complete.

Example Datasets

In this example, you are wrangling data from orders placed in different regions on a quarterly basis. When a new file drops, you want to be able to swap out the current dataset that is assigned to the recipe and swap in the new one. Then, run the job.

Example Files:

The following files are stored in HDFS:

Assumptions

You have already created a flow, which contains the following imported dataset and recipe:

Base URL:

For purposes of this example, the base URL for the platform is the following:

http://www.example.com:3005

Step - Import Dataset

The following steps describe how to create an imported dataset and assign it to the flow that has already been created (flowId=2).

Steps:

1. To create an imported dataset, you must acquire the following information about the source. 

   1. path
   2. type
   3. name
   4. description
   5. bucket (if a file stored on S3)

2. In this example, the file you are importing is MyCo-orders-west-Q2.txt. Since the files are similar in nature and are stored in the same directory, you can acquire this information by gathering the information from the imported dataset that is already part of the flow. Execute the following:
   

   Endpoint	http://www.example.com:3005/v4/importedDatasets
   Authentication	Required
   Method	POST
   Request Body	{ "path": "hdfs:///user/orders/MyCo-orders-west-Q2.txt", "name": "MyCo-orders-west-Q2.txt", "description": "MyCo-orders-west-Q2" }

   
   

3. The response should be a 201 - Created status code with something like the following:

   { "id": 12, "size": "281032", "path": "hdfs:///user/orders/MyCo-orders-west-Q2.txt", "dynamicPath": null, "workspaceId": 1, "isSchematized": false, "isDynamic": false, "disableTypeInference": false, "createdAt": "2018-10-29T23:15:01.831Z", "updatedAt": "2018-10-29T23:15:01.889Z", "parsingRecipe": { "id": 11 }, "runParameters": [], "name": "MyCo-orders-west-Q2.txt.txt", "description": "MyCo-orders-west-Q2.txt", "creator": { "id": 1 }, "updater": { "id": 1 }, "connection": null }

4. You must retain the id value so you can reference it when you create the recipe.

5. See 

   operation/createImportedDataset

Step - Swap Dataset from Recipe

The next step is to swap the primary input dataset for the recipe to point at the newly imported dataset. This step automatically adds the imported dataset to the flow and drops the previous imported dataset from the flow. 

1. Use the following to swap the primary input dataset for the recipe:

   Endpoint	http://www.example.com:3005/v4/wrangledDatasets/9/primaryInputDataset
   Authentication	Required
   Method	PUT
   Request Body	{ "importedDataset": { "id": 12 } }

2. The response should be a 200 - OK status code with something like the following:

   { "id": 9, "wrangled": true, "createdAt": "2019-03-03T17:58:53.979Z", "updatedAt": "2019-03-03T18:01:11.310Z", "recipe": { "id": 9, "name": "POS-r01", "description": null, "active": true, "nextPortId": 1, "createdAt": "2019-03-03T17:58:53.965Z", "updatedAt": "2019-03-03T18:01:11.308Z", "currentEdit": { "id": 8 }, "redoLeafEdit": { "id": 7 }, "creator": { "id": 1 }, "updater": { "id": 1 } }, "referenceInfo": null, "activeSample": { "id": 7 }, "creator": { "id": 1 }, "updater": { "id": 1 }, "referencedFlowNode": null, "flow": { "id": 2 } }

3. The new imported dataset is now the primary input for the recipe, and the old imported dataset has been removed from the flow. 

operation/updateInputDataset

Step - Swap Bucket and Path for Imported Dataset

For a file-based backend datastore, you can change the source of your imported dataset to use a different bucket and path for your imported dataset.

1. Use the following to swap the source of your imported dataset for the recipe:

   Endpoint	http://www.example.com:3005/v4/importedDatasets/9/
   Authentication	Required
   Method	PUT
   Request Body	{ "bucket": "MyBucket", "path": "/path/to/my/file.csv", }

2. The response should be a 200 - OK status code with the imported dataset definition.

3. The new definition of the imported dataset is now applicable to all objects that reference it.

operation/updateImportedDataset

Step - Rerun Job

To execute a job on this recipe, you can simply re-run any job that was executed on the old imported dataset, since you reference the job by jobId and wrangledDataset (recipe) Id. 

{
  "wrangledDataset": {
    "id": 9
  }
}

The job is re-run as it was previously specified.

If you need to modify any job parameters, you must create a new job definition.

Step - Monitor Your Job

After the job has been queued, you can track it to completion. See API Workflow - Develop a Flow.

Step - Schedule Your Job

When you are satisfied with how your flow is working, you can set up periodic schedules using a third-party tool to execute the job on a regular basis. 

The tool must hit the above endpoints to swap in the new dataset and run the job.