...

Excerpt

To prevent overwhelming the client or significantly impacting performance,  D s product r true  generates one or more samples of the data for display and manipulation in the client application. Since  D s product  supports a variety of clients and use cases, you can change the size of samples, the scope of the sample, and the method by which the sample is created. This section provides background information on how the product manages dataset sampling.

How Sampling Works

Info

NOTE: Generated samples are created by executing jobs on the applicable running environment. Quick Scan samples are executed in D s photon . Full Scan samples are generated in the applicable running environment on the cluster. Each running environment has a proprietary method of calculating the available volume of data in memory which is used for executing the sampling job that is launched in the running environment. As a result, the number of rows returned for the same sample type across different running environments can vary significantly.

Initial Data

When a dataset is first created, a background job begins to generate a sample using the first set of rows of the dataset. This initial data sample is usually very quick to generate, so that you can get to work right away on your transformations.

• The default sample is the initial sample.
• If your source of data is a directory containing multiple files, the initial sample for the combined dataset is generated from the first set of rows in the first filename listed in the directory.
  

  • The maximum number of files in a directory that can be read in the initial sample is limited by parameter for performance reasons. 

• • • For more information, see Workspace Settings Page.

...

• • If you are wrangling a dataset with parameters, the initial sample loaded in the Transformer page is taken from the first matching dataset.

• If the matching file is a multi-sheet Excel file, the sample is taken from the first sheet in the file.

• By default, each initial sample is either: 
  • 10 MB in size
  or the entire dataset if it's smaller.  
  • Limited by the maximum number of files
  • The entire dataset
• If the source data is larger than 10MB in size, a random sample is automatically generated for you when the recipe is first loaded in the Transformer page. 
  • The initial sample is selected by default. When the automatic random sample has finished generation, it can be manually selected for display.
  If your source of data is a directory containing multiple files, the initial sample for the combined dataset is generated from the first set of rows in the first filename listed in the directory.
  • If the matching file is a multi-sheet Excel file, the sample is taken from the first sheet in the file.If you are wrangling a dataset with parameters, the initial sample loaded in the Transformer page is taken from the first matching dataset.

Generating samples

Additional samples can be generated from the context panel on the right side of the Transformer page. Sample jobs are independent job executions. When a sample job succeeds or fails, a notification is displayed for you.

As you develop your recipe, you might need to take new samples of the data. For example, you might need to focus on the mismatched or invalid values that appear in a single column. Through the Transformer page, you can specify the type of sample that you wish to create and initiate the job to create the sample. This sampling job occurs in the background.

You can create a new sample at any time. When a sample is created, it is stored within your storage directory on the backend datastore.

Info
NOTE: The Initial Data sample contains raw data from the source. Any generated sample is stored in JSONLines format with additional metadata on the sample. These different storage formats can result is differences between initial and generated sample sizes.

For more information on creating samples, see Samples Panel.

...

Important notes on sampling

• Sampling Depending on the running environment, sampling jobs may incur costs. These costs may vary between

  D s photon

  and your clustered running environments, depending on type of sample and cost of job execution.
• When sampling from compressed data, the data is uncompressed and then expanded. As a result, the sample size reflects the uncompressed data.
• Changes to preceding steps that alter the number of rows or columns in your dataset can invalidate the current sample, which means that the sample is no longer a valid representation of the state of the dataset in the recipe. In this case, 

  D s product

   automatically switches you back to the most recently collected sample that is currently valid. Details are below.

...

• Locate the in-progress sampling job in the Samples panel. Click X.
• Click the Jobs Job History icon in the left nav bar. Select Sample jobs. For more information, see Sample Jobs Page.

...

• When a new sample is generated, any Sort transformations that have been applied previously must be re-applied. Depending on the type of output, sort order may not be preserved.

• Samples taken from a dataset with parameters are limited to a maximum of 50 files when executed on the

  D s photon

  running environment. You can modify parameters as they apply to sampling jobs. See Samples Panel.

Sample Invalidation

With each step that is added or modified to your recipe,

D s product

 checks to see if the current sample is valid. Samples are valid based on the state of your flow and recipe at the step when the sample was collected. If you add steps before the step where it was created, the currently active sample can be invalidated. For example, if you change the source of data, then the sample in the Transformer page no longer applies, and a new sample must be displayed.

...

For more information on sample types, see Sample Types.

D s also
inCQL true label ((label = "sample") OR (label = "sampling"))