...
- For more information on how to apply cluster clean, see Standardize Page.
- For more information on other methods of standardization, see Overview of Standardization.
Artifacts:
When a cluster clean step is added to your recipe, the number of individual changes can be many megabytes of data. Instead of storing these objects within the recipe definition, they are stored as a set of artifacts in the artifact storage database and referenced from the recipe.
- These artifacts exist outside the scope of the recipe file.
These artifacts must be stored in a
for the step to be editable and exportable.D s item item database Info NOTE: If the artifact storage service is disabled, this feature is unusable.
- When a flow is exported, an
artifact.datafile is included as part of the export. This file must be imported with the flow definition, or the cluster clean step in the imported flow is broken. For more information, see Export Flow.
Enable
This feature may require enabling in the product.
...
When you standardize using a spelling-based algorithm, the following values are clustered:
| Source Value | New Value | ||
| Apple | |||
| apple | |||
| Åpple | |||
| Unclustered values | |||
| pear | |||
| pair | |||
| pare |
After you select the cluster of values at top, you can enter apple, in the right context panel to replace that cluster of values with a single string.
In the above, the unclustered values are dissimilar in spelling, but in English, they sound the same (homonyms). When you select the Pronunciation-based algorithm, these values are clustered:
| Source Value | New Value | |
pear | ||
| pair | ||
| pare | ||
| Unclustered values | ||
| Apple | apple | |
| apple | apple | |
| Åpple | apple |
...
The six source values have been reduced to two final values through two different methods of clustering. See below for more information on the clustering algorithms.
| Source Value | New Value | ||
pear | pear | ||
| pair | pear | ||
| pare | pear | ||
| Apple | apple | ||
| apple | apple | ||
| Åpple | apple |
...
| Info |
|---|
NOTE: These steps are applied to an internal representation of the data. Your dataset and recipe are not changed by this comparison. Changes are only applied if you choose to modify the values and add the mapping. |
...
- Remove accents from characters, so that only ASCII characters remain.
- Change all characters to lowercase.
- Remove whitespace.
- Split the string on punctuation, any remaining whitespace, and control characters. Remaining characters are assembled into groups called tokens.
- Sort the tokens and remove any duplicates.
- Join the tokens back together.
- Compare all tokenized values in the column for purposes of clustering.
...
