Release Notes for Dataprep by Trifacta

Contents:

These release notes apply to the following product tiers of Dataprep by Trifacta®:

• Dataprep by Trifacta Enterprise Edition
• Dataprep by Trifacta Professional Edition
• Dataprep by Trifacta Starter Edition
• Dataprep by Trifacta Premium
• Dataprep by Trifacta Standard
• Dataprep by Trifacta Legacy

For more information, see Product Editions.

For release notes from previous releases, see Earlier Releases of Dataprep by Trifacta.

May 13, 2002

Release 9.2 - push 2

Changes in System Behavior

Intelligent caching:

Due to technical issues, the intelligent caching of recipe steps feature for performance improvements has been disabled.

When the technical issues are addressed, this feature will be enabled.

April 20, 2022

Release 9.2

What's New

Lock/unlock column data type:

You can now lock or unlock a column's data type. When the data type is locked, the Trifacta application no longer attempts to infer the column's data type when subsequent recipe steps are applied. 

For more information, see Change Column Data Type.

Connectivity:

Early Preview (read-only) connections available with this release:

• Marketo

  NOTE: Marketo connections require an OAuth 2.0 client to be created in the Trifacta application. For more information, see OAuth 2.0 for Marketo.

• For more information, see Early Preview Connection Types.

Connectivity:

• Google Analytics are now generally available and supported on Dataprep by Trifacta.
  • For more information on creating the connection object, see Google Analytics Connections.

Publish Array data type as arrays to BigQuery:

You can now publish  Dataprep by Trifacta®  Array data type as BigQuery arrays. 

• For more information, see Publishing Actions.
• For more information, see BigQuery Data Type Conversions.

Parameterize data in hidden folders:

Optionally, you can scan hidden folders for wildcard- or pattern-based matches when building your parameterized imported datasets. 

For more information on including hidden files, see Dataprep Project Settings Page.

For more information on creating datasets with parameters from files, see Parameterize Files for Import.

Simplified permissions for publishing to BigQuery:

By default,  Dataprep by Trifacta requires that the bigquery.datasets.create permission is enabled for each user of the product to run Dataflow jobs on BigQuery data sources. In some environments, this permission cannot be provided to users, and these Dataflow jobs fail.

As a workaround, you can provide to  Dataprep by Trifacta a pre-existing BigQuery dataset, in which intermediate query results can be stored. When this dataset is provided to the Trifacta application, temporary tables are created within it as part of Dataflow job execution, and the bigquery.datasets.create permission is not required. 

For more information on configuring the BigQuery temp dataset for the Trifacta application, see Dataprep Project Settings Page.

Documentation:

Published documented solution for integrating  Dataprep by Trifacta with your Virtual Private Cloud Service Controls (VPC SC). For more information on this integration, see Configure VPC-SC Perimeter.

Changes in System Behavior

Set column data type transformation locks the column's type by default:

Starting in this release, the column data type is locked by default when you change the column data type.

If required, you can unlock the column's data type. For more information, see Change Column Data Type.

Connectivity:

• The Google Analytics connection type now supports the UniversalAnalytics schema.

  NOTE: Previously, this schema was called GoogleAnalytics by the driver vendor. You may need to update your custom SQL queries to reference this new schema name.

  • For more information, see Google Analytics Connections.

Generate an initial sample:

When generating an initial sample from a set of files in a directory, the maximum number of files that can be read is now limited to 50.

• Previously, the Trifacta application read files until either 10MB of data or all matching files had been scanned.
• This change is to limit the number of files that must be read for various operations in the Transformer page. It only applies to generating the initial sample type. Other sampling types, such as random sample, can scan the full set of files.

As needed, an administrator can change this maximum limit.

• For more information, see Dataprep Project Settings Page.
• For more information on sampling, see Overview of Sampling.

Performance:

The intelligent caching of recipe steps feature for performance improvements has been made available again. The issues that required removing it from the platform have been addressed.

This feature can be enabled by an administrator.

For more information, see Dataprep Project Settings Page.

Email notifications:

In a future release, the setting for email notifications based on job success will default to Default (Any Jobs) at the project or workspace level and at the flow level. This change means that the user who executes a job and others who have access to the flow receive, by default, an email notification whenever a job executes for flows where email notification settings have never been modified. As part of this change, each email will contain a richer set of information about the job that was executed. 

If needed, this new default setting can be modified:

• Project owners and administrators change default value of the email notification settings. For more information, see Dataprep Project Settings Page.
• Individual users can override these settings for individual flows. For more information, see Manage Flow Notifications Dialog.

Deprecated

None.

Key Bug Fixes

New Known Issues

March 15, 2022

Release 9.1

What's New

Encryption:

• Support for use of customer-managed encryption keys (CMEK) during Dataflow job execution. Trifacta application can also check for use of CMEKs before writing results to BigQuery or  Cloud Storage.

  Private Preview: This feature is disabled by default. For more information on enabling this feature in your project, please contact Trifacta Support.

  Feature Availability: This feature is available in

  Dataprep by Trifacta Enterprise Edition only.

  • For more information, see Overview of CMEK.

  • For more information on enabling, see Dataprep Project Settings Page.

JavaScript User Defined Functions:

• Create user-defined functions (UDFs) in JavaScript and upload them to your project for use in your recipe steps. JavaScript UDFs enable users to create customized and consistent functions to meet their specific requirements.
  
  
  NOTE: This feature is in Beta release.
  
  
  Feature Availability: This feature is not available in

  Dataprep by Trifacta Legacy only.

• • This feature is enabled by default. For more information, see Dataprep Project Settings Page.
  • For more information, see JavaScript UDFs.
  • When enabled, JavaScript UDFs are defined through the Library page. For more information, see User Defined Functions Page.

Connectivity:

• Enable connectivity between the Trifacta application and your cloud databases using SSH tunneling is generally available with this release.

  Tip: This feature is now generally available.

  NOTE: For this release, SSH tunneling can be enabled on the following connection types: Oracle Database, PostgreSQL, MySQL, and Microsoft SQL Server.

  For more information, see Configure SSH Tunnel Connectivity.

Connectivity:

Early Preview (read-only) connections available with this release:

• Instagram Ads
• For more information, see Early Preview Connection Types.

Job execution:

The Trifacta application can check for changes to your dataset's schemas before jobs are executed and optionally halt job execution to prevent data corruption.

• These options can be configured by a project administrator.
  
  Feature Availability: This feature is not available in

  Dataprep by Trifacta Legacy only.
  
  For more information, see Dataprep Project Settings Page.

• For more information, see Overview of Schema Management.

Dataset configuration:

For an imported dataset, you can configure settings through a new interface, including column names and column data types to use in the Trifacta application.

• For more information, see Dataprep Project Settings Page.

• For more information on how to use the feature, see Dataset Configuration Settings.

Sample Job IDs:

When a sample is collected, a job ID is generated and displayed in the Trifacta application.   These job IDs enable you to identify the sample jobs.

• For more information, see Generate a Sample.
• For more information, see Samples Panel.
• For more information, see Sample Jobs Page.

Import:

For long-loading Parquet datasets, you can monitor the ingest process as you continue your work.

For more information, see Flow View Page.

Changes in System Behavior

Publishing:

Beginning in this release, you can publish  Dataprep by Trifacta Array type columns to BigQuery as BigQuery arrays for Trifacta primitive data types. Arrays containing non-primitive data types continue to be published as String values.

• For more information, see Improvements to the Type System.
• This change can be reverted to previous String publishing behavior on individual outputs. See BigQuery Table Settings.

Performance:

A recent release introduced improved performance through intelligent caching of recipe steps.

• This feature was released as a Beta feature.

• Due to some recently discovered issues, this feature has been disabled for the time being. It cannot be enabled by a workspace administrator at this time.

  NOTE: If this Beta feature had been enabled in your environment, you may experience a reduction in performance when moving between recipe steps in the Transformer page.

• The feature will be re-enabled in a future release.

Deprecated

None.

Key Bug Fixes

New Known Issues

None.

February 9, 2022

Release 9.0

What's New

JavaScript User Defined Functions:

Create user-defined functions (UDFs) in JavaScript and upload them to your project for use in your recipe steps. JavaScript UDFs enable users to create customized and consistent functions to meet their specific requirements.

For more information, see JavaScript UDFs.

When enabled, JavaScript UDFs are defined through the Library page. For more information, see User Defined Functions Page.

Connectivity:

Build connections to accessible REST API endpoints.

For more information, see REST API Connections.

Connectivity:

Early Preview (read-only) connections available with this release:

• LinkedIn Ads
• Zendesk
• For more information, see Early Preview Connection Types.

Dataset Schema Refresh:

You can now refresh your imported datasets with the current schema information from the source file or table. Schema refresh enables you to capture any changes to the columns in your dataset.

• For more information, see Overview of Schema Management.
• Dataset schemas can be refreshed through the following pages:
  • See Library Page.
  • See Dataset Details Page.
  • In Flow View. For more information, see View for Imported Datasets.

Changes in System Behavior

None.

Deprecated

None.

Key Bug Fixes

New Known Issues

None.

January 27, 2022

Release 8.11 - push 2

What's New

None.

Changes in System Behavior

None.

Deprecated

None.

Key Bug Fixes

New Known Issues

None.

January 20, 2022

Release 8.11

What's New

BigQuery Running Environment:

Beginning in this release, sampling jobs can be executed in BigQuery.

• For more information, see Flow Optimization Settings Dialog.
• For more information, see BigQuery Running Environment.

Connectivity:

Early Preview (read-only) connections available with this release:

• Asana
• Exact Online
• Facebook Ads
• Jira by Atlassian
• Pinterest
• QuickBooks Online
• Trino
• For more information, see Early Preview Connection Types.

Session Management:

You can view the current and recent sessions of the Trifacta application. You can review the devices that are authorized and revoke any unfamiliar devices.

• For more information, see Preferences Page.
• For more information, see Sessions Page.

Performance:

• Improved performance during design time through intelligent caching of recipe steps. 
  
  NOTE: This feature is in Beta release.
  
  
  • A workspace administrator may need to enable this feature in your project. See Dataprep Project Settings Page.

• Improvements in job execution performance, due to skipping some output validation steps for file-based outputs.

  NOTE: When scheduled or API jobs are executed, no validations are performed of any writesettings objects. Issues with these objects may cause failures during transformation or publishing stages of job execution.

  • A workspace administrator may need to enable this feature in your project. See Dataprep Project Settings Page.

Changes in System Behavior

Sample sizes can be increased up to 40MB

Prior to this release, the size of a sample was capped at 10MB. This size represented:

• the actual size of the sample object stored in the base storage layer
• the default maximum size of the sample displayed in the Trifacta application. This sample size can be reduced from 10MB, if needed.

Beginning in this release:

• The actual size of the stored sample has increased to 40MB.

  NOTE: On backend storage, sample sizes are now four times larger than in previous releases. For datasources that require decompression or conversion, actual storage sizes may exceed this 40 MB limit.

• The size of the sample displayed for a recipe can be configured to be up to 40MB in size by individual users.

For more information, see Change Recipe Sample Size.

Data type mismatches can now be written out in CSV format

Beginning in this release, for CSV outputs mismatched values are written as regular values by default. In prior releases, mismatched values were written as null values in CSV outputs.

See Improvements to the Type System.

Deprecated

None.

Key Bug Fixes

None.

New Known Issues

December 7, 2021

Release 8.10

What's New

User management:

Introducing user and role management. In the Admin console in the Trifacta application, you can enable and disable user access and determine access levels to individual object types, such as flows, connections, and plans. 

• For more information, see Changes to User Management.
• For more information on role-based access, see Overview of Authorization.

Connectivity:

• Enable connectivity between the Trifacta application and your cloud databases using SSH tunneling.

  NOTE: In this release, this feature must be enabled by request. For more information, please contact Trifacta Support.

  
  

  NOTE: SSH tunneling is enabled on a per-connection basis. For this release, SSH tunneling can be enabled on the following connection types: Oracle Database, PostgreSQL , MySQL , and Microsoft SQL Server .

  For more information, see Configure SSH Tunnel Connectivity.

• Early Preview (read-only) connections available with this release:
  
  Feature Availability: This feature is available in the following editions:

  
  
  • Dataprep by Trifacta Enterprise Edition
  • Dataprep by Trifacta Professional Edition
  • Dataprep by Trifacta Premium

  
  

• • SurveyMonkey
  • Xero
  • For more information, see Early Preview Connection Types

Session Management:

You can view the current and recent sessions for your account in the Trifacta application. As needed, you can revoke any unfamiliar devices or sessions. For more information, see Sessions Page.

Changes in System Behavior

Ingestion:

Maximum permitted record length has been increased from 1 MB to 20 MB. For more information, see Working with JSON v2.

Split transformation:

When splitting a column based on positions, the positions no longer need to be listed in numeric order. See Changes to the Language.

Deprecated

None.

Key Bug Fixes

None.

New Known Issues

/[|]/

November 23, 2021

Release 8.9

What's New

Refer and Earn:

Beginning in this release, for every new sign-up you refer, you get a reward of your choice. For more information, see Referrals Page.

Self-serve upgrades from your free trial: 

Through the trial expiration page, you can review and select the preferred plan that suits you. Provide the required card details through the application and subscribe to your preferred plan.

For more information, see Start a Subscription.

BigQuery Running Environment:

Beginning in this release, imported datasets created with customer SQL are supported for execution in the BigQuery running environment. For more information, see BigQuery Running Environment.

Connectivity:

Early Preview (read-only) connections available with this release:

• Presto
• Microsoft Advertising
• For more information, see Early Preview Connection Types.

Plans:

• Create plan tasks to delete files and folders from file-based backend storage.

  Feature Availability: This feature is available in the following editions:

  
  
  • Dataprep by Trifacta® Enterprise Edition
  • Dataprep by Trifacta Professional Edition
  • Dataprep by Trifacta Premium

  
  

  For more information, see Create Delete Task.

• You can now reference output metadata from within your plans. See Plan Metadata References.

Collaboration:

You can view the list of collaborators and their corresponding avatars on shareable objects, such as Flows, Plans, and Connections pages.

• For more information, see Flows Page.
• For more information, see Connections Page.
• For more information, see Plans Page.

Sampling:

• Adjust the size of samples loaded in the browser for your current recipe to improve performance and address low-memory conditions. See  Change Recipe Sample Size.

Changes in System Behavior

None.

Deprecated

None.

Key Bug Fixes

New Known Issues

None.

October 12, 2021

Release 8.8

What's New

Project Usage:

• VCU usage and active users are now displayed in the Trifacta application for administrators. For more information, see Usage Page.

Trifacta Photon:

• You can now configure the Trifacta application to execute Trifacta Photon jobs in your VPC.
  
  NOTE: This feature is in Beta release.
  
  
  Feature Availability: This feature is available in

  Dataprep by Trifacta Enterprise Edition only.
  
  For more information, please contact  Trifacta Support.

Changes

Cancellation of jobs is temporarily disabled:

In previous releases, you could cancel in-progress jobs for flow and sampling jobs through the Trifacta application. As of this release, canceling of job types, such as sampling, transformation, and profiling jobs, is temporarily disabled.

Job cancellation may be re-enabled in the future.

Billing:

Charges for your project and user usage of  Dataprep by Trifacta are applied to your account based on the UTC (Greenwich) time zone. However, Google Marketplace tracks and reports usage based on the Pacific (U.S. West Coast) time zone, so some discrepancies in reporting have been observed.

Beginning at the end of October 2021, these discrepancies will be addressed. The daily reporting interval will be changed to start and end at midnight Pacific time to match how Google Marketplace reports. However, the usage tracking will remain based on the UTC time zone.

vCPU usage has been tracked on an hourly basis and will be unchanged.

For more information, see Usage Page.

Import:

Improvements have been made in how double quotes are handled in CSV files during import to align  Dataprep by Trifacta with other systems that support CSV import. 

• Example values in source CSV file:

  """My product""",In stock,"16,000",0.05

  Note that the value 16,000 must be double-quoted, since the value contains a comma, which is the field delimiter.

• Previously, this value appeared in the Transformer page in columns as the following:

  c1	c2	c3	c4
  """My product"""	In stock	"16,000"	0.05

• As of this version, the Trifacta application handles the values in a better manner when displaying them in the Transformer page:

  c1	c2	c3	c4
  "My product"	In stock	16,000	0.05

  • c1: Escaped values (tripe double-quotes) in the source no longer render in the application as triple double-quotes and are represented as quoted values.

  • c3: Note that the double quotes in c3 have been stripped. Leading and trailing quotes are trimmed if the quotes are balanced within a cell.

    NOTE: This change in behavior applies only to newly created imported datasets sourced from a CSV file. Existing imported datasets should not be affected. However, if a newly imported dataset is transformed by a previously existing recipe that compensated for the extra quotes in the Transformer page, the effects on output data could be unpredictable. These recipes and their steps should be reviewed.

    This change does apply to any newly imported dataset sourced from CSV and may cause the data to change. For example, if you export an older flow and import into a new workspace or project, this change in parsing behavior applies to the datasets that are newly created in the new environment. Recipes may require review upon import.

• When results are generated in CSV, output files should continue to reflect the formatting of the source data before import. See above.

  Tip: You can also choose the Include quotes option when creating a CSV output.

  • When profiling is enabled, values that appear in CSV as "" are now marked as missing.

API:

• To prevent overloading mission-critical API endpoints, rate limiting on a select set of API endpoints has been implemented in the platform. For more information, see Changes to the APIs.

BigQuery Running Environment:

When running jobs in BigQuery, some additional data types, functions, and transformations are now supported:

• Data types: The following data types are now supported for execution in BigQuery:

  • Arrays
  • Objects (Maps)
• Aggregation functions: 
  • LIST
  • LISTIF
  • UNIQUE
  • See Aggregate Functions.
• Date functions:
  • WEEKNUM
  • CONVERTFROMUTC
  • CONVERTTOUTC
  • CONVERTTIMEZONE
    
  • DATEDIF: All unit types are now supported.
  • See Date Functions.

• String functions:

  • SUBSTITUTE
  • PROPER
  • REMOVESYMBOLS
  • DOUBLEMETAPHONE
    
  • See String Functions.
    

• Nested functions:

  • ARRAYCONCAT
  • ARRAYCROSS
  • ARRAYINTERSECT
  • ARRAYLEN
  • ARRAYSTOMAP
  • ARRAYUNIQUE
  • ARRAYZIP
  • FILTEROBJECT
  • KEYS
  • ARRAYELEMENTAT
  • LISTAVERAGE
  • LISTMAX
  • LISTMIN
  • LISTMODE
  • LISTSTDEV
  • LISTSUM
  • LISTVAR
  • ARRAYSORT
  • ARRAYINDEXOF
  • ARRAYMERGEELEMENTS
  • ARRAYRIGHTINDEXOF
  • ARRAYSLICE
  • See Nested Functions.
• Other functions:
  • IPTOINT
  • IPFROMINT
  • See Other Functions.

• Transformations:
  

  Search term	Transform
  Unnest elements	unnest
  Expand Array to rows	flatten
  Extract between delimiters	extractbetweendelimiters
  Unpivot	unpivot
  Standardize column	standardize
  Nest columns	nest
  Extract matches to Array	extractlist
  Replace between delimiters	replacebetweenpatterns
  Scale to min max	scaleminmax
  Scale to mean	scalestandardize
  Convert key/value to Object	extractkv
  Join	Join datasets For more information, see Join Types.

  • Legend:

    • Search term: the value you enter in the Transform Builder
    • Transform: name of the underlying transform

    • For more information, see Transformation Reference.

BigQuery Running Environment

• Support for reading Google Cloud Storage files for execution in BigQuery.
  
  Feature Availability: This feature is available in the following editions:

  
  
  • Dataprep by Trifacta Enterprise Edition
  • Dataprep by Trifacta Professional Edition
  • Dataprep by Trifacta Premium

  
  
  
  
  • For more information, see Flow Optimization Settings Dialog.
  • For more information, see Google Cloud Storage Access.

Deprecated

None.

Known Issues

None.

Fixes

September 15, 2021

Release 8.7

What's New

Templates:

From the Flows page, you can now access pre-configured templates directly from the templates gallery.

• For more information, see Flows Page.
• For more information on using a template in the product, see Start with a Template.

Browsers:

• Update to supported browsers:
  • Mozilla Firefox is generally supported.
  • Microsoft Edge is now supported.
    
    
    NOTE: This feature is in Beta release.
  • New versions of supported browsers are now supported.
  • For more information, see Browser Requirements.

Plans:

• Create plan tasks to deliver messages to a specified Slack channel.

  Feature Availability: This feature is available in the following editions:

  
  
  • Dataprep by Trifacta® Enterprise Edition
  • Dataprep by Trifacta Professional Edition
  • Dataprep by Trifacta Premium

  
  

  For more information, see Create Slack Task.

Import data:

• When you are importing from or writing to  Cloud Storage, you can choose to display hidden files and folders for access to them.

  Tip: Use this option to access files generated for your job's visual profile and then publish them to BigQuery for additional analysis.

  For more information, see Import Data Page.

Sharing:

• Paste in a comma-separated list of email addresses to share flows, plans, or connections with multiple users at the same time.
  • See Share Flow Dialog.
  • See Share Connection Dialog.

Publishing:

• Strict type matching for publishing to BigQuery Datetime columns. 

  Tip: You can enable or disable strict type matching during publication to BigQuery. Strict type matching is enabled by default for new flows. You can disable the flag to revert to previous BigQuery publishing behaviors. See BigQuery Table Settings.

  For more information, see BigQuery Data Type Conversions.

Recipe panel:

• Some enhancements to flagging steps for review. See Flag for Review.

Changes

None.

Deprecated

API:

• Deprecated API endpoint to transfer assets between users has been removed from the platform. This endpoint was previously replaced by an improved method of transfer.
• Some connection-related endpoints have been deprecated. These endpoints have little value for public use.
• For more information, see Changes to the APIs.

Known Issues

if(ismissing($col),NULL(),$col)

Fixes

August 16, 2021

Release 8.6

What's New

Template Gallery:

Collaboration:

• Flow editors and plan collaborators can be permitted to schedule jobs. See Dataprep Project Settings Page.

Connectivity:

• Upload tabular data from PDF documents. See Import PDF Data.

• Early Preview (read-only) connections available with this release:
  

  Feature Availability: This feature is available in the following editions:

  
  
  • Dataprep by Trifacta Enterprise Edition
  • Dataprep by Trifacta Professional Edition
  • Dataprep by Trifacta Premium

  
  

• • Google Ads
  • NetSuite
  • For more information, see Early Preview Connection Types.

Performance:

• Conversion jobs are now processed asynchronously. 

• Better management of file locking and concurrency during job execution. 

Better Handling of JSON files:

The Trifacta application now supports the regularly formatted JSON files during import. You can now import flat JSON records contained in a single array object. With this, each array is treated as a single line and imported as a new row. For more information, see Working with JSON v2

Usage reporting:

Detailed reporting on vCPU and active users is now available in the Trifacta application.

For more information, see Usage Page.

Changes

Dataflow machines:

• The following machine types are now available when running a Dataflow job:

  "e2-standard-2",
  "e2-standard-4",
  "e2-standard-8",
  "e2-standard-16",
  "e2-standard-32"

Deprecated

None.

Known Issues

• TD-63564: Schedules created by a flow collaborator with editor access stop working if the collaborator is removed from the flow.

  • Tip: Flow owners can delete the schedule and create a new one. When this issue is fixed, the original schedule will continue to be executed under the flow owner's account.

  • Collaborators with viewer access cannot create schedules.
    

Fixes

• TD-61478: Time-based data types are imported as String type from BigQuery sources when type inference is disabled.

Earlier Releases

For release notes from previous releases, see Earlier Releases of Dataprep by Trifacta.