Contents:
Contents:
Base Storage Layer
The base storage layer is the datastore where the Dataprep by Trifacta Cloud uploads data, generates profiles, results, and samples. By default, job results are written on the base storage layer. You can configure the base storage layer and other required settings.
Tip: The base storage layer must be a file-based system.
Uses of base storage layer
In general, all base storage layers provide similar capabilities for storing, creating, reading, and writing datasets.
The base storage layer enables you to perform the following functions:
- Storing datasets: You can upload or store datasets in directories on the base storage layer. See below.
- Creating datasets: You can read in from datasources stored in the storage layer. A source may be a single file or a folder of files.
- Storage of samples: Any samples that you generate are stored in the base storage layer.
- Ingested data: Some data like Excel and PDF are stored as binary (non-text) files. These files must be read and converted to CSVs, which are stored on the base storage layer.
Cached data: You can enable a cache on the base storage layer, which allows data that has been ingested to remain on the base storage layer for a period of time. This cache allows for faster performance if you need to use the data at a later time.
- Writing Results: After you run the job, you can write the results to the storage layer.
Base storage layer directories
The following directories and their sub-directories are created and maintained on the base storage layer:
| Directory | Description |
|---|---|
/trifacta/uploads | Storage of datasets uploaded through the Dataprep by Trifacta application. Directories beneath this one are listed by the internal identifier for each user of the product who has uploaded at least one file. Avoid using |
/trifacta/queryResults | Default storage of results generated job executions. Directories beneath this one are listed by the internal identifier for each user of the product who has run at least one job. For each user, these sub-directories are the default storage location for job results. These locations can be modified. See Preferences Page. |
/trifacta/dictionaries | Storage of custom dictionary files uploaded by users. NOTE: This feature applies to Designer Cloud Enterprise Edition only. It is not often used. |
/trifacta/tempfiles | Temporary storage location for files required for use of the product. NOTE: The |
User-specific directories
The following directories are created by default on the base storage layer for the Dataprep by Trifacta Cloud to store user data.
By default, these directories are stored in the following:
<bucket_name>/<userId>
where:
<bucket_name>is the name of the bucket where user data is stored.userId>is the username that is used to log in to the product.
| Directory | Description |
|---|---|
jobrun | Storage of generated samples. |
temp | Temporary storage |
upload | Depending on your configuration, uploaded files may be stored in this per-user directory. |
These directories may be modified by individual users. For more information, see Storage Config Page.
Minimum Permissions
Dataprep by Trifacta requires the following operating system level permissions on the listed directories and sub-directories:
| Directory | Owner Min Permissions | Group Min Permissions | World Min Permissions |
|---|---|---|---|
/trifacta/uploads | read+write+execute | none | none |
/trifacta/queryResults | read+write+execute | none | none |
/trifacta/dictionaries | read+write+execute | none | none |
/trifacta/tempfiles | read+write+execute | none | none |
Available base storage layers
The Dataprep by Trifacta Cloud supports the following base storage layers.
NOTE: In some deployments, the base storage layer is pre-configured for you and cannot be modified. After the base storage layer has been defined, you cannot change it.
NOTE: For all storage layers, the source data is untouched. Results are written to a location whenever a job is executed on a source dataset.
Cloud Storage
Cloud Storage is a service for storing your objects in the Google Cloud Platform. You store objects in containers called buckets . All buckets are associated with a project. After you create a project, you can create cloud storage buckets, upload objects to your buckets, and download objects from your buckets. For more information, see Google Cloud Storage Access.
Encryption on base storage layer
For data that is transferred to and from the base storage layer:
- Data in transit is encrypted using HTTPS.
Data at rest is unencrypted by default.
NOTE: Optionally, you can deploy customer-managed encryption keys for encryption of data at rest. For more information, see Overview of CMEK.
Management of base storage layer
Maintenance of the base storage layer must be in accordance with your enterprise policies.
Unless the base storage layer is managed by Alteryx Inc, it is the responsibility of the customer to maintain access and perform any required backups of data stored in the base storage layer.
NOTE: Except for temporary files, the Dataprep by Trifacta Cloud does not perform any cleanup of the base storage layer.
External Storage
You can create connections to external storage systems. You can integrate the Dataprep by Trifacta Cloud with an external datastore. Depending on the type of connection and your permissions, the connection can be:
- read-only
- write-only
- read-write
You can create and edit connections between the Dataprep by Trifacta Cloud and external data stores. You can create either file-based or table-based connections to individual storage units, such as databases or buckets.
NOTE: In your environment, creation of connections may be limited to administrators only. For more information, contact your workspace administrator.
NOTE: After you create a connection, you cannot change its connection type. You must delete the connection and start again.
File-based systems
In addition to the base storage layer, you may be able to connect to other file-based systems. For example, if your base storage layer is HDFS, you can also connect to S3.
NOTE: If HDFS is specified as your base storage layer, you cannot publish to Redshift.
For more information, see Connection Types.
Cloud data warehouses
The Dataprep by Trifacta application can be leveraged for loading and transforming data in data warehouses in the cloud. These integrations offer high performance access to reading in datasets from these and other sources, performing transformations, and writing results back to the data warehouse as needed.
BigQuery
The native BigQuery integration supports reading of BigQuery data and writing to new or existing tables in BigQuery.
NOTE: Some permissions may need to be enabled in your user accounts. For more information, see Required Dataprep User Permissions.
For more information, see BigQuery Connections.
Tip: You can access data in BigQuery databases that are owned by other Google projects. For more information, see Access Cross-Project BigQuery Datasets.
Relational systems
When you are working with relational data, you can configure the database connections after you have completed the platform configuration and have validated that it is working for locally uploaded files.
NOTE: Database connections cannot be deleted if their databases host imported datasets that are in use by the Dataprep by Trifacta Cloud. Remove these imported datasets before deleting the connection.
For more information, see Using Databases.
Management of external storage
To integrate with an external system, the Dataprep by Trifacta application requires:
- Basic ability to connect to the hosting node of the external system through your network or cloud-based infrastructure
- Requisite permissions to support the browsing, reading and/or writing of data to the storage system
- A defined connection between the application and the storage system.
Except for cleanup of temporary files, the Dataprep by Trifacta application does not maintain external storage systems.
This page has no comments.