Private Data Processing

In the private data handling scenario, Alteryx Analytics Cloud requires clear boundaries of ownership. The shared responsibility matrix represents these boundaries of ownership.

At the highest level, Alteryx requires a set of permissions to run a private data plane. However, you will own the AWS account, the IAM credentials, and the IAM policy.

Alteryx provides a Cloud Formation template that defines the necessary permissions you can use to assist in completing this step.

At the next level down, Alteryx defines a specification for the VPC. This includes the definition of a number of subnets, CIDR blocks, route tables, and endpoints.

You must implement the VPC according to this spec. Alteryx provides Cloud Formation templates to assist in subnet and route table creation which you can use to assist in completing this step.

Once you’ve completed setup of the AWS account and VPC, sign in to the Alteryx Analytics Cloud to trigger the provisioning process that creates your private data processing cluster. The list of resources varies depending on which services you enable in the private data plane, but includes temporary storage, a Kubernetes cluster, compute nodes, secret management, and elastic spark processing.

The Alteryx Analytics Cloud will create and maintain these resources for you using automated provisioning pipelines in Terraform.

After provisioning the needed resources, Alteryxdeploys and maintains the software necessary to process your data within the private cluster. This includes a few long-running services and on-demand jobs.

Private Data Handling utilizes a number of AWS services inside the customer VPC to handle data processing tasks. These are the services used:

Private data handling is currently available in a variety of regions. In order to provide private data handling in a region, the region must:

In addition to these regional requirements, Alteryx also needs to replicate the container images to local repositories to improve design-time and runtime performance.

These are the regions where private data handling is available:

Alteryx Analytics Cloud runs a number of jobs and services inside the private data plane.

Kubernetes On-demand Jobs

For Kubernetes on-demand jobs, Alteryx Analytics Cloud retrieves a container image (from cache or from a central store) and deploys it within an ephemeral pod that lasts for the duration of the job. All executables are in Java or Python.

Kubernetes Long-running Services

VM Long-running Services

Provisioning a private data plane consists of 2 primary steps:

Private data planes use Infrastructure as Code (IaC). Alteryx Analytics Cloud uses Terraform Cloud to manage this. Terraform is an IaC tool that lets you define and manage infrastructure resources through human-readable configuration files. Terraform Cloud is a SaaS product provided by Hashicorp. To create and manage private data handling resources, Alteryx Analytics Cloud uses a set of terraform files, Terraform Cloud APIs, and private Terraform Cloud agents running on Alteryx infrastructure.

When you enable and provision private data handling through the Cloud Portal, Alteryx Analytics Cloud creates these resources in the AWS infrastructure:

You can find container images for on-demand jobs in a private container image repository that the private data plane has access to.

Alteryx deploys and maintains long-running services in your EKS cluster using Argo CD. Argo CD is a declarative, GitOps continuous delivery tool for kubernetes.

Data plane provisioning triggers from the Admin Console inside Alteryx Analytics Cloud. You need Admin privileges within a workspace in order to see it.

Make sure that Private Data Storage shows “Successfully Configured” before proceeding. If the status is “Not Configured,” go to Private Data Storage first, then return to this step.

Under the Private Data Processing section, there are 5 fields to fill out. These values come from completing the steps in Setup AWS Account and VPC.

Select Create to trigger the deployment of the cluster and resources in the AWS account. This runs a set of validation checks to verify the correct configuration of the AWS account. If there are incorrectly configured permissions, or the creation or tagging of the VPC resources is incorrect, you’ll receive an error message with a description that should point you in the right direction.

Once the initial validation checks complete, provisioning will commence. A message box on the screen periodically refreshes with status updates.

If your private data storage uses a cross-account role, then in order for your new private data plane to be able to read/write from your private data storage, you need to update that role to append a trust relationship with your new kubernetes cluster role as follows:

{
    "Sid": "",
    "Effect": "Allow",
    "Principal": {
         "AWS": "arn:aws:iam::<accountid>:role/aac-<xxxx-xxxxxxxxxxxx>-cluster-role"
    },
    "Action": "sts:AssumeRole"
}

For more information, go to https://docs.aws.amazon.com/directoryservice/latest/admin-guide/edit_trust.html.

Configure EMR serverless if you are using Spark/EMR processing.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EMRServerlessAccess",
            "Effect": "Allow",
            "Action": [
                "emr-serverless:CreateApplication",
                "emr-serverless:UpdateApplication",
                "emr-serverless:DeleteApplication",
                "emr-serverless:ListApplications",
                "emr-serverless:GetApplication",
                "emr-serverless:StartApplication",
                "emr-serverless:StopApplication",
                "emr-serverless:StartJobRun",
                "emr-serverless:CancelJobRun",
                "emr-serverless:ListJobRuns",
                "emr-serverless:GetJobRun"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowNetworkInterfaceCreationViaEMRServerless",
            "Effect": "Allow",
            "Action": "ec2:CreateNetworkInterface",
            "Resource": [
                "arn:aws:ec2:*:*:network-interface/*",
                "arn:aws:ec2:*:*:security-group/*",
                "arn:aws:ec2:*:*:subnet/*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:CalledViaLast": "ops.emr-serverless.amazonaws.com"
                }
            }
        },
        {
            "Sid":"AllowEMRServerlessServiceLinkedRoleCreation",
            "Effect":"Allow",
            "Action":"iam:CreateServiceLinkedRole",
            "Resource":"arn:aws:iam::<accountid>:role/aws-service-role/ops.emr-serverless.amazonaws.com/AWSServiceRoleForAmazonEMRServerless"
        },
        {
            "Sid": "AllowPassingRuntimeRole",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::<accountid>:role/aac-<xxxx-xxxxxxxxxxxx>-emr-serverless-spark-execution",
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": "emr-serverless.amazonaws.com"
                }
            }
        },
        {
            "Sid": "S3ResourceBucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::aac-<xxxx-xxxxxxxxxxxx>-emr-logs",
                "arn:aws:s3:::aac-<xxxx-xxxxxxxxxxxx>-emr-logs/*"
            ]
        }
    ]
}

{
    "Sid": "",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::<accountid>:role/aac-<xxxx-xxxxxxxxxxxx>-emr-serverless-spark-execution"
    },
    "Action": "sts:AssumeRole"
},
{
    "Sid": "",
    "Effect": "Allow",
    "Principal": {
        "Service": "emr-serverless.amazonaws.com"
    },
    "Action": "sts:AssumeRole"
}

When provisioning your private data plane, these are some common trouble spots that we see. These are organized based on when you’d run across them.

On the Private Data Handling page in Alteryx Analytics Cloud, you’ll see a note that provisioning failed, but without a good error message for what went wrong.

In the AWS admin console, you’ll see a NodeCreationFailure issue type with a description of Instances failed to join the kubernetes cluster. This typically indicates that there’s no route that allows the EKS subnets egress out to the internet. The new EKS nodes need access to an AWS EC2 Service Endpoint to attach to the EKS Cluster.

Your job will fail and you’ll see a red “x” on the wrangle step of the workflow (rather than a green check mark).