Page tree

 

Contents:


NOTE: High availability on the platform is in Beta release.

This section contains additional configuration tasks for setting up high availability on the Trifacta® node.

Limitations

The following limitations apply to this feature:

  • This form of high availability applies to  Trifacta Wrangler Enterprise in on-premises deployments. Deployments on AWS and Azure are not supported.
  • Job canceling does not work.
  • When HA is enabled, the restart feature in the Admin Settings page does not work. You must restart using the command line.
  • The platform must be installed on /opt/trifacta on every failover node.
  • This feature does not apply to the following components:
    • Hadoop cluster (See previous link.)
    • webhdfs/httpfs
    • Hive
    • Sentry
    • Navigator
    • Atlas
    • any other application/infrastructure with which the Trifacta platform can integrate

Log File Setup

Since the Trifacta nodes are mounting to the same directory, a configuration flag forces log files to be put into a subdirectory containing the hostname of the node from which the file was generated. The resulting directory structure:

/opt/trifacta/
   --- conf
   --- bin
   --- logs/
       --- node1/webapp.log
       --- node1/data-service.log
       --- node1/spark-job-service.log
       --- node1/...
       --- node2/webapp.log
       --- node2/data-service.log
       --- node2/spark-job-service.log
       --- node2/...
       ---- ......
  • Job logs are already prefixed with the job-group id, so there is no need to prefix them with the hostname. 
  • The Download Logs should work independent of the node serving the endpoint, so there is no hostname prefixing required for the job logs.

To enable separation of log files into separate directories by node, please complete the following:

Steps:

  1. Login as admin to the server (main) node in your HA environment.
  2. To apply this configuration change, login as an administrator to the Trifacta node. Then, edit trifacta-conf.json. Some of these settings may not be available through the Admin Settings Page. For more information, see Platform Configuration Methods.
  3. Set the following parameter to true:

    "feature.highAvailability.separateLogFilesByName": false,
  4. Save the file.
  5. Before restarting the server, please complete any remaining configuration changes.

Disable restart from the Admin Settings Page

The preferred method of restarting the platform is to use the Restart button in the Admin Settings page. When a Trifacta node is part of a high availability environment, the node cannot be restarted through the Admin Settings page, which only restarts the node. Other nodes in the environment are not restarted, and any configuration changes are not propagated. 

NOTE: In an HA environment, each node must be started, stopped, and restarted through the command line. See Start and Stop the Platform.

To disable use of the Restart button in the Admin Settings page, please complete the following steps.

Steps:

  1. Login as admin to the server (main) node in your HA environment.
  2. To apply this configuration change, login as an administrator to the Trifacta node. Then, edit trifacta-conf.json. Some of these settings may not be available through the Admin Settings Page. For more information, see Platform Configuration Methods.
  3. Set the following parameter to false:

    "feature.highAvailability.enableRestartFromAdminUI": true,
  4. Save the file.
  5. Before restarting the server, please complete any remaining configuration changes.

NFS Mount Setup

Each node shares the following resources:

  • Trifacta databases
  • Directories shared via NFS mount:

    /opt/trifacta/logs
    /opt/trifacta/conf

Example NFS Mount

The following sections contain an example configuration for the NFS mount to be shared across all nodes.

NOTE: There are many ways to configure an NFS mount. The following example applies to CentOS 7. Please modify these commands for different operating systems and versions as needed.

Server-side Configuration

NFS service

Use the following command to install the required NFS packages on the server:

yum install nfs-utils nfs-utils-lib

Enable and start NFS services:

systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap

systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

Shared directories

The following directories shared across all nodes:

/opt/trifacta/logs
/opt/trifacta/conf
  1. To export the shared directories from the server, edit the following file:

    /etc/exports
  2. Append the following lines to the end of the file:

    /opt/trifacta/conf 192.168.1.102(rw,sync,no_root_squash,no_all_squash)
    /opt/trifacta/logs 192.168.1.102(rw,sync,no_root_squash,no_all_squash)
    PropertyDescription
    /opt/trifacta/conf
    /opt/trifacta/logs
    Directories to share across NFS 
     192.168.1.102

    IP address for client.

    NOTE: Insert the above commands once for each client IP address.

     rwRead/write permissions on the shared folder. 
    syncEnables synchronizing shared directories.
    no_root_squashEnables root privilege.
    no_all_squashEnables user's authority.
  3. Restart the nfs-server service:
systemctl restart nfs-server

Client-side Configuration

On each client node, install NFS packages:

yum install nfs-utils nfs-utils-lib

Enable and start the NFS services:

systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap

systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

Mount NFS Shared Directories on Client

Mount the shared directories from the server to the client as shown below:

mount -t nfs 192.168.1.101:/opt/trifacta/logs /opt/trifacta/logs
mount -t nfs 192.168.1.101:/opt/trifacta/conf /opt/trifacta/conf

NOTE: Repeat the above on each client node.

If you encounter a Connection timed out error, please see the Troubleshooting section below.

Verifying NFS Shared Directories On Clients

Verify that the shared directories from the server are mounted:

mount | grep nfs

Example output:

rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
192.168.1.101:/opt/trifacta/logs on /opt/trifacta/logs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.9.167,local_lock=none,addr=10.0.6.77)

Auto-mount NFS Shared Directories

The above manual steps must be repeated whenever the platform is rebooted. 

To automatically mount the shared directories on reboot:

  1. Edit the following file:

    /etc/fstab
  2. Add the following lines:

    192.168.1.101:/opt/trifacta/logs /opt/trifacta/logs nfs rw,sync,hard,intr 0 0
    192.168.1.101:/opt/trifacta/conf /opt/trifacta/conf nfs rw,sync,hard,intr 0 0
  3. Reboot the client system and check the whether the NFS shared directories were mounted or not:

    mount | grep nfs

Integrate with HAProxy Load Balancer

After you have installed the software on each node in your high availability environment and configured your NFS mount, please complete the following instructions to integrate each node with a load balancer.

NOTE: HAProxy is the recommended load balancer. These instructions apply to HAProxy. If you are using a different load balancer, please consult the documentation that came with your product.

Configuration example:

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local1 notice

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    option                  log-health-checks
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           1s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main *:8088
#    acl url_static       path_beg       -i /static /images /javascript /stylesheets
#    acl url_static       path_end       -i .jpg .gif .png .css .js
     mode http
#    use_backend static          if url_static
    default_backend             app

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
#    balance     roundrobin
#    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
    balance     roundrobin
    option httpchk GET /v2/ping
    server app1 192.168.1.101:3005 check
    server app2 192.168.1.102:3005 check

Key Settings:

SettingDescription
balanceSet this value to the method of load balancing. Round robin is recommended.
option httpchk

Please set the value to the following:

GET /v2/ping
server app1Set this value to the IP address of the server node.

server app2

server app3

Please insert separate entries for the IP addresses of each client node.

Platform Start

To restart all nodes of the platform, please complete the following steps:

  1. Apply any configuration changes to  trifacta-conf.json on the server. Since the configuration directory is now shared across all nodes, these changes are automatically applied to all client nodes.

  2. Verify that all nodes are pointing to the same instances of the Trifacta databases.

  3. Start the master node.

    NOTE: Nodes in an HA environment must be restarted at the command line. See Start and Stop the Platform.

  4. Wait for it to complete restart. 

  5. Start each client node one at a time.

    NOTE: Do not start multiple nodes at the same time, which can cause migration errors.

  6. When all nodes are restarted, you may continue with any necessary configuration steps through the (main) server node.

Troubleshooting

When mounting the shared directories on the client, a Connection timeout error occurs

If you encounter the error:

mount.nfs: Connection timed out

This error may be due to the firewall blocking the NFS server. 

 Solution:

  1. To access NFS shared directories from remote clients, the following ports in the NFS server iptables / firewall must be allowed:

    rpcinfo -p
    Sample output:
    
        program vers proto   port  service
        100000    4   tcp    111  portmapper
        100000    3   tcp    111  portmapper
        100000    2   tcp    111  portmapper
        100000    4   udp    111  portmapper
        100000    3   udp    111  portmapper
        100000    2   udp    111  portmapper
        100024    1   udp  60985  status
        100024    1   tcp  54302  status
        100005    1   udp  20048  mountd
        100005    1   tcp  20048  mountd
        100005    2   udp  20048  mountd
        100005    2   tcp  20048  mountd
        100005    3   udp  20048  mountd
        100005    3   tcp  20048  mountd
        100003    3   tcp   2049  nfs
        100003    4   tcp   2049  nfs
        100227    3   tcp   2049  nfs_acl
        100003    3   udp   2049  nfs
        100003    4   udp   2049  nfs
        100227    3   udp   2049  nfs_acl
        100021    1   udp  46666  nlockmgr
        100021    3   udp  46666  nlockmgr
        100021    4   udp  46666  nlockmgr
        100021    1   tcp  42955  nlockmgr
        100021    3   tcp  42955  nlockmgr
        100021    4   tcp  42955  nlockmgr
        100011    1   udp    875  rquotad
        100011    2   udp    875  rquotad
        100011    1   tcp    875  rquotad
        100011    2   tcp    875  rquotad
  2. Run the following commands on the NFS server:

    firewall-cmd --permanent --add-port=111/tcp
    firewall-cmd --permanent --add-port=54302/tcp
    firewall-cmd --permanent --add-port=20048/tcp
    firewall-cmd --permanent --add-port=2049/tcp
    firewall-cmd --permanent --add-port=46666/tcp
    firewall-cmd --permanent --add-port=42955/tcp
    firewall-cmd --permanent --add-port=875/tcp
  3. To apply the change, restart the firewall service:

    firewall-cmd --reload
  4. Mount the shared directories on each client with the following commands:

    mount -t nfs 192.168.1.101:/opt/trifacta/logs /opt/trifacta/logs
    mount -t nfs 192.168.1.101:/opt/trifacta/conf /opt/trifacta/conf
  5. The NFS shared directories should mount without error.

This page has no comments.