This section contains additional configuration tasks for setting up high availability on the .

• The  can also integrate with a highly available Hadoop cluster. For more information, see Enable Integration with Cluster High Availability.

Limitations

For more information on deploying high availability, see Install for High Availability.

Log File Setup

Since the platform nodes are mounting to the same directory, a configuration flag forces log files to be put into a subdirectory containing the hostname of the node from which the file was generated. The resulting directory structure:

/opt/trifacta/
   --- conf
   --- bin
   --- logs/
       --- node1/webapp.log
       --- node1/data-service.log
       --- node1/spark-job-service.log
       --- node1/...
       --- node2/webapp.log
       --- node2/data-service.log
       --- node2/spark-job-service.log
       --- node2/...
       ---- ......

• Job logs are already prefixed with the job-group id, so there is no need to prefix them with the hostname. 
• The Download Logs should work independent of the node serving the endpoint, so there is no hostname prefixing required for the job logs.

To enable separation of log files into separate directories by node, please complete the following:

Steps:

1. Login as admin to the server (main) node in your HA environment.
2. 

3. Set the following parameter to true:

   "feature.highAvailability.separateLogFilesByName": false,

4. Save the file.
5. Before restarting the server, please complete any remaining configuration changes.

Disable restart from the Admin Settings Page

The preferred method of restarting the platform is to use the Restart button in the Admin Settings page. When a  is part of a high availability environment, the node cannot be restarted through the Admin Settings page, which only restarts the node. Other nodes in the environment are not restarted, and any configuration changes are not propagated. 

To disable use of the Restart button in the Admin Settings page, please complete the following steps.

Steps:

1. Login as admin to the server (main) node in your HA environment.
2. 

3. Set the following parameter to false:

   "feature.highAvailability.enableRestartFromAdminUI": true,

4. Save the file.
5. Before restarting the server, please complete any remaining configuration changes.

NFS Mount Setup

Each node shares the following resources:

• 

• Directories shared via NFS mount:

  /opt/trifacta/logs /opt/trifacta/conf

Example NFS Mount

The following sections contain an example configuration for the NFS mount to be shared across all nodes.

Server-side Configuration

NFS service

Use the following command to install the required NFS packages on the server:

yum install nfs-utils nfs-utils-lib

Enable and start NFS services:

systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap

systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

Shared directories

The following directories shared across all nodes:

/opt/trifacta/logs
/opt/trifacta/conf

1. To export the shared directories from the server, edit the following file:

   /etc/exports

2. Append the following lines to the end of the file:
   
   

   /opt/trifacta/conf 192.168.1.102(rw,sync,no_root_squash,no_all_squash) /opt/trifacta/logs 192.168.1.102(rw,sync,no_root_squash,no_all_squash)

   Property	Description
   /opt/trifacta/conf /opt/trifacta/logs	Directories to share across NFS
   192.168.1.102	IP address for client. NOTE: Insert the above commands once for each client IP address.
   rw	Read/write permissions on the shared folder.
   sync	Enables synchronizing shared directories.
   no_root_squash	Enables root privilege.
   no_all_squash	Enables user's authority.

3. Restart the nfs-server service:

systemctl restart nfs-server

Client-side Configuration

On each client node, install NFS packages:

yum install nfs-utils nfs-utils-lib

Enable and start the NFS services:

systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap

systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

Mount NFS Shared Directories on Client

Mount the shared directories from the server to the client as shown below:

mount -t nfs 192.168.1.101:/opt/trifacta/logs /opt/trifacta/logs
mount -t nfs 192.168.1.101:/opt/trifacta/conf /opt/trifacta/conf

If you encounter a Connection timed out error, please see the Troubleshooting section below.

Verifying NFS Shared Directories On Clients

Verify that the shared directories from the server are mounted:

mount | grep nfs

Example output:

rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
192.168.1.101:/opt/trifacta/logs on /opt/trifacta/logs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.9.167,local_lock=none,addr=10.0.6.77)

Auto-mount NFS Shared Directories

The above manual steps must be repeated whenever the platform is rebooted. 

To automatically mount the shared directories on reboot:

1. Edit the following file:

   /etc/fstab

2. Add the following lines:

   192.168.1.101:/opt/trifacta/logs /opt/trifacta/logs nfs rw,sync,hard,intr 0 0 192.168.1.101:/opt/trifacta/conf /opt/trifacta/conf nfs rw,sync,hard,intr 0 0

3. Reboot the client system and check the whether the NFS shared directories were mounted or not:

   mount | grep nfs

Integrate with HAProxy Load Balancer

After you have installed the software on each node in your high availability environment and configured your NFS mount, please complete the following instructions to integrate each node with a load balancer.

Configuration example:

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local1 notice

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    option                  log-health-checks
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           1s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main *:8088
#    acl url_static       path_beg       -i /static /images /javascript /stylesheets
#    acl url_static       path_end       -i .jpg .gif .png .css .js
     mode http
#    use_backend static          if url_static
    default_backend             app

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
#    balance     roundrobin
#    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
    balance     roundrobin
    option httpchk GET /v2/ping
    server app1 192.168.1.101:3005 check
    server app2 192.168.1.102:3005 check

Key Settings:

GET /v2/ping

Platform Start

To restart all nodes of the platform, please complete the following steps:

1. Apply any configuration changes to  on the server. Since the configuration directory is now shared across all nodes, these changes are automatically applied to all client nodes.

2. Verify that all nodes are pointing to the same instances of the .

3. Start the master node.

   NOTE: Nodes in an HA environment must be restarted at the command line. See Start and Stop the Platform.

4. Wait for it to complete restart. 

5. Start each client node one at a time.

   NOTE: Do not start multiple nodes at the same time, which can cause migration errors.

6. When all nodes are restarted, you may continue with any necessary configuration steps through the (main) server node.

Troubleshooting

When mounting the shared directories on the client, a Connection timeout error occurs

If you encounter the error:

mount.nfs: Connection timed out

This error may be due to the firewall blocking the NFS server. 

 Solution:

1. To access NFS shared directories from remote clients, the following ports in the NFS server iptables / firewall must be allowed:
   
   

   rpcinfo -p Sample output: program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 60985 status 100024 1 tcp 54302 status 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 2 udp 20048 mountd 100005 2 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100227 3 tcp 2049 nfs_acl 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100227 3 udp 2049 nfs_acl 100021 1 udp 46666 nlockmgr 100021 3 udp 46666 nlockmgr 100021 4 udp 46666 nlockmgr 100021 1 tcp 42955 nlockmgr 100021 3 tcp 42955 nlockmgr 100021 4 tcp 42955 nlockmgr 100011 1 udp 875 rquotad 100011 2 udp 875 rquotad 100011 1 tcp 875 rquotad 100011 2 tcp 875 rquotad

2. Run the following commands on the NFS server:

   firewall-cmd --permanent --add-port=111/tcp firewall-cmd --permanent --add-port=54302/tcp firewall-cmd --permanent --add-port=20048/tcp firewall-cmd --permanent --add-port=2049/tcp firewall-cmd --permanent --add-port=46666/tcp firewall-cmd --permanent --add-port=42955/tcp firewall-cmd --permanent --add-port=875/tcp

3. To apply the change, restart the firewall service:

   firewall-cmd --reload

4. Mount the shared directories on each client with the following commands:

   mount -t nfs 192.168.1.101:/opt/trifacta/logs /opt/trifacta/logs mount -t nfs 192.168.1.101:/opt/trifacta/conf /opt/trifacta/conf

5. The NFS shared directories should mount without error.