CliQr is now part of Cisco Learn More About Cisco

High Availability Best Practices

Overview

A typical High Availability (HA) network is depicted in HA Installer Architecture.

This section identifies the typical requirements for each component.

A basic deployment with 2 cloud regions, typically requires the following VMs:

Required Components    Non-HANon-HA Standalone
HANotes
CCM1 VM1 VM2 VMsAccess to the CCM UI
MGMTPOSTGRES-1 VM2 VMsCCM Database
CCM LB--1 VMLB in HA mode
CCO2 VMs2 VMs6 VMs3 VMs per cloud region for HA mode
CCO LB--2 VMsLB in HA mode – 1 VM per cloud region
AMQP2 VMs2 VMs4 VMsPer cloud region
AMQP LB--2 VM  LB in HA mode
Monitor1 VM1 VM1 VMRequired for logs
Total6 VMs7 VMs20 VMsTypical BASIC deployments

General Requirements for All Components

CloudCenter components installed in HA mode must adhere to the following requirements:

  • To avoid a single point of failure, set up each component's highly available VM in a separate zone/region.

  • Provide a localhost and ensure that the hostname of the host is resolvable in the /etc/hosts file:
    • Localhost: In the following example, add a localhost for 127.0.0.1 and the IP address for the component(s) – hapgrs1 and hapgrs2in the /etc/hosts file.

    • Hostname:

      • Cannot contain any special characters, including underscore, hyphen, or space for the database servers as it causes the authentication to fail.

      • Changing the hostname after you install and configure the VMs may cause unknown issues.

      • The hostname cannot be on the same line as the localhost – add the hostname on a separate line in the /etc/hosts file.

Load Balancer Requirements

As a best practice, we recommend that you configure one load balancer for each component in HA mode to provide true HA support:

  • CCM VMs
  • CCO VMs
  • AMQP VMs

 When configuring the load balancer for each component as follows:

  • Use TCP protocol for listeners – for example, you can configure a generic load balancer application to use TCP and ensure that the certificate exchange procedure is transparent.
  • Use SSL certificates for communication between the CCM, CCO, and AMQP load balancers.

  • Enable the session stickiness policy. Refer to your load balancer documentation for additional details.
  • Use the IP address of the load balancer when configuring the CCO IP field from the CCM UI.
  • If you are upgrading a component from non-HA mode to HA mode, you may sometimes add new VMs to an existing deployment. In this case, replace the previously-configured component IP address with the IP address of the load balancer for this component.

CCM Database HA

To configure the CCM Database HA, you must install the following servers:

  • MGMTPOSTGRES_MASTER: A server that modifies data and carries the load.
  • MGMTPOSTGRES_SLAVE: A server that responds to or replicates changes made in the master server.
  • Virtual IP (VIP): The PostgreSQL instances launched in the cloud must be capable of handling network routing for your enterprise through the VIP configured by your cloud administrator.

Additional Nuances:

  • Place the two database servers on the same cloud or datacenter as the two CCM servers. This ensures:
    • A master-slave HA setup allows the secondary sever to seamlessly takes over, if the primary server fails at any point.
    • When a failed server comes back online, data is synchronized from the new master server automatically. The server that was offline becomes the slave server. When the old master server is back online, it rejoins the cluster and becomes the new slave server automatically.
  • Ensure that you accurately route the Virtual IP configuration to the appropriate MGMTPOSTGRES instance. 

  • The Tomcat service on a secondary CCM server does not restart after the Configure_HA field in the CCM wizard is executed from a primary CCM – if sudo privileges are not available on the secondary CCM server. If you need to start the Tomcat service remotely, then configure sudo privileges on the secondary CCM server.

CCM HA

The CloudCenter CCM and database servers work together to provide the following HA solution:

  • Synchronous transaction – Not considered committed until all servers have completed the transaction.
  • Failover – Ensures that a failover does not lose any data. If one of the CCM servers goes offline, the other server takes over as the primary server and continues to handle all required activities.
  • Consistent – Results returned to both servers as the database and configuration changes on the servers are kept in sync.
Icon

CloudCenter does not support cross-region configuration for CCM HA.

To configure CCM HA, you must install the following servers:

  • Two CCM servers:
    • CCM_SA_PRIMARY 
    • CCM_SA_SECONDARY
  • One Load balancer: The two CCM servers run concurrently behind a load balancer.

Additional Nuances:

  • Custom certificates, if used, must only be generated from the PRIMARY CCM and copied to ALL other components (for example, other CCMs, all CCOs, AMQPs, Monitors, and so forth).

CCO HA

Each server is active and all servers in the cluster perform orchestration tasks in parallel. If one of the servers in the cluster goes offline, the other active servers continue to handle orchestration tasks. When the offline server comes back online, data is synchronized from the active servers automatically.

Icon

CloudCenter does not support cross-region configuration for CCO HA.

To configure CCO HA, you must install the following servers:

  • Three VMs to support a 3-node MongoDB cluster:
    • CCO_PRIMARY (run the CCO configuration wizard on the CCO_PRIMARY.)
    • CCO_SECONDARY
    • CCO_TERTIARY.
  • One Load balancer: The THREE CCO servers run concurrently behind a load balancer.

    Icon

    When configuring the CCO in HA mode for the first time, you must point the load balancer to each CCO and register each of the three CCOs with the CCM. See the Register the CCO with the CCM section for procedural details.

AMQP HA

Icon

CloudCenter does not support cross-region configuration for AMQP HA.

Prerequisites before you begin the AMQP HA configuration:

  • Configure the CCO server to read the AMQP server's IP address – after setting up a load balancer on top of a RabbitMQ cluster.

  • In addition to the hostname requirements in the /etc/hosts file mentioned in the General Requirements section above – You must add the hostnames for both the primary and secondary AMQP instances in the /etc/hosts file that already exists in both AMQP servers.

    The /etc/hosts file

    Add the hostname file in the /etc folder and then restart the server – this step is mandatory to install AMQP in HA mode – You must add the hostnames for both the primary and secondary AMQP instances in the /etc/hostname file that already exists in both AMQP servers.

    The /etc/hostname file

     

To configure AMQP HA, you must install the following servers:

  • Two AMQP servers:
    • AMQP_PRIMARY 
    • AMQP_SECONDARY
  • One Load balancer: The two AMQP servers run concurrently behind a load balancer.
  • No labels