High Availability Best Practices
The following image identifies two High Availability (HA) requirements:
- Do NOT add any special characters in the hostname.
- Do NOT add the hostname on the same line as the localhost. Add the hostname on a separate line in the /etc/hosts file
The hostname cannot contain space, underscore, hyphen, or any other special characters for the database servers as it causes the authentication to fail.
Changing the hostnames after you install and configure the VMs may cause unknown issues.
Be sure to set up each highly available VM in a separate zone to avoid situations that may contribute to a single point of failure (for example, a power outage that affects both VMs if they are situated in the same zone).
The HA setup is not supported with CentOS6 and RHEL6 images for all CloudCenter releases.
- MGMTPOSTGRES_MASTER: A server that modifies data. This server carries the load.
- MGMTPOSTGRES_SLAVE: A server that responds to or replicates changes made in the master server.
- Place the two
servers and the two database servers on the same cloud or datacenter. This ensures that:
- A master-slave HA setup allows the slave sever to seamlessly takes over, if the master server fails at any point
- When a failed server comes back online, data is synchronized from the new master server automatically. The server that was offline becomes the slave server. When the old master server is back online, it rejoins the cluster and becomes the new slave server automatically.
- Configure the database servers before configuring the CCM servers – The CCM servers are dependent on the database servers being set up and running.
- Configure the load balancer to dispatch traffic to one of the instances and the enable the session stickiness policy. Refer to your load balancer documentation for additional details.
- Ensure that you accurately route the Virtual IP configuration to the appropriate PostgreSQL instance – The PostgreSQL instances launched in the cloud must be capable of handling network routing for your enterprise through the Virtual IP configured by your cloud administrator.
The CloudCenter HA solution:
- Is Synchronous — the transaction is not considered committed until all servers have completed the transaction.
- Ensures that a failover does not lose any data. If one of the CCM servers goes offline, the other server takes over as the primary server and continues to handle all required activities.
- Returns consistent results to both servers as the database and configuration changes on the servers are kept in sync.
To ensure HA, the CloudCenter CCM anddatabase servers work together to allow a second server to take over quickly if the primary server fails.
To ensure better performance for a HA setup, the network link between the two CCMs in a HA setup should have a minimum requirement of 100Mbps bandwidth and a network latency of less than 20ms.
The CloudCenter platform uses Hazelcast in CCM HA setup. Hazelcast recommends clustered VMs be deployed within one region. To adhere to the Hazelcast recommendation, the CloudCenter platform should ideally be deployed in the same region with multiple zones.
To provide HA for the CCM server in a CloudCenter deployment, you must install the following servers:
- Two database servers: MGMTPOSTGRES_MASTER and MGMTPOSTGRES_SLAVE for replication setup.
- Two CCM servers: CCM_SA_PRIMARY and CCM_SA_SECONDARY for high availability setup.
The CCO HA procedure is intricate and requires deeper DevOps knowledge. First consult with your company's DevOps team and ensure that each requirement in this section is addressed.
The CCO servers run concurrently behind a load balancer. Each server is active and all servers in the cluster perform orchestration tasks in parallel. If one of the servers in the cluster goes offline, the other active servers continue to handle orchestration tasks. When the offline server comes back online, data is synchronized from the active servers automatically.
CloudCenter does not support cross-region configuration for CCO HA.
Be sure to launch all three CCO servers in different fault domains to ensure HA.
To configure CCO HA for CloudCenter 4.7.x:
- Install 3 VMs to support a 3-node MongoDB cluster: CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY.
- Run the configuration wizard on the CCM_PRIMARY.
To configure CCO HA for CloudCenter 4.6.x:
- CCO1 = Configure CCO HA and MongoDB HA
- CCO2 = Configure CCO HA and MongoDB HA
- CCO3 = MongoDB HA (MongoDB minimal requirement)
Using a CCO Load Balancer?Icon
SSL certificates are essential for communication between the CCM and CCO. Be sure to use the TCP protocol for CCO Load Balancer listeners. For example, you can configure a generic load balancer application to use TCP and ensure that the certificate exchange procedure is transparent.
- Use the CloudCenter UI and configure the CCO IP field with the IP address of the load balancer.
If you are adding new CCOs to an existing deployment, replace the previously configured CCO IP address with the IP address of the load balancer.
A load balancer must be placed on top of a RabbitMQ cluster to provide true HA support for a CloudCenter deployment.
If each AMQP server is placed in a separate region, be aware that the HA solution does not work across regions.
To ensure end-to-end AMQP HA, a CloudCenter deployment requires:
- A clustered AMQP server setup with mirroring – see https://www.rabbitmq.com/clustering.html for additional context on clustering servers.
- A load balancer to manage all AMQP servers – see the relevant documentation for your respective load balancers to configure this setup for your environment.
You should have already met these requirements before you begin the AMQP HA configuration:
Configured and setup the database.
- Configured the AMQP cluster using the DNS name – not the IP address.
- Configured a load balancer to manage the AMQP servers.
- Configured a clustered AMQP server setup with a mirroring policy applied to all CCO queues.
- Configured the CCO server to read the AMQP server's IP address – after setting up a load balancer on top of a RabbitMQ cluster.
CloudCenter 4.6 platform does not support HA for Guacamole.
CloudCenter 4.7.0, Cisco supports HA for Guacamole.
If you are using Guacamole in HA mode along with a load balancer, you must ensure that the individual IP address of the nodes are able to establish reverse connection between the connected nodes.
- No labels