Cluster Setup

This process begins with an installed micro cloud, which must then be cloned across several nodes. You connect to each node in turn and tell it which roles it is to serve, thereby distributing the processing load for maximum performance.

Roles

A Stackato node can take on one or more of the following roles:

The command line tool used to configure Stackato servers is called kato. You can see a list of the available roles at the command line by running the kato info command.

Setup of cluster nodes is done using the kato node setup, add, attach, and remove sub-commands.

The kato info command will show:

  • assigned roles: roles currently configured to run on the node
  • available roles: roles which can be added with kato role add

Preparing the Core Node

In a Stackato cluster, one node is dedicated as the Core node. This node will have a controller, primary, base, and router role but can also include additional roles.

Boot a Stackato VM and set up the Core node as described below, then add the other nodes and assign roles.

CORE_IP

A static IP address is necessary to provide a consistent network interface for other nodes to connect to. If your IaaS or cloud orchestration software provide IP addresses which persist indefinitely and are not reset on reboot you may not have to set this explicitly.

Take note of the IP address of the Core node. It will be required when configuring additional nodes in the following steps, so that they can attach to the Core node.

Make sure that the IP address of its eth0 interface is registering the correct address, which may not be the case if you have set a static IP and not yet rebooted or restarted networking. To check the IP address, run:

$ ifconfig eth0

If necessary, set the static IP address:

$ kato op static_ip

Note

If the IP address of the Core node changes, the kato node migrate command must be run on all nodes in the cluster (starting with the Core node) to set the new CORE_IP.

Hostname

Next, set the fully qualified hostname of the Core node. This is required so that Stackato's internal configuration matches the DNS record created for this system.

To set the hostname, run:

$ kato node rename hostname.example.com --no-restart

This hostname will become the basename of the "API endpoint" address used by clients (e.g. "https://api.hostname.example.com").

Note

If you are building a cluster with multiple Routers separate from the Core node, the load balancer or gateway router must take on the API endpoint address. Consult the Load Balancer and Multiple Routers section below.

Wildcard DNS

A wildcard DNS record is necessary to resolve not only the API endpoint, but all applications which will subsequently be deployed on the PaaS. Create a wildcard DNS record for the Core node (or Load Balancer/Router).

Core Node

On the Core node, execute the following command:

$ kato node setup core api.hostname.example.com

This sets up the Core node with just the implicit controller, primary, and router roles.

If you intend to set up the rest of the cluster immediately, you would carry on to enable those roles you ultimately intend to run on the Core node. For example, to set up a Core node with the controller, primary router, and dea roles:

$ kato node setup core api.hostname.example.com
$ kato role add dea

Then proceed to configure the other VMs by attaching them to the Core node and assigning their particular roles.

Attaching Nodes and Enabling Roles

Adding nodes to the cluster involves attaching the new VMs to the Core node's IP address using the kato node attach command. This command will check that the new node has a version number compatible with the Core node before attaching it.

Roles can be added (or removed) on the new node after attaching using the kato role command, but it is generally preferable to enable roles during the kato attach step using the -e (enable) option as described below for each of the node types.

Router Nodes

In smaller clusters, the Router role can be run on the Core Node. To run its own on a separate node:

$ kato node attach -e router CORE_IP

Note that the public DNS entry for the Stackato cluster's API endpoint must resolve to the Router if it is separate from the Core Node. For clusters requiring multiple Routers, see the Load Balancer and Multiple Routers section below.

Data Services Nodes

Data services can share a single node (small clusters) or run on separate nodes (recommended for production clusters). To set up all available data services on a single node and attach it to the Core node, run the following command on the data services node:

$ kato node attach -e data-services CORE_IP

Note

The Harbor port service needs a publicly routable IP and exposed port range if you want to provide externally accessible TCP and UDP ports for user applications. See the Harbor Requirements & Setup documentation for details.

DEA Nodes

Nodes which stage application code and run application containers are called Droplet Execution Agents (DEAs). Once the controller node is running, you can begin to add some of these nodes with the kato node attach command. To turn a generic Stackato VM into a DEA and connect it to the Core node:

$ kato node attach -e dea CORE_IP

Continue this process until you have added all the desired DEA nodes.

Verification

To verify that all the cluster nodes are configured as expected, run the following command on the Core node:

$ kato status --all

Removing Nodes

Use the kato node remove to remove a node from the cluster. Run the following command on the core node.

$ kato node remove NODE_IP

Role Configuration using the Management Console

Once cluster nodes are connected to the Core node, roles can be enabled or disabled using the Cluster Admin interface in the Management Console.

Example Clusters

Single-Node

This is a configuration (not actually a cluster) which you would not generally deploy in production, but it helps to illustrate the role architecture in Stackato. A node in this configuration will function much like a micro cloud, but can be used as the starting point for building a cluster later.

All that is required here is to enable all roles except for mdns (not used in a clustered or cloud-hosted environment):

$ kato node setup core api.hostname.example.com
$ kato role add --all-but mdns

Three-Node

This is the smallest viable cluster deployment, but it lacks the fault tolerance of larger configurations:

  • 1 Core node consisting of primary, controller, and router (and supporting processes)
  • 1 data-services node running the database, messaging and filesystem services
  • 1 DEA (Droplet Execution Agent) node

This configuration can support more users and applications than a single node, but the failure of any single node will impact hosted applications.

Five-Node

A typical small Stackato cluster deployment might look like this:

  • 1 Core node consisting of primary, controller, and router (and supporting processes)
  • 1 data-services node running the database, messaging and filesystem services
  • 3 DEA (Droplet Execution Agent) nodes

In this configuration, fault tolerance (and limited scalability) is introduced in the pool of DEA nodes. If any single DEA node fails, application instances will be automatically redeployed to the remaining DEA nodes with little or no application down time.

20-Node

A larger cluster requires more separation and duplication of roles for scalability and fault tolerance. For example:

  • 1 Core node running the primary and controller roles (with supporting processes)
  • 1 supplemental Controller node (sharing a filesystem and PostgreSQL database with the Core node)
  • 1 Load Balancer (Stackato VM or hardware)
  • 2 Router nodes
  • 1 Filesystem service node
  • 1 PostgreSQL + MySQL data service node
  • 1 MongoDB, Redis, RabbitMQ + other data service node
  • 12 DEA (Droplet Execution Agent) nodes

In this configuration:

  • application instances span a larger group of DEA nodes so applications can be easily scaled to meet increasing demand
  • web requests are evenly distributed between two Router nodes, either of which can fail without any interruption of service
  • any data service node failure will be localized, not affecting data services on other nodes
  • the auxiliary controller balances the load on the Management Console and system management tasks

Roles Requiring Persistent or Shared Storage

Though all roles can run using the VM's default filesystem, in production clusters some roles should always be backed by a persistent filesystem (block storage/EBS volumes) to provide scalable storage space and easy snapshotting. Nodes with the following roles should have their /var/stackato/services directory on persistent storage:

  • Data Services: MySQL, PostgreSQL, MongoDB, Redis
  • Filesystem Service
  • Memcache
  • RabbitMQ
  • Harbor

Note

Though Memcache and Redis are in-memory data stores, system service info data is stored on disk, so backing them with a persistent filesystem is recommended.

In clusters with multiple Cloud Controllers, the nodes must share a common /home/stackato/stackato/data mount point as described below in order to work together properly.

See the Persistent Storage documentation for instructions on relocating service data, application droplets, and containers.

Port Configuration

The Stackato micro cloud runs with the following ports exposed:

Port Type Service
22 tcp ssh
25 tcp smtp
80 tcp http
111 tcp portmapper
111 udp portmapper
443 tcp https
3306 tcp mysql
5432 tcp postgresql
5678 tcp DEA directory server
9001 tcp supervisord

On a production cluster, or a micro cloud running on a cloud hosting provider, only ports 22 (SSH), 80 (HTTPS) and 443 (HTTPS) need to be exposed externally (e.g. for the Router / Core node).

Within the cluster (i.e. behind the firewall), it is advisable to allow communication between the cluster nodes on all ports. This can be done safely by using the security group / security policy tools provided by your hypervisor:

If you wish to restrict ports between some nodes (e.g. if you do not have the option to use security groups), the following summary describes which ports are used by which components. Source nodes initiate the communication, Destination nodes need to listen on the specified port.

Port Range Type Source Destination Required by
22 tcp all nodes all nodes ssh/scp/sshfs
4222 tcp all nodes controller NATS
3306 tcp dea,controller mysql nodes MySQL
5432 tcp dea,controller postgresql nodes PostgreSQL
5454 tcp all nodes controller redis
6464 tcp all nodes all nodes applog (redis)
7000 - 7999 tcp all nodes all nodes kato log tail
7474 tcp all nodes all nodes config (redis)
9001 tcp controller all nodes supervisord
9022 tcp dea controller droplets
9022 tcp controller dea droplets
9025 tcp controller router stackato-rest
9026 tcp controller all nodes stackato-rest
41000 - 61000 tcp dea,controller service nodes service gateways

Each node can be internally firewalled using iptables to apply the above rules.

Comments:

  • Ports 80 and 443 need only be open to the world on router nodes.
  • Port 4222 should be open on all nodes for NATS communication with the MBUS IP (core Cloud Controller)
  • Port 9022 should be open to allow transfer of droplets to and from the DEAs, and Cloud Controllers.
  • Port 7845 is required if you plan to stream logs from all nodes in a cluster using kato log tail command.
  • External access on port 22 can be restricted if necessary to the subnet you expect to connect from. If you are providing the stackato ssh feature to your users (recommended), define a distinct security group for the public-facing Cloud Controller node that is the same as a generic Stackato group, but has the additional policy of allowing SSH (Port 22) from hosts external to the cluster.
  • Within the cluster, port 22 should be open on all hosts to allow administrative access over SSH. Port 22 is also used to mount Filesystem service partitions in application containers on the DEA nodes (via SSHFS).
  • The optional Harbor port service has a configurable port range (default 41000 - 61000) which can be exposed externally if required.

Service Nodes

In addition to the ports listed above for service nodes and gateways, several service nodes assign a port for each individual user-requested service instance. These ranges should be kept open between DEA nodes and their respective service nodes. The default ranges are:

  • harbor: 35000 - 40000
  • memcached: 45001 - 50000
  • mongodb: 15001 - 25000
  • rabbit: 35001 - 40000
  • rabbit3: 25001 - 30000
  • redis: 5000 - 15000

Note

You can check the currently configured port range for each service with kato config (e.g. kato config get redis_node port_range).

Note

Harbor (Port Service) Node Configuration

The optional Harbor TCP/UDP port service must be set up on a node with a public network interface if you wish to enable port forwarding for user applications. The security group or firewall settings for this node should make the configured port range accessible publicly. See Harbor Setup for full configuration instructions.

Multiple Controllers

A Stackato cluster can have multiple controller nodes running on separate VMs to improve redundancy. The key element in designing this redundancy is to have all controller nodes share the following two important data directories on a high-availability filesystem server:

  • /home/stackato/stackato/data
  • /var/stackato/data/cloud_controller_ng/tmp/staged_droplet_uploads

For example, to share /home/stackato/stackato/data:

  • Create a shared filesystem on a Network Attached Storage device. [1]

  • Stop the controller process on the Core node before proceeding further:

    $ kato stop controller
  • On the Core node and each additional controller node:

    • Create a mount point:

      $ sudo mkdir /mnt/controller
    • Mount the shared filesystem on the mount point. [1] For example:

      $ sshfs -o idmap=user -o reconnect -o allow_other -o ServerAliveInterval=15 stackato@10.0.0.3:/mnt/add-volume/stackato-shared/ /mnt/controller
    • Set aside the original /home/stackato/stackato/data:

      $ mv /home/stackato/stackato/data /home/stackato/stackato/data.old
    • Create a symlink from /home/stackato/stackato/data to the mount point:

      $ ln -s /mnt/controller /home/stackato/stackato/data
  • On the Core node, start the controller process:

    $ kato start controller
  • Run the following command on the additional Controller nodes to enable only the controller process:

    $ kato node attach -e controller *CORE_IP*
[1](1, 2)

The type of filesystem, storage server, and network mount method are left to the discretion of the administrator. When using sshfs (recommended) be sure to set the following options:

  • idmap=user
  • reconnect
  • allow_other

Load Balancer and Multiple Routers

For large scale deployments requiring multiple Router nodes, a load balancer must be configured to distribute connections between the Routers. Though most users will prefer to use a hardware load balancer or elastic load balancing service provided by the cloud hosting provider, a Stackato VM can be configured to take on this role.

The kato node setup load_balancer command retrieves IP addresses of every router in the cluster and configures an nginx process to distribute load (via round-robin) among a pool of Routers and handle SSL termination.

For example, to setup a cluster with a Stackato Load Balancer and multiple Routers:

Rename the Load Balancer

The Load Balancer is the primary point of entry to the cluster. It must have a public-facing IP address and take on the primary hostname for the system as configured in DNS. Run the following on Load Balancer node:

$ kato node rename *hostname.example.com*

Set up the Core Node

The Core node will need to temporarily take on the API endpoint hostname of the Stackato system (i.e. the same name as the Load Balancer above). Run the following on the Core node:

$ kato node rename *hostname.example.com*

If it is not already configured as the Core node, do so now:

$ kato node setup core api.\ *hostname.example.com*

The kato node rename command above is being used to set internal Stackato parameters, but all hosts on a network should ultimately have unique hostnames. After setup, rename the Core node manually by editing /etc/hostname and /etc/hosts, then sudo service hostname restart.

Set up Supplemental Routers

As with the Core node, you will need to run kato node rename on each router with the same API endpoint hostname. Run the following on each Router:

$ kato node rename *hostname.example.com*

Then enable the 'router' role and attach the node to the cluster:

$ kato node attach -e router <MBUS_IP>

As above, rename each host manually after configuration to give them unique hostnames. The MBUS_IP is the network interface of the Core node (usually eth0).

Configure the Stackato Load Balancer

Note

A Stackato node configured as a Load Balancer cannot have any other roles enabled.

Attach the Stackato VM to the Core node:

$ kato node attach <MBUS_IP>

To set up the node as a Load Balancer automatically:

$ kato node setup load_balancer --force

This command fetches the IP addresses of all configured routers in the cluster.

To set up the Load Balancer manually, specify the IP addresses of the Router nodes. For example:

$ kato node setup load_balancer 10.5.31.140 10.5.31.145

Load Balancer SSL Certificates

The load balancer terminates SSL connections, so SSL certificates must be set up and maintained on this node, and the router nodes that the load balancer distributes connections to. The SSL certs on the load balancer and routers must match in order for application SSO and AOK to work correctly.

See the Using your own SSL certificate and CA Certificate Chaining sections for Stackato Load Balancer instructions.

For other load balancers, consult the documentation for your device or service on uploading/updating server certificates.