Occasionally, Helion Stackato may not swap memory correctly. For example, an idle cloud controller should use less than 1GB. However, it uses upwards of 2GB. You can resolve this issue by using one of the following solutions:
Restart the cloud controller manually using the
kato restart controller command.
This action will cause the temporary downtime of the API endpoint.
Configure the cloud controller to restart automatically:
sshinto the core node.
bundle exec rake stackato_clock:startcommand.
This command monitors the memory usage of both the
cloud_controller and its VM and restarts the
cloud_controller when the
metric exceeds the setting specified in
resource_monitoring. To verify the setting, run the
kato config get cloud_controller_ng
If you experience issues with establishing or maintaining WebSocket connections
logs subdomain of your cluster endpoint on port 80 because of network
configuration or connectivity issues, you can use the
--no-ws flag when
stackato push and
stackato logs commands.
When patching, the admin is prompted for the password at least once or more.
Upgrades to Helion Stackato 3.6 require more disk space than previous upgrades. See the Free Disk Space section of the Upgrade documentation for guidelines.
Users may report that
stackato ssh -a <app name> fails with the
error: Could not connect to CC API [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
Running the following command on all DEA nodes in the cluster will fix the issue:
When adding a node to a cluster that enables the DEA role, you must
either enter the password for the core node during the attach process or
transfer the public key from the
DEA node to the core node using
ssh-copy-id prior to attaching it.
If this is not done, the
stackato ssh command will not be able to
connect to any applications running on that DEA.
For DEA Auto Scaling, the
will need to be run on the DEA before it is saved as a template to allow
the new nodes to connect to the Core automatically.
UTM gateways such as Fortinet Fortigate or McAfee Web Gateway can interfere with upgrades, patches, or normal application staging if they are configured to inspect all HTTP or HTTPS traffic between Helion Stackato and upstream package repositories.
See the UTM / Gateway Firewalls section for a description of the problem and some suggested solutions.
If you attempt to boot a Helion Stackato VM on a network without DHCP enabled, with very restrictive security groups, or host-only networking, you may see the following error:
Timeout running firstboot task: /home/stackato/stackato/etc/firstboot/tasks/05-wait-for-config-redis.sh
In this state, the Helion Stackato roles will fail to start as they cannot
connect to the shared configuration system used by
kato. To fix
this, configure the network interface manually (
kato op static_ip
will not be available).
Log in via the hypervisor console as the 'stackato' system user and edit
/s/etc/network/interfaces-dhcp.tpl. Add the appropriate settings for
your network in the
primary network interface block. For example:
# The primary network interface auto eth0 iface eth0 inet static address 192.168.0.200 netmask 255.255.255.0 gateway 192.168.0.1 dns-nameservers 192.168.0.1
Save the changes to
interfaces-dhcp.tpl then run the following:
$ sudo cp /s/etc/network/interfaces-dhcp.tpl /etc/network/interfaces $ touch /home/stackato/.stackato-firstboot $ rm /home/stackato/.stackato-firstboot-error $ sudo reboot
If "Require sudo" is configured for Docker apps, removing this permission for users with deployed Docker apps will prohibit those users from successfully taking any further start or deploy actions on those apps. If the user attempts to restart or redeploy the application, it will fail. The Health Manager will continue to attempt to restart the Docker app, so the user (or an admin) should stop or delete it.
If the IP address of the core node changes, you must reconfigure the cluster to use the new MBUS IP address. Run kato node migrate (or kato op static_ip) on the core node, then kato node attach on all other cluster nodes to set the new MBUS IP.
The New Relic system monitoring (
newrelic-sysmond) and PHP
application monitoring (
newrelic-php5) packages are no longer
pre-installed in the default Helion Stackato Docker base container.
security_group key in the EC2 section of
been changed to
security_group_ids. Specify the AWS Security Groups
by ID rather than name in this setting.
If you are upgrading a Helion Stackato cluster that already has DEA autoscaling configured, the settings should be manually modified to use the new key.
Helion Stackato ships with PHP 5.5 in the default Docker image. This version is more recent than the one available in the standard Ubuntu repositories.
To get updates for PHP 5.5 or add additional modules (globally or as user-defined requirements for applications), a third-party package repository must be added to the Allowed Repos list.
Helion Stackato administrators can add the PPA for PHP 5.5 maintained by Ondřej Surý, or a different repository supporting compatible packages:
kato config push cloud_controller_ng allowed_repos \ "deb http://ppa.launchpad.net/ondrej/php5/ubuntu precise main"
In the Management Console settings
kato data users export and
kato data users import commands
(for saving and loading lists of users to and from CSV files) are
non-functional in the current release. The :ref`kato data import
<kato-command-ref-data-import>` and kato data export commands are still available for
migrating users from one Helion Stackato system to another.
Certain Helion Stackato processes noted below use non-standard logging formats and are therefore not included in the Cloud Events log stream (for example, in the Cloud Events view of the Management Console).
router2g(Helion Stackato router): manually inspect
/s/logs/router2g.logon Router nodes
stackato_rest(Stackato-specific web service): manually inspect
/s/logs/stackato_rest.logon Controller nodes
harbor_proxy_connector(Harbor): manually inspect
/s/logs/harbor_proxy_connector.logon Harbor nodes
cc_upload_server(CC upload server): manually inspect
/s/logs/cloud_controller_upload_server.logon Controller nodes
stackato-tty.log(Helion Stackato TTY console): manually inspect
/s/logs/stackato-tty.logon all nodes
Buildpacks used to rely on the
config_vars feature of bin/release
to set environment variables, but this has been deprecated by Heroku.
The replacement mechanism is to create a shell script in $HOME/.profile.d to set environment variables. This mechanism is fully supported in Helion Stackato 3.0 / Cloud Foundry v2, and is used by all of the built-in buildpacks.
When using the Legacy Buildpack, the environment
variable values specified in
manifest.yml (within the
cannot be updated without re-pushing the application with new settings.
Changes to variables made in the Management Console will be overwritten
by the original ones defined at push when the application is restarted.
To modify custom environment variables, push the application again after
changing the values in
Helion Stackato 2.10 shipped with RabbitMQ 2.4 (
rabbitmq service) enabled by
default. Helion Stackato 3.0 has RabbitMQ 2.4 and 3.1, (
rabbitmq3 respectively) but both are disabled by default. If you are
importing RabbitMQ 2.4 service data from a 2.10 cluster using
data import be sure you have the 'rabbitmq' service enabled first.
There are changes in RabbitMQ 3.0 that are incompatible with version 2.4.
With Helion Stackato set in Maintenance Mode, all
_gateway processes will report the following error once per minute:
Failed registering with cloud controller, status=503
This is normal, and can be safely ignored. The service nodes will reconnect once the Controller is taken out of Maintenance Mode.
If the core node of a Helion Stackato cluster is offline for more than 90
kato status will show processes on other nodes (systail,
apptail, router and others) in a
FATAL or (hung)
STARTING state. These
processes will not automatically reconnect to the core node.
To correct this, run
kato start (for
FATAL processes) or
STARTING processes) on all affected nodes.
Cluster configurations make use of private IP addresses for identifying the various cluster nodes. Best practice is to avoid the literal use of these addresses wherever possible, because these addresses are subject to change with cluster configuration.
If the VM instance can locally resolve a hostname rather than an IP address, it is generally best practice to use the hostname.
If not, Helion Stackato provides various
so that applications do not need to hard-code them at install time.
Some examples are
Using these variables is strongly encouraged.
A known issue is that some applications have install procedures that cannot be configured to make
use of these variables. If the server that provides the app's database
mysql_gateway/node) gets moved to another location, the only way for the app to
become aware of the new credentials is by restaging the app as noted below.
A restart is not sufficient.
Choose one of the following according to need, either:
$ stackato delete -n $ stackato push -n
$ stackato delete -n $ stackato update -n
Another possible workaround in such cases is to write a script that will pull the credentials
VCAP_SERVICES and update the app's config as needed, then add this script to the