HPE Helion Stackato |
© Copyright 2016 Hewlett Packard Enterprise Development LP
Occasionally, Helion Stackato may not swap memory correctly. For example, an idle cloud controller should use less than 1GB. However, it uses upwards of 2GB. You can resolve this issue by using one of the following solutions:
Restart the cloud controller manually using the kato restart controller command.
Important
This action will cause the temporary downtime of the API endpoint.
Configure the cloud controller to restart automatically:
ssh into the core node./s/code/cloud_controller_ng directory.bundle exec rake stackato_clock:start command.This command monitors the memory usage of both the cloud_controller and its VM and restarts the cloud_controller when the
metric exceeds the setting specified in resource_monitoring. To verify the setting, run the kato config get cloud_controller_ng
resource_monitoring command.
If you experience issues with establishing or maintaining WebSocket connections
to the logs subdomain of your cluster endpoint on port 80 because of network
configuration or connectivity issues, you can use the --no-ws flag when
running the stackato push and stackato logs commands.
When patching, the admin is prompted for the password at least once or more.
Upgrades to Helion Stackato 3.6 require more disk space than previous upgrades. See the Free Disk Space section of the Upgrade documentation for guidelines.
Users may report that stackato ssh -a <app name> fails with the
following error:
error: Could not connect to CC API [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
Running the following command on all DEA nodes in the cluster will fix the issue:
/usr/bin/stackato-update-ssl-cert-from-core
When adding a node to a cluster that enables the DEA role, you must
either enter the password for the core node during the attach process or
transfer the public key from the
DEA node to the core node using ssh-copy-id prior to attaching it.
If this is not done, the stackato ssh command will not be able to
connect to any applications running on that DEA.
For DEA Auto Scaling, the ssh-copy-id command
will need to be run on the DEA before it is saved as a template to allow
the new nodes to connect to the Core automatically.
UTM gateways such as Fortinet Fortigate or McAfee Web Gateway can interfere with upgrades, patches, or normal application staging if they are configured to inspect all HTTP or HTTPS traffic between Helion Stackato and upstream package repositories.
See the UTM / Gateway Firewalls section for a description of the problem and some suggested solutions.
If you attempt to boot a Helion Stackato VM on a network without DHCP enabled, with very restrictive security groups, or host-only networking, you may see the following error:
Timeout running firstboot task: /home/stackato/stackato/etc/firstboot/tasks/05-wait-for-config-redis.sh
In this state, the Helion Stackato roles will fail to start as they cannot
connect to the shared configuration system used by kato. To fix
this, configure the network interface manually (kato op static_ip
will not be available).
Log in via the hypervisor console as the 'stackato' system user and edit
/s/etc/network/interfaces-dhcp.tpl. Add the appropriate settings for
your network in the primary network interface block. For example:
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.0.200
netmask 255.255.255.0
gateway 192.168.0.1
dns-nameservers 192.168.0.1
Save the changes to interfaces-dhcp.tpl then run the following:
$ sudo cp /s/etc/network/interfaces-dhcp.tpl /etc/network/interfaces
$ touch /home/stackato/.stackato-firstboot
$ rm /home/stackato/.stackato-firstboot-error
$ sudo reboot
Some users report being unable to perform CLI operations if the same
user account is also logged in to the web UI from more than one browser.
The message returned by the stackato client in this case is:
You are not authorized to perform the requested action (403)
If you encounter this state, logging out of the Web Management Console unblocks the CLI operations.
If "Require sudo" is configured for Docker apps, removing this permission for users with deployed Docker apps will prohibit those users from successfully taking any further start or deploy actions on those apps. If the user attempts to restart or redeploy the application, it will fail. The Health Manager will continue to attempt to restart the Docker app, so the user (or an admin) should stop or delete it.
If the IP address of the core node changes, you must reconfigure the cluster to use the new MBUS IP address. Run kato node migrate (or kato op static_ip) on the core node, then kato node attach on all other cluster nodes to set the new MBUS IP.
The New Relic system monitoring (newrelic-sysmond) and PHP
application monitoring (newrelic-php5) packages are no longer
pre-installed in the default Helion Stackato Docker base container.
To allow end users to install these packages in the requirements: section of their application config, admins should add New Relic repository URLS and keys to the system configuration.
The security_group key in the EC2 section of autoscaling.yml has
been changed to security_group_ids. Specify the AWS Security Groups
by ID rather than name in this setting.
If you are upgrading a Helion Stackato cluster that already has DEA autoscaling configured, the settings should be manually modified to use the new key.
Helion Stackato ships with PHP 5.5 in the default Docker image. This version is more recent than the one available in the standard Ubuntu repositories.
To get updates for PHP 5.5 or add additional modules (globally or as user-defined requirements for applications), a third-party package repository must be added to the Allowed Repos list.
Helion Stackato administrators can add the PPA for PHP 5.5 maintained by Ondřej Surý, or a different repository supporting compatible packages:
Using kato:
kato config push cloud_controller_ng allowed_repos \
"deb http://ppa.launchpad.net/ondrej/php5/ubuntu precise main"
In the Management Console settings
The kato data users export and kato data users import commands
(for saving and loading lists of users to and from CSV files) are
non-functional in the current release. The :ref`kato data import
<kato-command-ref-data-import>` and kato data export commands are still available for
migrating users from one Helion Stackato system to another.
Certain Helion Stackato processes noted below use non-standard logging formats and are therefore not included in the Cloud Events log stream (for example, in the Cloud Events view of the Management Console).
router2g (Helion Stackato router): manually inspect /s/logs/router2g.log on Router nodesstackato_rest (Stackato-specific web service): manually inspect /s/logs/stackato_rest.log on Controller nodesharbor_proxy_connector (Harbor): manually inspect /s/logs/harbor_proxy_connector.log on Harbor nodescc_upload_server (CC upload server): manually inspect /s/logs/cloud_controller_upload_server.log on Controller nodesstackato-tty.log (Helion Stackato TTY console): manually inspect /s/logs/stackato-tty.log on all nodesBuildpacks used to rely on the config_vars feature of bin/release
to set environment variables, but this has been deprecated by Heroku.
The replacement mechanism is to create a shell script in $HOME/.profile.d to set environment variables. This mechanism is fully supported in Helion Stackato 3.0 / Cloud Foundry v2, and is used by all of the built-in buildpacks.
When using the Legacy Buildpack, the environment
variable values specified in manifest.yml (within the env: block)
cannot be updated without re-pushing the application with new settings.
Changes to variables made in the Management Console will be overwritten
by the original ones defined at push when the application is restarted.
To modify custom environment variables, push the application again after
changing the values in manifest.yml.
Helion Stackato 2.10 shipped with RabbitMQ 2.4 (rabbitmq service) enabled by
default. Helion Stackato 3.0 has RabbitMQ 2.4 and 3.1, (rabbitmq and
rabbitmq3 respectively) but both are disabled by default. If you are
importing RabbitMQ 2.4 service data from a 2.10 cluster using kato
data import be sure you have the 'rabbitmq' service enabled first.
Note
There are changes in RabbitMQ 3.0 that are incompatible with version 2.4.
With Helion Stackato set in Maintenance Mode, all
_gateway processes will report the following error once per minute:
Failed registering with cloud controller, status=503
This is normal, and can be safely ignored. The service nodes will reconnect once the Controller is taken out of Maintenance Mode.
If the core node of a Helion Stackato cluster is offline for more than 90
seconds, kato status will show processes on other nodes (systail,
apptail, router and others) in a FATAL or (hung) STARTING state. These
processes will not automatically reconnect to the core node.
To correct this, run kato start (for FATAL processes) or kato
restart (for STARTING processes) on all affected nodes.
Cluster configurations make use of private IP addresses for identifying the various cluster nodes. Best practice is to avoid the literal use of these addresses wherever possible, because these addresses are subject to change with cluster configuration.
If the VM instance can locally resolve a hostname rather than an IP address, it is generally best practice to use the hostname.
If not, Helion Stackato provides various
environment variables
so that applications do not need to hard-code them at install time.
Some examples are VCAP_SERVICES, STACKATO_SERVICES, and DATABASE_URL.
Using these variables is strongly encouraged.
A known issue is that some applications have install procedures that cannot be configured to make
use of these variables. If the server that provides the app's database
(for example, mysql_gateway/node) gets moved to another location, the only way for the app to
become aware of the new credentials is by restaging the app as noted below.
A restart is not sufficient.
Choose one of the following according to need, either:
$ stackato delete -n
$ stackato push -n
or:
$ stackato delete -n
$ stackato update -n
Another possible workaround in such cases is to write a script that will pull the credentials
from VCAP_SERVICES and update the app's config as needed, then add this script to the
pre-running hooks.