Known Issues

Issues with Helion Stackato Not Swapping Memory Correctly

Occasionally, Helion Stackato may not swap memory correctly. For example, an idle cloud controller should use less than 1GB. However, it uses upwards of 2GB. You can resolve this issue by using one of the following solutions:

  • Restart the cloud controller manually using the kato restart controller command.

    Important

    This action will cause the temporary downtime of the API endpoint.

  • Configure the cloud controller to restart automatically:

    1. ssh into the core node.
    2. Navigate to the /s/code/cloud_controller_ng directory.
    3. Run the bundle exec rake stackato_clock:start command.

    This command monitors the memory usage of both the cloud_controller and its VM and restarts the cloud_controller when the metric exceeds the setting specified in resource_monitoring. To verify the setting, run the kato config get cloud_controller_ng resource_monitoring command.

Issues with Establishing or Maintaining WebSocket Connections

If you experience issues with establishing or maintaining WebSocket connections to the logs subdomain of your cluster endpoint on port 80 because of network configuration or connectivity issues, you can use the --no-ws flag when running the stackato push and stackato logs commands.

Multiple Admin Password Prompts During Patch

When patching, the admin is prompted for the password at least once or more.

Upgrade Disk Requirements

Upgrades to Helion Stackato 3.6 require more disk space than previous upgrades. See the Free Disk Space section of the Upgrade documentation for guidelines.

stackato ssh Bad Handshake

Users may report that stackato ssh -a <app name> fails with the following error:

error: Could not connect to CC API [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

Running the following command on all DEA nodes in the cluster will fix the issue:

/usr/bin/stackato-update-ssl-cert-from-core

Password Prompt When Attaching DEA

When adding a node to a cluster that enables the DEA role, you must either enter the password for the core node during the attach process or transfer the public key from the DEA node to the core node using ssh-copy-id prior to attaching it.

If this is not done, the stackato ssh command will not be able to connect to any applications running on that DEA.

For DEA Auto Scaling, the ssh-copy-id command will need to be run on the DEA before it is saved as a template to allow the new nodes to connect to the Core automatically.

UTM Gateways and Upgrade, Patch, and Staging Problems

UTM gateways such as Fortinet Fortigate or McAfee Web Gateway can interfere with upgrades, patches, or normal application staging if they are configured to inspect all HTTP or HTTPS traffic between Helion Stackato and upstream package repositories.

See the UTM / Gateway Firewalls section for a description of the problem and some suggested solutions.

Booting Without Network/DHCP

If you attempt to boot a Helion Stackato VM on a network without DHCP enabled, with very restrictive security groups, or host-only networking, you may see the following error:

Timeout running firstboot task: /home/stackato/stackato/etc/firstboot/tasks/05-wait-for-config-redis.sh

In this state, the Helion Stackato roles will fail to start as they cannot connect to the shared configuration system used by kato. To fix this, configure the network interface manually (kato op static_ip will not be available).

Log in via the hypervisor console as the 'stackato' system user and edit /s/etc/network/interfaces-dhcp.tpl. Add the appropriate settings for your network in the primary network interface block. For example:

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.0.200
        netmask 255.255.255.0
        gateway 192.168.0.1
        dns-nameservers 192.168.0.1

Save the changes to interfaces-dhcp.tpl then run the following:

$ sudo cp /s/etc/network/interfaces-dhcp.tpl /etc/network/interfaces
$ touch /home/stackato/.stackato-firstboot
$ rm /home/stackato/.stackato-firstboot-error
$ sudo reboot

Not Authorized (403) While Logged into Web Console

Some users report being unable to perform CLI operations if the same user account is also logged in to the web UI from more than one browser. The message returned by the stackato client in this case is:

You are not authorized to perform the requested action (403)

If you encounter this state, logging out of the Web Management Console unblocks the CLI operations.

Removing sudo Permissions for Users With Deployed Docker Applications

If "Require sudo" is configured for Docker apps, removing this permission for users with deployed Docker apps will prohibit those users from successfully taking any further start or deploy actions on those apps. If the user attempts to restart or redeploy the application, it will fail. The Health Manager will continue to attempt to restart the Docker app, so the user (or an admin) should stop or delete it.

Changing the MBUS IP (Core Node)

If the IP address of the core node changes, you must reconfigure the cluster to use the new MBUS IP address. Run kato node migrate (or kato op static_ip) on the core node, then kato node attach on all other cluster nodes to set the new MBUS IP.

Adding New Relic Repositories

The New Relic system monitoring (newrelic-sysmond) and PHP application monitoring (newrelic-php5) packages are no longer pre-installed in the default Helion Stackato Docker base container.

To allow end users to install these packages in the requirements: section of their application config, admins should add New Relic repository URLS and keys to the system configuration.

security_groups_ids in autoscaling.yaml

The security_group key in the EC2 section of autoscaling.yml has been changed to security_group_ids. Specify the AWS Security Groups by ID rather than name in this setting.

If you are upgrading a Helion Stackato cluster that already has DEA autoscaling configured, the settings should be manually modified to use the new key.

PHP 5.5 Repositories

Helion Stackato ships with PHP 5.5 in the default Docker image. This version is more recent than the one available in the standard Ubuntu repositories.

To get updates for PHP 5.5 or add additional modules (globally or as user-defined requirements for applications), a third-party package repository must be added to the Allowed Repos list.

Helion Stackato administrators can add the PPA for PHP 5.5 maintained by Ondřej Surý, or a different repository supporting compatible packages:

  • Using kato:

    kato config push cloud_controller_ng allowed_repos \
    "deb http://ppa.launchpad.net/ondrej/php5/ubuntu precise main"
    
  • In the Management Console settings

kato data users export|import Broken

The kato data users export and kato data users import commands (for saving and loading lists of users to and from CSV files) are non-functional in the current release. The :ref`kato data import <kato-command-ref-data-import>` and kato data export commands are still available for migrating users from one Helion Stackato system to another.

Cloud Events Do Not Gather Logs from Some Processes

Certain Helion Stackato processes noted below use non-standard logging formats and are therefore not included in the Cloud Events log stream (for example, in the Cloud Events view of the Management Console).

  • router2g (Helion Stackato router): manually inspect /s/logs/router2g.log on Router nodes
  • stackato_rest (Stackato-specific web service): manually inspect /s/logs/stackato_rest.log on Controller nodes
  • harbor_proxy_connector (Harbor): manually inspect /s/logs/harbor_proxy_connector.log on Harbor nodes
  • cc_upload_server (CC upload server): manually inspect /s/logs/cloud_controller_upload_server.log on Controller nodes
  • stackato-tty.log (Helion Stackato TTY console): manually inspect /s/logs/stackato-tty.log on all nodes

Buildpack config_vars is Deprecated

Buildpacks used to rely on the config_vars feature of bin/release to set environment variables, but this has been deprecated by Heroku.

The replacement mechanism is to create a shell script in $HOME/.profile.d to set environment variables. This mechanism is fully supported in Helion Stackato 3.0 / Cloud Foundry v2, and is used by all of the built-in buildpacks.

Legacy Buildpack and Environment Variables

When using the Legacy Buildpack, the environment variable values specified in manifest.yml (within the env: block) cannot be updated without re-pushing the application with new settings. Changes to variables made in the Management Console will be overwritten by the original ones defined at push when the application is restarted.

To modify custom environment variables, push the application again after changing the values in manifest.yml.

Importing Apps using RabbitMQ 2.4

Helion Stackato 2.10 shipped with RabbitMQ 2.4 (rabbitmq service) enabled by default. Helion Stackato 3.0 has RabbitMQ 2.4 and 3.1, (rabbitmq and rabbitmq3 respectively) but both are disabled by default. If you are importing RabbitMQ 2.4 service data from a 2.10 cluster using kato data import be sure you have the 'rabbitmq' service enabled first.

Note

There are changes in RabbitMQ 3.0 that are incompatible with version 2.4.

Service Gateway Log Errors in Maintenance Mode

With Helion Stackato set in Maintenance Mode, all _gateway processes will report the following error once per minute:

Failed registering with cloud controller, status=503

This is normal, and can be safely ignored. The service nodes will reconnect once the Controller is taken out of Maintenance Mode.

Nodes with FATAL or Perpetually STARTING Processes

If the core node of a Helion Stackato cluster is offline for more than 90 seconds, kato status will show processes on other nodes (systail, apptail, router and others) in a FATAL or (hung) STARTING state. These processes will not automatically reconnect to the core node.

To correct this, run kato start (for FATAL processes) or kato restart (for STARTING processes) on all affected nodes.

Avoiding Application Reliance on IP Addresses

Cluster configurations make use of private IP addresses for identifying the various cluster nodes. Best practice is to avoid the literal use of these addresses wherever possible, because these addresses are subject to change with cluster configuration.

If the VM instance can locally resolve a hostname rather than an IP address, it is generally best practice to use the hostname.

If not, Helion Stackato provides various environment variables so that applications do not need to hard-code them at install time. Some examples are VCAP_SERVICES, STACKATO_SERVICES, and DATABASE_URL. Using these variables is strongly encouraged.

A known issue is that some applications have install procedures that cannot be configured to make use of these variables. If the server that provides the app's database (for example, mysql_gateway/node) gets moved to another location, the only way for the app to become aware of the new credentials is by restaging the app as noted below. A restart is not sufficient.

Choose one of the following according to need, either:

$ stackato delete -n
$ stackato push -n

or:

$ stackato delete -n
$ stackato update -n

Another possible workaround in such cases is to write a script that will pull the credentials from VCAP_SERVICES and update the app's config as needed, then add this script to the pre-running hooks.