Managing Hardware Nodes

Click the Hardware nodes option to see the information about your cluster state and manage the hardware nodes:

Cluster State

Once inside, you’ll see the list of regions, with expandable hardware sets and assigned hardware nodes, that your Jelastic cluster consists of:

jca hardware nodes section
The following information about your hardware is available within the columns of this list:
  • Name - hardware node name
  • Load Rating - a ratio of the allocated and free resources (higher value means higher load on the node); use the Interval drop-down list at the tools panel to select the required time frame (5, 15, 30, or 60 minutes)
  • Status - the possible values are:
    • ACTIVE - node operates normally; new containers can be added
    • INFRASTRUCTURE_NODE - infrastructure node; can't be used for evacuation
    • MAINTENANCE - node is under maintenance; the existing containers are working, but the new ones cannot be added
    • EVACUATING - the process of containers evacuation is run; node can't be edited or removed
    • EVACUATED - evacuation has been successfully completed
    • EVACUATION_FAILED - the process of evacuation has failed
    • BROKEN - some errors have occurred in node’s work
  • Virtualization - container-based virtualization solution used on a node: Virtuozzo (VZ-6 and VZ-7) or, for Windows-based nodes, Parallels Virtuozzo Containers (PVC)
  • Memory - the total RAM available on the hardware node (the value to the right) and the amount in use (to the left); hover over for additional details
  • Disk - the already allocated disk space (to the left) compared to the overall capacity of the hardware node disk (to the right)
  • Load Average 5/10/15 min - the average load rating for the denoted periods (higher value means higher load)
  • Swap - the amount of allocated and used swap space (to the right and left respectively)

Note: If the admin panel couldn't get the information about a server, the appropriate line will be colored in red. Hover over node Status to view additional details.
Select a particular node (click on it) to view the detailed information within the right-hand panel:

hardware nodes additional info
Here, several tabs are available:
  • State - shows a set of node’s parameters (can vary based on the OS type)
  • Load Info - lists all the stacks used on the node and the amount of the appropriate containers; can be filtered by status (Total, Launching, Down, Sleep, Running) using the drop-down list at the top
  • Containers - displays a list of all containers on the node with some additional information (Type, Node ID, Memory, Disk, and Environment). Also, you can migrate the selected containers by clicking the Migrate button at the top and specifying the target hardware Node and preferable Migration Mode

    Tips:

    • The icons before the container specify its state (green - running, red - stopped, blue - sleeping, grey - problematic)
    • Clicking on the particular environment name will redirect you to the Cluster > Environments JCA section, where you can view the detailed info (e.g. container owner).
    • You can select multiple containers for migration by using the Shift and Ctrl buttons on your keyboard.
  • Evacuation State - provides the Name of a node which is evacuating or has already been evacuated (the current state is displayed in the Value column). Using the buttons at the top, you can Refresh the list to see the latest changes or Stop the evacuation process
  • Cluster Info - shows the server’s roles (Client, Chunk Server, MDS)

Components Check

Components check allows the Jelastic Core to query a particular hardware node and verify whether it is appropriately configured to be used by the platform.

The main check application runs a list of tests on a hardware node and gathers the information about its state. Herewith, each check application represents a separate dynamic module, which allows adding new checks easily.

Currently, the following modules are used by default:
  • ip_tables checks firewall configuration on the hardware node
  • routes checks the correctness of the routes
  • vzshaper checks if the shaper is enabled at the hardware node
  • kernel_modules checks if all the required kernel modules are loaded in the kernel
  • masquerading checks network masquerading rules
  • ipv6 (for the Linux-based nodes only) checks IPv6 configuration
  • timezone checks if the hardware node’s timezone is UTC (and corrects this if it isn’t)
  • pva_agent checks Virtuozzo agent availability and configuration
  • vzsettings checks Virtuozzo configuration on the hardware node
  • docker (for the Linux-based nodes only) checks if the hardware node is connected to the shared storage for Docker templates and corrects this if it isn’t
  • repositories (for the Windows-based nodes only) checks if the hardware node connects to the correct Spacewalk channel. During the platform updates, it also changes the channel to the correct one

The results of these checkups can be seen in the State tab.

If the hardware node is configured correctly, all the modules will be marked with the green OK label. However, if any configuration tests failed, the corresponding hardware node’s line will be marked in red. You can hover over it to see the message, which explains the issue (for example, “module_name is not enabled”). Also, the same error message can be seen within the State tab as a value for the problematic module.

Add a New Hardware Node

You can perform a new hardware node addition to the Platform in automatic mode. Herewith, for the Virtuozzo Storage node installation, some extra pre-configurations are required. So, in case you need any assistance with this, please contact the Jelastic team.

At the top of the Hardware Nodes section, click Add and select the preferable option:
  • SSH Command - run the provided script on your server to automatically fetch required information and add the node
  • UI Wizard - add a node with the possibility to manually define all options using UI form

SSH Command

Within the opened window, use the Hardnode Group drop-down list (automatically hidden if there is just one hardware node group on the platform) to select where to include your new server. Next, copy the provided command.

add hardware node via ssh command
Note: During the command generation, the appropriate access token is added for the account (remains valid for 2 days).
Run the provided command on your server, and the node will be included into the cluster automatically (the script fetches all the required information by itself). Herewith, for the advanced installation with a possibility to manually redefine options, it is recommended to initiate addition via UI wizard.

UI Wizard

Note: In the case of hardware node reinstallation with the same hostname, you need to perform the following additional steps:
  • locate the jelastic puppet server UUID
    prlctl list | grep puppet
  • access the container
    prlctl enter {UUID}
  • get a list of all nodes
    puppet cert list -a
  • remove the registration of the appropriate hardware node
    puppet cert clean {hardware_node_name}

If you get an error during the “Run puppet agent” installation step, you need to validate the certificate by executing the next command on the above-mentioned container:

puppet cert sign {hardware_node_name}
Proceed by clicking Resume Installation in JCA.
1. In the Add Hardware Node dialog, provide the following fields:
  • Display Name - any preferable name for your hardware
    Note: On the platform with Jelastic version prior to the 5.6, a fully qualified domain name of the node should be specified (i.e. Display Name should be obligatorily defined as a value returned by the hostname command executed in the hardware node’s terminal).
  • IP Address - LAN of added hardware node with SSH protocol port after the colons “:” mark (22 by default)
  • Credentials - section to specify administrator account’s Login & Password, required for authentication
  • Hardnode Group - select a set of hardware from the list of available ones
  • Status - a status the added hardware node will get after installation (ACTIVE, BROKEN, INFRASTRUCTURE_NODE or MAINTENANCE)
  • Virtualization - choose a type of container-based virtualization solution to be used on the hardware node (VZ-6, VZ-7, or PVC for Windows hosting)
  • Install VZ - checkbox, which defines Virtuozzo software redeployment to the latest version supported on the hardware (reinstallation of VZ7 is supported since the Jelastic 5.5 version)

    Note: For this option to work, you need to correctly specify the Virtualization setting:

    • VZ7 for the hardware based on CentOS 7 or Virtuozzo 7
    • VZ6 for the hardware based on CentOS 6 or Virtuozzo 6

    Be aware that all of the data on your hardware node will be permanently lost.

    add hardware node install vz option
  • Advanced Settings
    • Use as Docker host - marks the current hardware node as a Docker host for the region
    • Skip configuration - allows to instantly add a node, skipping the configurations described in the next step (can be required for reinstallation of a node after being removed with the force flag)
    • Use L2 overlay network - automatically configures L2 overlay network for the platform on the specified IP address (using the Open vSwitch technology)

      Note: Tick this option only in case the L2 overlay network is not configured for the platform yet.


add hardware node via ui wizard

Click Add to proceed.

2. The node will appear in the list with INSTALLING status. Upon its selection, you can track the process within the corresponding Installation tab to the right.

hardware node installation section
Here, you can see the following information:
  • Start Time - actions’ initiation time
  • Action - a particular operation, required for hardware node installation
  • Status - the performed action result (OK, WARNING, or ERROR)

Note: If an issue occurs during installation, the corresponding step will be marked with either warning or error status within the list. The first result corresponds to some non-critical failure so that the installation will proceed further, whilst the second one will cause the process interruption.

hardware node installation error

If the installation fails, you’ll get the corresponding notification at the bottom of the installation section and email with the problem description. Also, you can hover over error string to get additional info.

As soon as you fix the problem, the process can be continued with the appropriate Resume Installation button at the top pane (not available if Install VZ was selected).

The installation process may take up to an hour, so please be patient. You’ll be informed about its completion via email.

Note: If new hardware is added into the region with the VZ6 node used as Docker host, you need to additionally remove docker-ce and install docker-engine instead:
  • access the newly added node and perform the following
    yum -y remove docker-ce
  • add docker-ce to the list of exceptions for yum
    echo 'exclude=docker-ce' >> /etc/yum.conf
  • install docker-engine
    yum install -y docker-engine

Modify Hardware Node

1. Select the required hardware node and click Edit at the toolbar above (or double-click the appropriate record).

edit hardware node button

2. Make the required changes in the opened Edit Hardware Node dialog.

It’s possible to change the main parameters you’ve previously stated in the Add Hardware Node window: Display Name, IP Address with SSH port, Hardnode Group, and Status.

edit hardware node dialog
Additionally, you can tick:
  • Edit Credentials - provides a new Login/Password for accessing hardware node

    Note: This operation is obligatory required after you’ve changed credentials on your hardware node to allow the platform to continue working with the node (to gather and process data, display statistics, etc).

    Herewith, an attempt to provide new credentials without updating them on the hardware node beforehand results in the appropriate error.
  • Advanced Settings - allows configuring L2 overlay network for the node and using this hardware server as Docker host for the nodes group

3. Once all the required changes are made, click Save at the bottom of the frame and the node settings will be modified.

Migrate/Evacuate Containers

The containers can be transferred to another hardware node if it is needed. You can move them one-by-one (migrate) or all containers at once (evacuate).

Note: The online migration is recommended whenever possible (as it applies no downtime on the transferred containers). If this option is not available (e.g. due to Virtuozzo infrastructure specifics or significant differences in the hardware CPU capabilities of the source and target hardware node), the offline migration can be used.

Containers Migration

1. Navigate to the Cluster > Hardware Nodes section and select the appropriate node.

2. Within the right-hand panel, switch to the Containers tab to see the list of containers, which are located on this node.

migrate conteiners to another node

Select the ones (use the Shift and Ctrl buttons for multiple records selection), which should be transferred, and click Migrate.

3. In the opened dialog box, specify the destination hardware Node and preferable Migration Mode (either online or offline).

containers migration dialog
Confirm an action with the Migrate button.

Containers Evacuation

1. In the Cluster > Hardware Nodes section, select a particular node and click Evacuate Containers.

evacuate containers from node
2. The appeared Evacuate Containers dialog allows you to see the list of active hardware sets, which can be used as target nodes for evacuation.

evacuate containers dialog
Here, you can:
  • proceed without selecting any hardware node - the target node is defined automatically due to the current load and allocation of the horizontally scaled components in environments (ensuring high availability)
  • choose a single node - evacuation is performed to this node only
  • select multiple nodes - distribution of containers is performed similarly to the first option but between the ticked hardware nodes only

Click Evacuate to proceed.

3. Navigate to the Evacuation State tab in the right panel to see the current state of the process. If needed, you can Stop the evacuation with the same-named button at the tools panel. Herewith, the already started processes of containers’ migration will be completed, but the remaining containers won’t be migrated.

Note: If any instance cannot be evacuated (e.g. due to occupation by some container operation) it will be skipped, but the process of evacuation will proceed. However, you will get the EVACUATION FAILED status (even if just one container was not migrated). The skipped containers can be migrated manually using the steps below.

4. After evacuation, you should perform an obligatory check that no containers are left on the node, using the next command:

prlctl list -a

listing containers on hardware node

Tip: Containers with "pool" in the Hostname can be ignored.

5. Ensure that no infrastructure containers (e.g. shared load balancers, docker host, uploader or any shared storages) are located on the node. If there are any, migrate them using the following command:

prlctl migrate

Tip: You can get all of the possible command options by running "man prlctl" in the console.

6. For other containers you should check the existence of the appropriate environments:
  • copy the domain name (e.g. env-1874316)
  • navigate to the JCA > Cluster > Environments section
  • enter the domain name to the Domain or Alias field and click Search

jca search environment by domain name
If such an environment exists, escalate this issue to Jelastic.

Remove Hardware Node

If you do decide to remove a particular hardware node from your Jelastic cluster, follow the instruction below.

Note: For the in-depth guide on user hardware node decommission, refer to the linked operations guide.

1. It is highly recommended to evacuate all the containers from such a hardware node before proceeding further.

2. Select the hardware node, which you want to Remove and click the appropriate button at the tools panel.

remove hardware node button
3. Confirm your decision by selecting OK in the appeared pop-up window.

remove hardware node confirmation

Tip: If there are containers on the hardware node, you need to confirm removal by ticking the Force checkbox.

4. After this operation, you should update the list to see the appropriate changes.

refresh hardware nodes list
Click the Refresh button to do it instantly or wait for the regular (every 20 seconds) automatic update.