Skip to content

Latest commit

 

History

History
52 lines (30 loc) · 3.76 KB

README.adoc

File metadata and controls

52 lines (30 loc) · 3.76 KB

Container Services Plugin

Table of Contents

The Container Services Plugin handles some of the quirks of running IIQ within a container or CaaS (container-as-a-service) environment, such as Rancher or ECS.

Container Setup Service

The Container Setup Service does some container-specific initialization on each host at startup. IIQ runs services very early on server startup, before the webapp is even available via HTTP, so this should happen within a few seconds.

Renaming

Containers run with dynamic hostnames, usually the random container ID. IIQ depends on a stable set of hostnames, e.g. for mapping which servers are task servers and which are request servers. IIQ will knowing only about the container hostname, creating a new Server object with that name.

On each IIQ node, the service will generate a new, predictable server name of the form <prefix>#, where the number is incremented from 1 to 100. The service will select the lowest numbered hostname that is not currently occupied by an active Server object.

If an existing, but inactive, Server already exists with a hostname, the old Server object will be retained and renamed, while the new Server (with the container name) will be deleted.

If no existing Server has a hostname, the new Server will be renamed to that value.

Additionally, the service will set the system property iiq.hostname so that the Util.getHostName() API returns the right value.

The default prefix is iiq, so your servers will be iiq1, iiq2, iiq3, etc. You can override the prefix in several places (trusted this order):

  1. The plugin settings,

  2. The ServiceDefinition for this service,

  3. The environment variable IIQ_PREFIX.

After rename setup

Since renaming the Server can mess with some settings, the following actions are taken after rename:

  1. An audit event of type ServerRename will be logged.

  2. If the ServiceDefinition has a list of monitoredApplications (list of names), those will be added to the Server configuration. This ensures that all servers have the same list of monitored applications.

  3. If the ServiceDefinition contains a Script attribute called threadScript, the Script will be invoked to get the number of task and request threads this server should run. The Script will receive a server (the Server object) and a type (a string, either requestThreads or taskThreads). Those values are added to the Server.

  4. Requests are labeled with the host running them. Any requests attached to the startup hostname (the container ID) will be renamed so that they refer to the new hostname.

  5. The Servicer is restarted so that services with host restrictions are properly assigned.

Shutdown handling

When the service is shut down, it will log a full thread dump and some basic memory statistics at WARN level. This may help track down the problem if containers are routinely failing their liveness or healthcheck probes.

If the service suspects that the entire appserver is shutting down (i.e. Tomcat is stopping), it will quickly flag the current Server as inactive, allowing the hostname to be reused immediately.

Orphaned request scanner service

The Orphaned Request Scanner Service checks every few minutes for background requests, including partitions, that are attached to servers that are no longer active. Those requests are reset using IIQ’s RequestProcessor API, the effects of which vary by request configuration.

In a container environment where server names are dynamic, this supplements IIQ’s Reanimator service, which only handles orphans attached to the current hostname.

Any requests that cannot be reset (e.g. because they are marked host specific) are never guaranteed to run again and will be deleted.

The scanner will begin scanning 90 seconds after startup and checks again every 120 seconds after that.