Skip to main content

Kubernetes Start-Up Guide

Introduction​

This guide provides an example setup to get started with Kubernetes 1.33 for Universal Automation Center. It describes how to configure a single namespace with the following:

  • 2 Universal Controller pods in High Availability
  • 1 Universal Agent pod running OMS
  • 2 Universal Agent pods running UAG
  • 1 non-Kubernetes Universal Agent running UAG

This diagram shows 2 Kubernetes namespaces, "Prod" and "Test", to demonstrate that multiple namespaces can operate in the same cluster. Throughout this document we only use the "Prod" namespace.

Terminology​

Throughout this document we use a set of terms specific to Kubernetes. Other container platforms such as OpenShift may use different terms for the same resources. This table provides a brief overview of each term.

Term

Definition

Cluster

A Cluster refers to all resources or activities in the Kubernetes environment, distinct from any other activities running on separate hardware or a separate cloud environment.

Node

A physical (or virtual) machine that runs the Kubernetes cluster. The pods run in a containerized layer across the nodes.

Deployment

A Deployment handles the process of rolling out updates to applications and ensuring the correct number of pods are running.

Pod

A pod typically comprises one application, running on one or more container(s). It is the smallest compute unit. The Universal Controller runs in a pod, and the Universal Agent runs in a different pod.

Image

The blueprint for a pod, containing all of the packages needed for the application.

Service

A service allows traffic to be load balanced into a set of pods. A service provides an IP address for a pod (or set of pods) inside the cluster. Other pods in the cluster can communicate via a service.

Ingress

An ingress exposes a service outside the cluster. This allows external clients to connect to pods inside the cluster, such as the UI for the Universal Controller or the OMS port for the Universal Agent.

Cluster Configuration​

Installing the Kubernetes cluster is beyond the scope of this document. The process varies depending on the platform and resource requirements, so follow the Kubernetes Documentation for your use case. We use the following components for this guide:

warning

For Microsoft Azure Kubernetes Service (AKS) clusters, the default ingress controller "Application Gateway Ingress Controller" is not supported for the configuration described in this document. This ingress controller does not allow configuring TLS Passthrough on an ingress, which is required for connecting external agents to OMS pods, which is described in more detail below. We recommend using HAProxy instead.

The HAProxy load balancer timeout configuration needs to be updated to be compatible with the Heartbeat Interval of the Universal Agents. The timeouts must be longer than the Agent Heartbeat interval, otherwise the agents will continually disconnect and reconnect to OMS as the proxy timeout elapses.

In HAProxy's configuration file (located at /etc/haproxy/haproxy.cfg) set the value in the ingress-router section:

listen ingress-router-443
  bind *:443
  mode tcp
  balance source
  timeout server 150s
  timeout client 150s

In this example we use 150 seconds, as the default Heartbeat Interval is 120 seconds. If you change the heartbeat of any agent in the cluster to a longer duration the proxy configuration must be updated accordingly.

Other load balancers will need a similar configuration. Once an external UAG is connected to an OMS pod, the configuration can be verified by waiting a few minutes to ensure they do not disconnect.

Creating Resources​

Throughout this document we define resources as YAML. There are two ways to create resources in Kubernetes:

Once logged in (see Kubectl CLI) you can run kubectl create -f FILENAME.yml to create resources. The format of the YAML file is the same as the Import YAML screen.

Kubectl CLI​

The kubectl command line utility allows you to manage cluster operations and applications from a terminal. Follow the steps for your Operating System to install it.

The install process will provide a kubeconfig file to run commands as the kubeadmin user. With this file in place you can run commands to view and edit cluster resources.

# kubectl get deployments
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
agent1   0/1     1            0           2s

Additional information on kubectl syntax can be found in the Kubernetes Documentation.

Pushing Images to the Cluster​

In order to deploy Universal Controller or Universal Agent pods you must obtain the corresponding Docker images and add them the cluster's Image Registry. Kubernetes does come with a registry by default, so one must be installed. The Kubernetes Documentation does not cover this, but this Medium Article makes the process very clear.

Universal Controller images are available in the Customer Portal. A customer username and password (provided by Stonebranch, Inc.) are required to access the Customer Portal.

Universal Agent images are also available in the Customer Portal. Alternatively, you can pull an image directly from the Red Hat Image Catalog using the instructions on that page.

We will describe the procedure for uploading an image from the Customer Portal, since it can apply to both Agent and Controller.

To push the images, you must use either Docker or Skopeo (a utility from Red Hat for managing Docker images).

The steps below describe the process for a Universal Controller image, but the same steps apply for the Universal Agent images as well. There are two methods for pushing images to the cluster:

Skopeo

Skopeo uses a single command to import, tag, and push the image.

skopeo copy \
  --dest-creds=registryUser:registryPassword \
  docker-archive:./universal-controller-7.8.0.0-build.95-docker-rhel-ubi9-x86_64.tar.bz2 \
  docker://registry.apps.example.com/prod/universal-controller:7.8.0.0

In the command we provide the credentials as a username and password.

The source is the local image file, specified as a docker archive.

The destination uses the registry Ingress, the namespace, the image name, and the desired remote tag.

With the image pushed you can verify it exists in the cluster by running curl on the registry:

# curl https://registry.apps.example.com/v2/sqa/universal-controller/tags/list -u registryUser:registryPassword
{"name":"sqa/universal-controller","tags":[7.8.0.0-ubi9"]}

Universal Controller Setup​

We will deploy two Universal Controllers in a High Availability configuration using a single external MySQL database. The Kubernetes controllers will use Deployment API Objects. This will allow you to modify the configuration (such as the Environment Variables), and Kubernetes will rollout new pods with the latest configuration. This is not possible with a standalone pod. Changing the configuration is not allowed, so you would have to delete the existing pod and define a new one.

Database​

In this guide we use a MySQL database that resides in a Virtual Machine outside of the Kubernetes cluster. The database configuration is beyond the scope of this document. For more information see Installing a Database.

We use mysql.example.com for the hostname of our example database.

Database Secret​

The database credentials must be provided in the Controller Deployment. The password should not be stored in plaintext, so you must create a Secret that the Deployment can reference. The following YAML creates the secret.

Database Secret
apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
  namespace: prod
stringData:
  password: super-secret-p4ssw0rd

Once created you can access the password field of the Secret in other resources.

Controller​

For the Universal Controllers we will create a Deployment with two pods, a Service to load balance between the pods, and an Ingress for exposing the Service outside of the cluster.

info

Throughout this section any certificates are truncated and invalid. Valid certificates must be supplied for the connections to work correctly.

Controller Deployment​

The following YAML creates a Deployment containing two Universal Controllers.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: uc-east-1
  namespace: prod
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  selector:
    matchLabels:
      app: uc-east-1
  template:
    metadata:
      labels:
        app: uc-east-1
    spec:
      enableServiceLinks: false
      containers:
        - name: uc-container
          image: registry.apps.example.com/prod/universal-controller:7.8.0.0-ubi9
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: "4"
              memory: 8Gi
            requests:
              cpu: 500m
              memory: 2000Mi
          env:
            - name: UC_DB_RDBMS
              value: mysql
            - name: UC_DB_URL
              value: jdbc:mysql://mysql.example.com:3306/
            - name: UC_DB_USER
              value: root
            - name: UC_DB_NAME
              value: UCEAST1
            - name: UC_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: password
            - name: UC_SYSTEM__IDENTIFIER
              value: Prod Controller 1
            - name: UC_LICENSE
              value: <license key>
            - name: UC_NODE_TRANSIENT
              value: 'true'
            - name: CATALINA_OPTS
              value: '-XX:MaxRAMFraction=2'

Memory Configuration​

Tomcat's memory configuration will be handled by an environment variable, either CATALINA_OPTS or JAVA_OPTS. Both are used when starting Tomcat, but only JAVA_OPTS is used when stopping Tomcat. For the memory configuration specifically, either variable can be used.

The default behavior for Tomcat is to set the maximum heap size to 25% of the available system memory. In Kubernetes the available memory is determined in one of two ways, depending on the Pod's configuration.

If no resource limits are defined for the Pod the available memory will be the total memory of the Node. This can allow the controller to use more memory than expected.

If resource limits are defined Tomcat will interpret the memory limit as the total system memory. This is because the Java option -XX:+UseContainerSupport is set by default, so Java is aware of the resource limit, treating it like the total memory of a Virtual Machine. This means Tomcat will be limited to 25% of the resource limit.

With this in mind there are 2 ways to configure the memory for Tomcat:

Dynamically set Max Heap Size

The value for the Max Heap Size can be set dynamically based on the Pod's resources.

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: uc-container
          resources:
            limits:
              memory: 8Gi
            requests:
              memory: 2000Mi
          env:
            - name: CATALINA_OPTS
              value: '-XX:MaxRAMFraction=2'

Here we set the Pod's memory limit to 8GB, which is enforced by Kubernetes.

Then we define the fraction of the memory limit to allocate to Tomcat using the -XX:MaxRAMFraction value, which defaults to 4.

This will give us a Max Heap Size of 4096MB, derived from memory limit (8GB) divided by MaxRAMFraction (2).

The dynamic option is preferred. It allows the heap size to scale with the memory limit. We also recommend defining resources on the Pods, so it makes sense to utilize those values for Tomcat's configuration.The Kubernetes Documentation doesn't cover java specifically, but helpful information can be found in the OpenShift Documentation that also applies to Kubernetes.

High Availability and Horizontal Scaling​

This example uses a standard High Availability configuration with a single Active and Passive pod. This has several benefits.

  • If Tomcat crashes, a new pod will replace the failed one and the application will start in around 30 seconds. Often this is fast enough for the failed Active node to remain designated as the Active node in the cluster when the new pod has started.
  • If the application becomes unresponsive, the Passive node will become Active, and the troublesome pod can be redeployed (for example, by deleting the pod).
  • REST API calls and browser sessions can be load balanced between pods, transparent to the user.

If additional pods are needed for horizontal scaling, the number of Replicas can be increased using kubectl:

# kubectl scale deployment/uc-east-1 --replicas=3
deployment.apps/uc-east-1 scaled

This will deploy an additional pod, and shortly a new Cluster Node will appear in the controller's UI, ready to accept traffic from the Service.

This process can also be automated using a Horizontal Pod Autoscaler. This will allow the Deployment to dynamically set the number of replicas based on the memory or CPU usage of the pods, either using a static value or based on the pod's resource definition.

Horizontal Pod Autoscaler

The following YAML will create an Autoscaler resource.

Autoscaler
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: controller-autoscaler
  namespace: prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: uc-east-1
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: AverageValue
          averageValue: 500m

This autoscaler targets our controller Deployment (uc-east-1).

We set the minimum and maximum number of replicas based on available resources.

The Resource section is where we define the criteria for the scaling behavior. In this example we use the CPU utilization, specifically the average utilization for all the pods in the Deployment. The value is based on CPU cores (where 500m is 5/1000s of a core).

A similar approach can be used for memory consumption. Modify the metrics field in the definition as follows:

Memory Consumption Autoscaler
kind: HorizontalPodAutoscaler
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: AverageValue
          averageValue: 8000Mi

This will scale up the replicas when the average pod memory consumption exceeds 8GB, and scale down as the value drops below 8GB.

The other approach for determining scaling requirements is based on the Deployment's resource definition.

Memory Utilization Autoscaler
kind: HorizontalPodAutoscaler
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 75

The Utilization is the ratio between the current resource usage and Request value. In this configuration it will scale the Deployment to keep the pods' memory consumption at 6GB (75% of the 8GB resource limit defined in the Deployment).

The same syntax is used for CPU utilization.

More information on resources and their use in autoscaling can be found in the Kubernetes Documentation.

Health Probes (support added in 7.9.0.0)​

There is an alternative for those who don't want to run multiple controllers but still want the resilience of a High Availability setup. Kubernetes allows pods to be configured with various Health Probes. In UAC 7.9.0.0 a new API endpoint was added to the Controller for these probes, and requires no authentication, so they will work on an initial deployment before the admin password is set or users are created.

First a Startup Probe checks when the API endpoint is available and returning 200 responses, indicating that the Controller has started. This will signal to Kubernetes that the pod is in Ready state. After this a Liveness Probe will run on an interval, checking this API endpoint. If the Controller enters an invalid state (for example, if it cannot read/write to the database), this API will start returning 503 responses. Based on the Liveness Probe configuration, after this API has failed the required number of times Kubernetes will restart the container. This is the same scenario that would trigger an HA failover, but now we use native Kubernetes functionality to failover with a single Controller instance. The following is a simplified version of the Controller Deployment with the probes added.

Controller Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: uc-east-1
  namespace: prod
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  selector:
    matchLabels:
      app: uc-east-1
  template:
    metadata:
      labels:
        app: uc-east-1
    spec:
      enableServiceLinks: false
      containers:
        - name: uc-container
          image: registry.apps.example.com/prod/universal-controller:7.8.0.0-ubi9
          imagePullPolicy: IfNotPresent
          env:
            - name: UC_DB_RDBMS
              value: mysql
            - name: UC_DB_URL
              value: jdbc:mysql://mysql.example.com:3306/
            - name: UC_DB_USER
              value: root
            - name: UC_DB_NAME
              value: UCEAST1
            - name: UC_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: password
         startupProbe:
           httpGet:
             path: /uc/resources/clusternode/healthcheck
             port: 8080
           initialDelaySeconds: 120
           timeoutSeconds: 5
           failureThreshold: 10
           periodSeconds: 30
         livenessProbe:
           httpGet:
             path: /uc/resources/clusternode/healthcheck
             port: 8080
           initialDelaySeconds: 5
           timeoutSeconds: 5
           failureThreshold: 3

The Startup Probe waits 120 seconds after startup before beginning (initialDelaySeconds). This should give the Controller time to startup and connect to the database. Then it will run an HTTP GET request on the /uc/resources/clusternode/healthcheck endpoint every 30 seconds (periodSeconds). If the request times out after 5 seconds (timeoutSeconds) or returns a non-200 response code it will fail. After 10 failures (failureThreshold) it will restart the container. Otherwise it will mark the pod as Ready. (These options might need to be adjusted if the probe fails before the database is bootstrapped on a fresh deployment.)

The Liveness Probe has the same options, but it will not begin executing until after the Startup Probe succeeds. This probe's configuration will affect the failover. In this case we consider the Controller to be unhealthy after 3 failed attempts with a 5 second timeout and the default period (10 seconds).

When a probe fails it will generate an Event, which can be viewed in the kubectl describe pod output. After the failure condition is met another Event shows that the container is restarted.

# kubectl describe pod/uc-east-1-5c859c6cf6-vg4fq -n prod
Name: uc-east-1-5c859c6cf6-vg4fq
Namespace: prod
...
Events:
  Type      Reason       Age                  From       Message
  ----      ------       ----                 ----       -------
  Normal    Created      5m17s                kubelet    Created container: uc-east-1-container
  Normal    Started      5m17s                kubelet    Started container uc-east-1-container
  Warning   Unhealthy    27s (x3 over 47s)    kubelet    Liveness probe failed: HTTP probe failed with statuscode: 503
  Normal    Killing      27s                  kubelet    Container uc-east-1-container failed liveness probe, will be restarted

Controller Service​

With the pods deployed, we need to create a service to allow clients to communicate with the Controller. We will create a service with the following YAML.

Controller Service
apiVersion: v1
kind: Service
metadata:
  name: uc-east-1-service
  namespace: prod
spec:
  selector:
    app: uc-east-1
  ports:
    - name: http
      protocol: TCP
      port: 8080

The selector determines which pods to connect to based on a label. In this case we use the label app with the value uc-east-1 to connect to our Controller pods.

Here we use the default type (ClusterIP) which provides an internal IP address for load balancing to endpoints. These endpoints are determined by the selector. Any traffic routed to the service will be load balanced between our two (or more) controller pods.

The port defines which port on the pod the service should connect to. Here we defined port 8080 for http traffic connecting to Tomcat.

Controller Ingress​

To connect to the service from outside the cluster we must expose it with an Ingress using the following YAML.

Controller Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: controller
  namespace: prod
spec:
  rules:
    - host: controller-prod.apps.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: uc-east-1-service
                port:
                  number: 8080

This creates a URL https://controller-prod.example.com that should be accessible from outside the cluster.

This configuration will use Edge TLS termination. The certificates of the Ingress Controller will be used for the controller URL. See Controller TLS Configuration for more details.

Edge termination encrypts traffic between clients and the Ingress Controller, but traffic from the Ingress Controller and pod will be unencrypted, so we must use the http port of the pod.

Controller TLS Configuration​

As stated in the last section, the Ingress will use Edge TLS termination. The Kubernetes Documentation describes the configuration in more detail.

The above example uses the certificates of the Ingress Controller for our URL, since no TLS configuration is present in the Ingress definition.

It is also possible to use different certificates for the Ingress. These are provided via a Secret. With the certificate files present in the current directory, create the Secret using kubectl:

Ingress Secret
kubectl create secret tls controller-ingress-certs --namespace prod --key server.key --cert server.crt

Now the Secret can be referenced in the Ingress definition. The hostname defined in the tls section must match the host defined in the rules section.

Ingress With Certificates
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: controller
  namespace: prod
spec:
  tls:
    - hosts:
        - controller-prod.example.com
      secretName: controller-ingress-certs
  rules:
    - host: controller-prod.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: uc-east-1-service
                port:
                  number: 8080

Verification​

Once all of the resources are created we can access the UI using the Ingress. Make sure to append /uc to the end of the URL to access the Controller login page.

Set the admin password and login. We should see two Cluster Nodes (not to be confused with the Kubernetes cluster's nodes).

We can verify the Service is correctly load balancing by using the Retrieve System Details API, which shows the name of the controller pod where the request ran.

For example, two consecutive requests are routed to different pods:

Universal Agent Setup​

We will deploy 4 Universal Agents: 1 OMS pod, 2 UAG pods, and 1 external agent that connects into the cluster. The Kubernetes agents will also use Deployments to allow rolling out new pods with the latest configuration.

OMS Server​

OMS Deployment​

The following YAML creates a Deployment containing 1 Universal Agent for OMS.

OMS Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oms-east-1
  namespace: prod
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: oms-east-1
  template:
    metadata:
      labels:
        app: oms-east-1
    spec:
      enableServiceLinks: false
      containers:
        - name: oms-container
          image: registry.apps.example.com/prod/universal-agent:7.8.0.0-ubi9
          imagePullPolicy: IfNotPresent
          env:
            - name: OMS_PORT
              value: "7878"
            - name: OMSAUTOSTART
              value: "YES"
            - name: UAGAUTOSTART
              value: "NO"

OMS Service​

With the pod deployed we need to create a service to allow clients to communicate with the OMS. We will create a service with the following YAML.

OMS Service
apiVersion: v1
kind: Service
metadata:
  name: oms-east-1-service
  namespace: prod
spec:
  selector:
    app: oms-east-1
  ports:
    - name: oms
      protocol: TCP
      port: 7878

The selector determines which pods to connect to based on a label. In this case we use the label app with the value oms-east-1 to connect to our OMS pod.

The port defines which port on the pod the service should connect to, in this case the default OMS port.

Other pods in the cluster can interact with the service by name. For example, you can add this OMS in the Universal Controller using any of the following hostnames:

  • oms-east-1-service

  • oms-east-1-service.prod.svc


  • oms-east-1-service.prod.svc.cluster.local

OMS Ingress​

To connect to the service from outside the cluster we must expose it with an Ingress using the following YAML.

OMS Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oms-east-1-ingress
  namespace: prod
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
    - host: oms-east-1.apps.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: oms-east-1-service
                port:
                  name: oms

This creates a URL https://oms-east-1.example.com that is accessible from outside the cluster.

DNS should direct traffic at from *.apps.example.com to the cluster's nodes. From there the Ingress Controller directs traffic to the correct service based on the Ingress's hostname (eg, oms-east-1 will go to the oms-east-1-service selected in the definition) using TLS SNI headers.

This configuration uses TLS Passthrough. This sends encrypted traffic directly to the OMS, without any TLS Termination from the Ingress Controller. All of the certificates and keys are handled directly by OMS and clients, so none are provided on the Ingress definition.

TLS SNI​

In order for traffic to be routed to the correct service the client must use TLS SNI headers. This will tell the load balancer (HAProxy in this case) which Ingress the client is trying to access. For Universal Agents this requires setting the oms_servers value for UAG using the format PORT@HOSTNAME:SNI-HOSTNAME which for Kubernetes will be PORT@INGRESS:INGRESS. In the above example that would be 443@oms-east-1.apps.example.com:oms-east-1.apps.example.com. The port is 443 because the Ingress is exposed on the default HTTPS port. The SNI Hostname is the same as the Ingress name, so it is specified twice. With this configuration UAG will be able to connect into the cluster, through the Ingress, to the service, and into the pod on the OMS port.

UAG Agents​

Deployment​

The following YAML creates a Deployment containing 1 Universal Agent running UAG.

1st UAG Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: uag-east-1
  namespace: prod
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: uag-east-1
  template:
    metadata:
      labels:
        app: uag-east-1
    spec:
      hostname: uag-east-1
      enableServiceLinks: false
      containers:
        - name: uag-container
          image: registry.apps.example.com/prod/universal-agent:7.8.0.0-ubi9
          imagePullPolicy: IfNotPresent
          env:
            - name: UAGOMSSERVERS
              value: oms-east-1-service
            - name: OMSAUTOSTART
              value: "NO"
            - name: UAGAUTOSTART
              value: "YES"
            - name: UAGNETNAME
              value: AGENT1

The majority of this is the same as the OMS Deployment.

The Agent will connect to the OMS we just deployed by connecting to its service (specified in the UAGOMSSERVERS configuration). Since it is inside the cluster it does not need to connect to the Ingress using TLS SNI.

We will set OMS to not start, UAG to start, and set a desired netname.

info

Note the hostname field (spec.template.spec.hostname). When not specified the container's hostname will be set to the Pod name. In a Deployment the Pod's name will include a randomly generated string. This will affect the name of the Agent as it appears in the Controller. Depending on the use case this might not be desired. Statically defining the hostname will ensure that when the Pod is recreated (for example, after making a configuration change, or when the Pod is restarted) the same Agent record will be used in the Controller. If the hostname is not specified a new Agent record will appear with the new Pod name included in the agent's name.

Another alternative is to use Transient Agents. By setting the environment variable UAGTRANSIENT: 'YES' the Agent record will be deleted from the Controller when the Agent stops. So when a Pod is restarted the old Agent (with the old Pod name) will be removed from the Controller, and a new Agent (with the new Pod name) will be created. It should be understood that Transient Agents should never be specified directly in any task definition. They are designed to accept work via an Agent Cluster, so when configuring an agent to be transient we must ensure that the agent registers with one or more Agent Clusters. This can be configured using the UAGAGENTCLUSTERS environment variable.

Next create a Deployment for the second UAG pod.

2nd UAG Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: uag-east-2
  namespace: prod
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: uag-east-2
  template:
    metadata:
      labels:
        app: uag-east-2
    spec:
      hostname: uag-east-2
      enableServiceLinks: false
      containers:
        - name: uag-container
          image: registry.apps.example.com/prod/universal-agent:7.8.0.0-ubi9
          imagePullPolicy: IfNotPresent
          env:
            - name: UAGOMSSERVERS
              value: oms-east-1-service
            - name: OMSAUTOSTART
              value: "NO"
            - name: UAGAUTOSTART
              value: "YES"
            - name: UAGNETNAME
              value: AGENT2
            - name: UBRMAXSSLPROTOCOL
              value: TLS1_3
            - name: UCMDMAXSSLPROTOCOL
              value: TLS1_3
            - name: UDMMAXSSLPROTOCOL
              value: TLS1_3

To create a second Agent, we only need to modify a few fields:

  • Deployment name
  • Selector labels
  • Deployment labels
  • Container hostname
  • Agent netname

To demonstrate other configuration options this Agent will use TLS1.3 for all connections between UCMD/UDM and UBroker.

info

Both Agents (and the external agent added in the next section) will connect to the OMS pod. As the number of Agents increases it is important to scale the number of OMS Servers accordingly. This will be case-specific, so we can't recommend any particular scale factor, but it is important to keep in mind.

With the OMS already added in the Controller when the UAG pods start we will see them appear under the OMS Server's agents.

The pod log shows that UAG successfully got a license from the controller.

# kubectl logs deployment/uag-east-2 -n prod
...
INFO [cbHello] (wmnmtsk.c:2248) OPSTRC002I License info: CUSTOMER: *Stonebranch Internal*, REGISTERED: YES, EXPIREDATE: 1798761599999, DISTAGENTS: 100, ZOSAGENTS: 100, LICZOSSECAGENTS: 0, IBMIAGENTS: 100, USAP: -1, UPPS: -1, SOA: 31
2025.09.30 14.47.00.105 UNV5340I [1000000001] License Information Received: CUSTOMER *Stonebranch Internal*, EXPIREDATE 1798761599 (Thu Dec 31 23:59:59 2026), DISTRIBUTED 100, ZOS 100, IBMI 100, USAP 4294967295, UPPS 4294967295, SOA 31, REGISTERED YES
INFO [cbHello] (wmnmtsk.c:2297) OPSTRC002I Successfully sent license to the broker

UAG Agent (External)​

We will also create another Agent running outside the cluster (it could be on-premises or in the cloud) that will connect to the Kubernetes OMS and Universal Controller. This will demonstrate the TLS SNI functionality.

This Agent can run on any platform, as long as it has network access to the Kubernetes cluster and is at least version 7.8.0.0 (to support TLS SNI).

The Agent should be installed in the normal way (described in the Universal Agent documentation). We only need to configure the UAG option for oms_servers. Depending on the platform this can be done during installation, otherwise the configuration file will need to be updated.

Platform

Configuration

Linux/Unix

During installation use the --oms_servers option

Windows (system mode)

During installation use the OMS_SERVERS option (or corresponding field in the Graphical Installer)

Windows (user mode)

During installation use the -oms_servers option (or corresponding field in the Graphical Installer)

IBM i

After installation modify the UAG configuration in UNVPRD780/UNVCONF(UAGS)

z/OS

After installation modify the UAG configuration in UNVCONF(UAGCFG00)

The value should be 443@oms-east-1.apps.example.com:oms-east-1.apps.example.com (the value from the Ingress definition). This specifies the TLS SNI hostname that allows UAG to connect to the OMS inside the cluster.

In this example we do a clean install on a CentOS 8 host. We can see UAG connects to the OMS and gets a license from the controller.

Then the Agent appears in the Controller under the OMS Server, alongside the Kubernetes agents.

UDM (support added in 7.9.0.0)​

UDM requires connecting to a remote agent's UBroker port for transferring files between agents. If one of these agents is in a Kubernetes cluster this connection needs to go through the Ingress Controller to connect to the pod, which requires the same TLS SNI headers added for OMS. This section describes how to configure a "UAG" pod to accept UDM connections from external agents.

UAG Service​

First, we need to create a service and ingress for connecting to the agent's broker port.

UAG Service
apiVersion: v1
kind: Service
metadata:
  name: uag-east-1-service
  namespace: prod
spec:
  selector:
    app: uag-east-1
  ports:
    - name: ubroker
      protocol: TCP
      port: 7887

The selector determines which pods to connect to based on a label. In this case, we use the label app with the value uag-east-1 to connect to our first UAG pod.

The port defines which port on the pod the service should connect to, in this case the default UBroker port.

UAG Ingress​

Next, we need to create an Ingress to access this service from outside the cluster.

UAG Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: uag-east-1-ingress
  namespace: prod
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
    - host: uag-east-1-prod.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: uag-east-1-service
                port:
                  name: ubroker

This creates a URL https://uag-east-1-prod.example.com that is accessible from outside the cluster.

This configuration uses TLS Passthrough. This sends encrypted traffic directly to UBroker, without any TLS Termination from the Ingress Controller. All of the certificates and keys are handled directly by UBroker and clients, so none are provided on the Ingress definition.

Verification​

With the service and ingress created we can now run UDM scripts from any agent (internal or external to the cluster) connecting to our UAG pod. The only change needed to connect to a Kubernetes agent compared with a normal agent is in the open command. Here is an example of normal script syntax to copy a single file from a remote agent to a local one:

UDM Script
open dst=local src=agent1.example.com port=7887 xfile=credentials.txt
copy src=/tmp/data.txt dst=/home/ubroker/data.txt
close

To run this same script for a Kubernetes agent we only need to change the open command:

UDM Script with SNI
open dst=local src=uag-east-1-prod.example.com port=443 sni=uag-east-1-prod.example.com
copy src=/tmp/data.txt dst=/home/ubroker/data.txt
close

Here we use the Ingress name for the target (src) and sni attribute (and the port must be 443 to hit the https endpoint of the Ingress Controller). As with OMS, this sets the TLS SNI headers which tells the Ingress Controller which Ingress it's using, and therefore which service to connect to.

File Transfer Tasks​

This same functionality can be achieved for File Transfer tasks in the controller. When using UDM in a File Transfer task the task will generate a UDM script and the agent will execute it. Normally the script will use the IP address reported from the agent for the open command. This will not work on Kubernetes, as the pod IP is a cluster-specific IP range that external agents cannot access. The script will also not include any SNI information. To get the correct connection information for Kubernetes agents we will use the UAG config option tls_sni_hostname (or environment variable UAGTLSSNIHOSTNAME). This tells the controller the hostname, port, and SNI hostname to use in UDM scripts.

Update the 1st UAG Deployment to use the UAGTLSSNIHOSTNAME variable with the values from the newly created ingress:

UAG Deployment with TLS SNI
apiVersion: apps/v1
kind: Deployment
metadata:
  name: uag-east-1
  namespace: prod
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: uag-east-1
  template:
    metadata:
      labels:
        app: uag-east-1
    spec:
      hostname: uag-east-1
      enableServiceLinks: false
      containers:
        - name: uag-container
          image: registry.apps.example.com/prod/universal-agent:7.8.0.0-ubi9
          imagePullPolicy: IfNotPresent
          env:
            - name: UAGOMSSERVERS
              value: oms-east-1-service
            - name: OMSAUTOSTART
              value: "NO"
            - name: UAGAUTOSTART
              value: "YES"
            - name: UAGNETNAME
              value: AGENT1
            - name: UAGTLSSNIHOSTNAME
              value: 443@uag-east-1-prod.example.com:uag-east-1-prod.example.com

This information will then appear in the controller, in the Agents page:

Running a File Transfer task using this agent will reveal how the correct script content is generated:

info

The tls_sni_hostname option should not be specified for external agents. This option is specifically for containerized agents for use with File Transfer tasks. Using the option for external agents can cause incorrect File Transfer behavior.

USAP​

In order to use the Universal Connector for SAP the Agent must have access to the SAP NW (NetWeaver) RFC libraries. These are not provided in the Universal Agent image and must be obtained from SAP. More information can be found in the Universal Agent documentation.

Once the libraries are downloaded they must be added to the container. There are two methods to accomplish this: create a new agent image with the libraries included, or mount a volume where the libraries are stored.

Create a new Image

One option is to build a new SAP-enabled image using the Universal Agent image as the source. With Docker installed follow these steps to create the image.

Step

Description

Copy

Copy the SAP-RFC libraries into a local directory: ./sap-libs

Create

Create a simple dockerfile

FROM universal-agent:7.8.0.0-ubi9            # name:tag of local base image
USER root                                    # root user required to create directories in /
RUN mkdir /sap                               # directory to store libraries
COPY sap-libs /sap/                          # copy local libraries into image
RUN chgrp -R 0 /sap && chmod -R g=u /sap     # set permissions for default user to access
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/sap    # set LD_LIBRARY_PATH to library location

Build

Build the new image using Docker

docker build -t universal-agent:7.8.0.0-ubi9-sap -f dockerfile .

This will create a new local image with the tag 7.8.0.0-ubi9-sap

Tag and Push

Add a remote tag and push the image to the cluster as described in Pushing Images to the Cluster

Pros and Cons​

There are pros and cons for each method. We describe a few here, but ultimately the best method depends on the individual use case.

Scenario

New Image

Persistent Volume

Red Hat Image Catalog

The image from the Red Hat catalog will need to be downloaded locally and modified.

Using the image from the Red Hat catalog requires no modification.

Updating Images

Changing source images requires building a new SAP image.

Updating to a new version (or a new image) requires no further changes.

Dependencies

No additional Kubernetes resources are required.

The Persistent Volume may impact which Node a Pod can run on, impacting load distribution.