3.3.1   Hexapodsim¶

The athexapod application requires the use of a simulated hexapod low-level controller when running in simulation mode. This simulator (hexapodsim) is accessed by a specific IP address and port. The hexapodsim app uses a Service from the Kubernetes Service APIs to setup the port. Kubernetes conjoins that with the deployed pod IP in an environment variable: HEXAPOD_SIM_SERVICE_HOST. The ATHexapod CSC code uses that variable to set the proper connection information. The hexapodsim application has its own chart that uses Deployment from the Kubernetes Workloads API. This chart contains no OSPL features since the simulator does not require it. The use of the Deployment allows the application to be brought up with the auxtel collector app, but remain up if the CSCs are taken down since those are run as Jobs.

3.3.2   Header Service¶

Both the aheaderservice and ccheaderservice apps require the use of an Ingress and Service Kubernetes APIs. This is only necessary while the header services apps leverage an internal web service for header file exposition. Once the header service moves to using the S3 LFA, the extra APIs will be removed.

3.4.1   Cluster Configuration¶

The main values.yaml file looks like:

cluster-config:
  namespaces:
    - auxtel
    - eas
    - maintel
    - obssys
    - kafka-producers
    - ospl-daemon

This sets the namespaces for all sites. This configuration can be overridden on a per site basis, but it is not recommended for summit environment.

The site specific configuration files only needs to contain the deployEnv keyword. The values-ncsa-teststand.yaml is shown as an example.

deployEnv: ncsa-teststand

If you want to override the list of namespaces for a particular site, this is how it would be done for a site specific file. Also included in the example is how one would specify extra namespace secrets.

cluster-config:
  namespaces:
    - test1
    - myspace
    - home

deployEnv: tucson-teststand

extraSecrets:
  - name: key1
    namespaces:
      - test1
      - home
  - name: key2
    namespaces:
      - myspace
      - home

3.4.2   OSPL Configuration¶

This is the ospl-config directory within the ArgoCD repository. The all-site configuration in values.yaml looks like this.

ospl-config:
  namespaces:
    - auxtel
    - eas
    - maintel
    - obssys
    - kafka-producers
    - ospl-daemon
  networkInterface: net1
  shmemSize: 504857600
  maxSamplesWarnAt: 50000
  schedulingClass: Default
  schedulingPriority: 0
  monitorStackSize: 6000000
  waterMarksWhcHigh: 8MB
  deliveryQueueMaxSamples: 10000
  squashParticipants: true
  namespacePolicyAlignee: Lazy
  domainReportEnabled: false
  ddsi2TracingEnabled: false
  ddsi2TracingVerbosity: finer
  ddsi2TracingLogfile: stdout
  durabilityServiceTracingEnabled: false
  durabilityServiceTracingVerbosity: FINER
  durabilityServiceTracingLogfile: stdout

The list of namespaces MUST contain at least the same namespaces as cluster-config. The networkInterface is the name specified by the multus CNI and is the same for all sites that we currently deploy to. The rest of the configuration is meant for handling setup, services and features related to the shared memory configuration. If you want to adjust configuration parameters for testing without effecting other sites, a site specific configuration file can be used.

3.4.3   OSPL Daemon Configuration¶

The OSPL daemon configuration has a global values.yaml file that sets the namespace, OSPL log files and OSPL version for all sites. All other configuration should be handled in a site YAML configuration file. The an example configuration for the summit is shown below.

ospl-daemon:
  image:
    repository: ts-dockerhub.lsst.org/ospl-daemon
    tag: c0016
    pullPolicy: Always
    nexus3: nexus3-docker
  env:
    LSST_DDS_PARTITION_PREFIX: summit
  shmemDir: /run/ospl
  osplVersion: V6.10.5
  initContainer:
    repository: lsstit/ddsnet4u
    tag: latest
    pullPolicy: Always

3.4.4   Kafka Producer Configuration¶

The Kafka producer configuration has a global values.yaml file that sets the namespace, OSPL log files and OSPL version and producer CSC configuration for all sites. A snippet of the configuration is shown below.

kafka-producers:
  namespace: kafka-producers
  ...
  producers:
    auxtel:
      cscs: >-
        ATAOS
        ATDome
        ATDomeTrajectory
        ATHexapod
        ATPneumatics
        ATPtg
        ATMCS
    maintel:
      cscs: >-
        MTAOS
        Dome
        MTDomeTrajectory
        MTPtg
    ...

Each key under producers is the name for that given producer along with the list of CSCs that producer will monitor.

The Docker image and other producer configuration is handled on a site basis. Here is an example from the values-ncsa-teststand.yaml.

kafka-producers:
  image:
    repository: ts-dockerhub.lsst.org/salkafka
    tag: c0016
    pullPolicy: Always
    nexus3: nexus3-docker
  env:
    lsstDdsDomain: ncsa
    brokerIp: cp-helm-charts-cp-kafka-headless.cp-helm-charts
    brokerPort: 9092
    registryAddr: https://lsst-schema-registry-nts-efd.ncsa.illinois.edu
    partitions: 1
    replication: 3
    waitAck: 1
    logLevel: 20

The env information is specifically tailored for the NCSA teststand. The image information is applied to all producers at this site. You can override both the producers deployed, reconfigure them if necessary or add new ones to a specific site. You can also change the image information for a given producer as well. You must ensure that the different image can interact with the others already deployed without interfering with their functioning. Below is an example of doing all the above.

kafka-producers:
  image:
    repository: ts-dockerhub.lsst.org/salkafka
    tag: c0016
    pullPolicy: Always
    nexus3: nexus3-docker
  env:
    lsstDdsDomain: ncsa
    brokerIp: cp-helm-charts-cp-kafka-headless.cp-helm-charts
    brokerPort: 9092
    registryAddr: https://lsst-schema-registry-nts-efd.ncsa.illinois.edu
    partitions: 1
    replication: 3
    waitAck: 1
    logLevel: 20
    extras:
      LSST_DDS_RESPONSIVENESS_TIMEOUT: 15s
  producers:
    comcam: null
    auxtel: null
    eas:
      cscs: >-
        DSM
    latiss: null
    test:
      image:
        tag: c0016.001
    ccarchiver:
      cscs: >-
        CCArchiver
    cccamera:
      cscs: >-
        CCCamera
    ccheaderservice:
      cscs: >-
        CCHeaderService

The null is how to remove producers from the values.yaml configuration. The eas producer changes the list of CSCs from DIMM, DSM, Environment to DSM. The test producer changes the site configured image tag to something different. The ccarchiver, cccamera and ccheaderservice producers are new ones specified for this site only.

3.4.5   CSC Configuration¶

Most CSC configuration consists of Docker image information and environment variables that must be set as well as the namespace that the CSC should belong to. The namespace is handled in the CSC values.yaml in order to have that applied uniformly across all sites. An example of a simple configuration showing a specific namespace is shown below.

csc:
  namespace: maintel

CSCs may have other values they need to apply regardless of site. Here is an example from the mtcamhexapod application.

csc:
  namespace: maintel
  env:
    OSPL_INFOFILE: /tmp/ospl-info-mtcamhexapod.log
    OSPL_ERRORFILE: /tmp/ospl-error-mtcamhexapod.log
  osplVersion: V6.10.4

Other global environment variables can be specified in this manner. Note that variables specified in the values.yaml can still be overridden by site-specific files.

The Docker image configuration is handled on a site basis to allow independent evolution. This also applies to the LSST_DDS_PARTITION_PREFIX environment variable since those are definitely site specific. Below is an example site configuration from the mtcamhexapod for the NCSA test stand.

csc:
  image:
    repository: ts-dockerhub.lsst.org/mthexapod
    tag: c0016
    pullPolicy: Always
    nexus3: nexus3-docker
  env:
    LSST_DDS_PARTITION_PREFIX: ncsa
    RUN_ARG: --simulate 1
  shmemDir: /scratch.local/ospl

The RUN_ARG configuration sets the index for the underlying component that the container will run and puts it into simulation mode. Other site specific environment variables can be listed in the env section if they are appropriate to running the CSC container.

All containers require the use of the Nexus3 repository, identified by the use of ts-dockerhub.lsst.org in the image.repository name. The image.nexus3 key must be configured in order for secret access to occur. The value in the image.nexus3 entry is specific to the Nexus3 instance that is based in Tucson. This may be expanded to other replications in the future. The site specific configuration for the mtcamhexapod application given previously shows this information is configured.

The CSC container may need to override the command script that the container automatically runs on startup. An example of how this is accomplished is shown below.

csc:
  image:
    repository: ts-dockerhub.lsst.org/atdometrajectory
    tag: c0016
    pullPolicy: Always
  env:
    LSST_DDS_PARTITION_PREFIX: lsatmcs
  entrypoint: |
    #!/usr/bin/env bash

    source ~/miniconda3/bin/activate

    source $OSPL_HOME/release.com

    source /home/saluser/.bashrc

    run_atdometrajectory.py

The script for the entrypoint must be entered line by line with an empty line between each one in order for the script to be created with the correct execution formatting. The pipe (|) at the end of the entrypoint keyword is required to help obtain the proper formatting. Using the entrypoint key activates the use of the ConfigMap API.

If a CSC requires a physical volume to read and/or write files out to, the pvcMountpoint and nfsMountpoint keys should be used. This should be a rarely used variant, but it is supported. The persistent volume claim created by using pvcMountpoint is local to the Kubernetes cluster and by default is not persisted if the volume claim disappears. The Header Service will use this when deployed to the summit until the S3 system is available. A configuration might look like the following.

csc:
  ...
  pvcMountpoint:
  - name: www
    path: /home/saluser/www
    accessMode: ReadWriteOnce
    claimSize: 50Gi

The description of the claimSize units can be found at this page.

The nfsMountpoint key can be used to attach to a NFS export. The mtaos leverages this mechanism for reading Bulter ingested files. An example configuration taken from the mtaos summit configuration, looks like this.

csc:
  ...
  nfsMountpoint:
  - name: comcam-data
    containerPath: /readonly/lsstdata/comcam
    readOnly: true
    server: comcam-arctl01.cp.lsst.org
    serverPath: /lsstdata

3.4.6   Collector Applications¶

As noted earlier, these applications are collections of individual CSC apps aligned with a particular subsystem. The all-site configuration handles ArgoCD information, CSCs that are not simulators and possibly indexed CSCs. Here is how the values.yaml file for the maintel app looks.

spec:
  destination:
    server: https://kubernetes.default.svc
  source:
    repoURL: https://github.com/lsst-ts/argocd-csc
    targetRevision: HEAD

noSim:
  - mtptg
  - mtdometrajectory

The spec section is specific to ArgoCD and should not be changed unless you really understand the consequences. The exceptions to this are the repoURL and targetRevision parameters. It is possible the GitHub repository moves during the lifetime of the project, so repoURL will need to be updated if that happens. There might also be a need to testing something that is not on the master branch of the repository. To support that, change the targetRevision to the appropriate branch name. Use this sparingly, as main configuration tracking is on the master branch.

The site specific configurations handle setting up the list of CSCs to run and the environment for that site. It can also handle the need for running simulators on the summit. Below is the configuration for maintel from the values-summit.yaml configuration.

env: summit
cscs:
  - mtaos
  - mtptg
  - mtmount
  - mtdome
  - mtdometrajectory
  - ccheaderservice
runAsSim:
  - mtaos
  - mtdome
  - mtmount

The env parameter sets the value-<env>.yaml for the listed CSC apps. This will change on a per site basis. The cscs parameter is the listing of the CSC apps that the collector app will control.

The auxtel collector app follows similar configuration mechanisms but controls a different list of CSC apps as does the obssys and eas collector apps.

The eas collector app has one other variation since it handles indexed CSCs all coming from the same application. Below is a section from the values.yaml file showing how indexed components are treated.

...
indexed:
  dimm: 4
  weatherstation: 14
  dsm: 3

The key points to the application directory that will be used and the number represents the length of the key. This is necessary since there are no tools in the Helm system that allow the length determination of a string. To use the indexed components, here is the values-summit.yaml configuration for the eas app.

env: summit
cscs:
  - dimm1
  - dimm2

Using the length provided in the values.yaml file, the number at the end of the name is retrieved. This is used to set the appropriate configuration file in the application directory for the specific index. The index configuration files look like values-<app name><#>.yaml and contain index specific setup. The index=1 DIMM component configuration values-dimm1.yaml is shown below.

csc:
  env:
    RUN_ARG: 1

The index specific configuration required will vary with the different CSCs.