Metadata-Version: 2.4
Name: cm-setup-ajr
Version: 2.5.10
Summary: AJR Cluster Manager Setup Plugin
Author-email: NVIDIA CORPORATION & AFFILIATES <sw-bright-tpsi-team@nvidia.onmicrosoft.com>
Maintainer-email: TPSI team <sw-bright-tpsi-team@nvidia.onmicrosoft.com>
License-Expression: LicenseRef-NvidiaProprietary
Project-URL: Homepage, https://www.nvidia.com
Project-URL: Bug Tracker, https://jirasw.nvidia.com/
Keywords: deployment,ajr
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: Topic :: System :: Installation/Setup
Classifier: Topic :: System :: Systems Administration
Classifier: Operating System :: POSIX :: Linux
Classifier: Environment :: Console
Classifier: Environment :: Console :: Curses
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Requires-Python: >=3.12.0
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Jinja2
Requires-Dist: PyYAML
Provides-Extra: test
Requires-Dist: cerberus<2,>=1.3.5; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Dynamic: license-file

# CM Setup AJR Plugin

AJR decoupled plugin for BCM cm-setup

## Run:ai API authentication

### User-cluster system nodes

The advice and job operators require a user-cluster node with at least one of
these role labels: `node-role.kubernetes.io/control-plane`,
`node-role.kubernetes.io/master`, or `node-role.kubernetes.io/runai-system`.
Required node affinity implements this OR condition, relying on BCM role
assignments to distinguish system nodes from GPU workers.

Both operators tolerate the standard `control-plane` and `master` `NoSchedule`
taints. Required node affinity and tolerations are installer-owned; configured
node selectors and pod affinity remain additional constraints. Kubernetes handles
node readiness, taints, and resource availability. If no eligible system node
exists, the pods remain Pending rather than falling back to ordinary workers.
The log-collector DaemonSet is unchanged and continues to collect compute-node logs.

### Credentials

The Run:ai install provisions credentials before deploying either AJR or its
user-cluster operators. No running AJR API is needed to bootstrap authentication.
It generates a 256-bit random bearer token, stores the original only in the user
namespace's `ajr-api-token` Secret (`token` key), and stores its SHA-256 hex digest
in the admin namespace's `ajr-api-token-digest` Secret (`token-sha256` key).
Helm values contain Secret references, not credentials. Slurm is unchanged.

Both operators use verified HTTPS. The installer copies only public certificates
from the TLS Secret referenced by the NMC Gateway HTTPS listener into the user
namespace's `ajr-api-gateway-ca` Secret. The configured AJR hostname must match
the server certificate. No TLS private key is copied, and verification is never
disabled. The supported listener has a single Kubernetes TLS Secret reference.

Reruns preserve a matching token/digest pair. If installation stops after storing
the user token, retrying creates its missing admin digest without rotation. A
missing user token with an existing digest, malformed token, or mismatched digest
stops the install: restore the matching credentials rather than deleting a
Secret to force regeneration. Concurrent creation conflicts also stop safely.
Credentials are retained during stage undo for retries; normal AJR namespace
removal deletes them.

Rotation is explicit: update the user token and its admin digest together.
Secret propagation can cause transient authentication failures. After a gateway
certificate change, update the public CA Secret and restart both operators;
the installer refuses to overwrite a different existing CA bundle automatically.

This requires gateway and operator charts/images with bearer/digest support.
Chart rendering checks reject older charts that ignore the authentication values;
the selected images must also implement the corresponding chart contract. Use
matching published artifacts in connected installations and airgap bundles.

## Slurm host logging through the BCM Gateway

When AJR manages Slurm host Alloy, the selected admin Kubernetes cluster must have
its head-node Gateway proxy enabled and Loki exposed through a Gateway HTTPRoute.
These are the standard NMC Kubernetes wizard choices. Enter the same Loki API
credentials in the AJR wizard.

The installer reads `ingressProxyListenPort` from that cluster and configures
`https://master.cm.cluster:<configured-port>/loki/api/v1/push`. It uses the public
certificate chain referenced by the HTTPS listener on `bcm-gateway/bcm-gateway`
for TLS trust, without copying the private key or disabling verification. The
certificate must cover `master.cm.cluster`; rerun the installer after a certificate
rotation to refresh the managed Alloy trust configuration.

Proxy settings, certificates, and credential format are validated before host
package installation. This is not an end-to-end Loki connectivity or authentication
test. After deployment, verify that host Alloy delivers logs without connection,
TLS, authentication, or retry errors.

This change does not affect the Run:ai Kubernetes log collector or preserve/merge
an independently managed host Alloy configuration. Do not deploy the managed
Alloy role over a custom configuration without reviewing that separate ownership
conflict.

## Getting Started

### Prerequisites

- **[uv](https://github.com/astral-sh/uv)**: An extremely fast Python package installer and resolver.
  ```bash
  curl -LsSf https://astral.sh/uv/install.sh | sh
  ```
- **Docker**: For building and testing in containers.

### Checking Out a Branch

```bash
git checkout main
git pull
git checkout -b feat/YOUR-JIRA-TICKET
```

## Development Workflow

### Quick Start (Makefile)

The `Makefile` is the primary interface for local development:

```bash
make help           # List available targets
make lint           # Run pre-commit linters
make test           # Run unit tests locally
make plugin         # Build the debian package (using Docker)
make clean          # Clean build artifacts
make shell          # Debug: Drop into the builder container shell
```

**Note**: `make plugin` generates a package versioned as `0.0.0~dev-<timestamp>` to distinguish local builds.

### Running Unit Tests

```bash
# Using Makefile
make test

# Or manually with uv
uv venv --python 3.12
source .venv/bin/activate
uv pip install . pytest pytest-cov
uv run pytest --cov=cmsetup --cov-report=term
```

### Running Pre-commit Hooks

```bash
uv pip install pre-commit
pre-commit install          # One-time setup
pre-commit run --all-files  # Run on all files
```

### Type Checking with mypy

mypy is configured in `pyproject.toml` with strict mode.

CI fetches and installs cmsetup to provide type stubs, then auto-installs any additional missing stubs:

```bash
# CI downloads cmsetup wheel from bcm/cm/cm-setup artifacts using VERSION_BCM
CMSETUP_REF="$(cat VERSION_BCM).0"
curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" \
  "${CI_API_V4_URL}/projects/bcm%2Fcm%2Fcm-setup/jobs/artifacts/${CMSETUP_REF}/download?job=build-python-package" \
  -o artifacts.zip
unzip -q artifacts.zip
uv pip install ./dist/*.whl

# Auto-install missing type stubs (workaround for https://github.com/python/mypy/issues/10863)
uv run mypy . > /dev/null 2>&1 || uv run mypy --install-types --non-interactive
uv run mypy cmsetup
```

To run locally (requires cmsetup installed):

```bash
uv pip install mypy pip  # pip is required for mypy --install-types
# Auto-install stubs if needed, then run type check
uv run mypy . > /dev/null 2>&1 || uv run mypy --install-types --non-interactive
uv run mypy cmsetup
```

> **Note**: The `cache_dir` option should NOT be set to `/dev/null` in `pyproject.toml` as mypy cache is required for auto stub install to work.

### Local Testing in Docker

```bash
# Build test container (includes cm-setup base + plugin)
docker build -f Dockerfile.test -t cm-plugin-test .

# Run plugin commands
docker run --rm cm-plugin-test cm-ajr-setup --help --no-distro-checks --skip-headnode-check
```

### Building the Debian Package

```bash
# Using Makefile (recommended)
make plugin

# Using Docker directly
docker build -f Dockerfile.build -t plugin-builder .
docker run --rm -v $(pwd):/shared plugin-builder sh -c "cp /build/deb/*.deb /shared/"
```

### Testing on Krusty

```bash
apt-get install ./<your-package-name>.deb
module load python3
cm-ajr-setup
```

## Key Configuration Files

| File | Purpose |
|------|---------|
| `VERSION` | Semantic version aligned with NMC (e.g., `2.2.0`) |
| `VERSION_BCM` | BCM major version, coupled to NMC release (e.g., `11` for NMC 2.2) |
| `pyproject.toml` | Package metadata, entry points, dependencies |
| `nfpm.yaml` | Debian package configuration |
| `.gitlab-ci.yml` | CI/CD pipeline variables |
| `cmsetup/plugins/autonomous_job_recovery/` | Plugin implementation |

## CI/CD Pipeline

The pipeline is defined in `cds/cicd-pipelines/pipelines/ncp/nmc/cm-setup-plugin-pipeline.yml`.

### Package Naming

Packages include BCM version: `cm-setup-ajr-bcm<VERSION_BCM>`

Example: `cm-setup-ajr-bcm11_2.2.0_amd64.deb`

### Release Process

#### 1. Merge Request (MR)
- **Automatic**: Runs linting, type checking (mypy), unit tests, and security scans.
- **Build**: Creates a test package with version `0.0.1~mr.<sha>`.
- **Publish**: Manual publishing to **Dev** Artifactory for testing.

#### 2. Main/Release Branch (Pre-release)
- **Build**: Creates package with version `<VERSION>~pr.<pipeline_iid>.<sha>`.
- **Slack Notification**: Team is notified when pre-release is published to Dev Artifactory.
- **QA Verification**: QA tests the pre-release package.
- **Release Promotion**: After QA sign-off:
    1. Ensure the `VERSION` file contains the version you want to release.
    2. Go to the pipeline for the latest commit.
    3. Trigger the manual **`promote-release`** job.
    4. This creates a Git tag matching the `VERSION` file.
    5. Bump `VERSION` to the next version in a new MR.

#### 3. Release (Git Tag)
- **Build**: Final package with exact version (e.g., `2.2.3`).
- **Dev Publish**: Automatically publishes to **Dev** Artifactory.
- **Slack Notification**: Team is notified of the promotion.
- **Release Notes**: Generates changelog and GitLab Release page.
- **QA Final Verification**: QA verifies the release package.
- **Prod Publish**: Manual trigger to publish to **Prod** Artifactory.
- **Slack Notification**: Team is notified when published to prod.

## Versioning and Branching

### Version Files

- **`VERSION`** - Plugin semantic version (must align with NMC major.minor)
- **`VERSION_BCM`** - BCM major version (integer, coupled to NMC release)

> **Note**: `VERSION_BCM` is not independently bumped. BCM is a component of NMC and they are released together (e.g., NMC 2.2 → BCM 11, NMC 2.3 → BCM 12).

### Branching Strategy

| Branch | Purpose |
|--------|---------|
| `main` | Latest NMC/BCM combination (active development) |
| `release/nmc-X.Y-bcmZ` | Maintenance branch for specific NMC/BCM combination |

### Version Alignment

- Plugin `VERSION` major.minor must match NMC version (e.g., `2.2.x` for NMC 2.2)
- Patch version can be bumped independently
- CI validates alignment on release branches

### Creating Release Branches

When a new NMC version is released (which includes a corresponding BCM version):

```bash
# Create release branch for current version
git checkout main
git checkout -b release/nmc-2.2-bcm11
git push origin release/nmc-2.2-bcm11

# Update main for next NMC/BCM version (both change together)
git checkout main
echo "2.3.0" > VERSION      # New NMC version
echo "12" > VERSION_BCM     # Corresponding BCM version for NMC 2.3
git add VERSION VERSION_BCM
git commit -m "Bump for NMC 2.3 / BCM 12"
```

### Hotfix on Release Branch

```bash
git checkout release/nmc-2.2-bcm11
# Make fix, bump patch version
echo "2.2.4" > VERSION
git commit -am "Fix: description"
# Create MR against release branch, then promote-release
```

## AHR integration CA support (draft)

The AHR integration questions are available in both Slurm and Run:ai mode when
the AHR backend is installed. The optional **CA bundle file** field accepts a
public PEM CA bundle for a privately signed endpoint; it must not contain keys.
With an empty field, setup discovers the installed AHR self-signed CA ConfigMap
only when the configured endpoint matches that AHR release. Publicly trusted
endpoints retain normal certificate verification. The saved configuration key
is `modules.autonomous_job_recovery.ahr.ca_bundle_file`.

This draft requires the companion Warthog client/chart support for
`anomaly-detection-isolator.break_and_fix.ca_bundle`. Warthog `1.7.0-patch5`
does **not** consume that setting. A compatible chart and isolator image must
be released and selected in the component manifest before this change is ready
to ship. This change intentionally does not invent or select a new artifact
version. CA rotation requires rerunning setup to refresh the Helm values.
