ci: add validate-profiles job to PR workflow

[hulynn]

Add validate-profiles job that runs on every pull request to catch blueprint and policy configuration regressions before merge:

• Validates all 4 inference profiles exist in blueprint.yaml (default, ncp, nim-local, vllm) with required fields
• Validates all policy preset YAML files have valid structure
• Validates base sandbox policy has required fields

This complements the existing test-unit and test-e2e-sandbox jobs by catching configuration breakage early in the PR cycle.

Summary

Add validate-profiles job to the PR workflow that validates blueprint inference profiles, policy preset YAML files, and base sandbox policy on every pull request. This catches configuration regressions before merge, complementing the existing test-unit and test-e2e-sandbox jobs.

Related Issue

None

Changes

• Add validate-profiles job to .github/workflows/pr.yaml
• Validates all 4 inference profiles (default, ncp, nim-local, vllm) in blueprint.yaml
• Validates all policy preset YAML files under nemoclaw-blueprint/policies/presets/
• Validates base sandbox policy nemoclaw-blueprint/policies/openclaw-sandbox.yaml

Type of Change

• Code change for a new feature, bug fix, or refactor.
• Code change with doc updates.
• Doc only. Prose changes without code sample modifications.
• Doc only. Includes code sample changes.

Testing

• make check passes.
• npm test passes.
• make docs builds without warnings. (for doc-only changes)

Checklist

General

• I have read and followed the contributing guide.
• I have read and followed the style guide. (for doc-only changes)

Code Changes

• make format applied (TypeScript and Python).
• Tests added or updated for new or changed behavior.
• No secrets, API keys, or credentials committed.
• Doc pages updated for any user-facing behavior changes (new commands, changed defaults, new features, bug fixes that contradict existing docs).

Doc Changes

• Follows the style guide. Try running the update-docs agent skill to draft changes while complying with the style guide. For example, prompt your agent with "/update-docs catch up the docs for the new changes I made in this PR."
• New pages include SPDX license header and frontmatter, if creating a new page.
• Cross-references and links verified.

Summary by CodeRabbit

• Chores

  • Added a CI job to validate blueprint and policy profiles on pull requests.
  • Added a standalone validation script to check blueprint and policy consistency and required fields.

• Tests

  • Updated end-to-end test expectations to include an additional inference profile (ncp) and adjusted the success message.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b7735991-4583-4d3c-8700-0a684c80760d

📥 Commits

Reviewing files that changed from the base of the PR and between 81795c2 and c0103ea.

📒 Files selected for processing (3)

• .github/workflows/pr.yaml
• test/e2e-test.sh
• test/validate-blueprint.py

✅ Files skipped from review due to trivial changes (2)

• test/validate-blueprint.py
• .github/workflows/pr.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

• test/e2e-test.sh

---

📝 Walkthrough

Walkthrough

Adds a GitHub Actions job that runs a new Python validator for blueprint and sandbox policy YAMLs, and updates an e2e test to expect an additional inference profile.

Changes

Cohort / File(s)	Summary
CI workflow \.github/workflows/pr.yaml	Added validate-profiles job: checks out repo, sets up Python 3.11, installs pyyaml, and runs python3 test/validate-blueprint.py.
Validation script test/validate-blueprint.py	New script that loads nemoclaw-blueprint/blueprint.yaml and nemoclaw-blueprint/policies/openclaw-sandbox.yaml, verifies profiles vs components.inference.profiles, ensures required profile fields (provider_type, endpoint/dynamic_endpoint) are present, and checks policy keys (version, network_policies).
E2E test update test/e2e-test.sh	Updated inline blueprint validation to require four inference profiles (default, vllm, nim-local, ncp) and adjusted the success message.

Sequence Diagram(s)

sequenceDiagram participant GH as "GitHub Actions" participant Runner as "Runner (ubuntu-latest)" participant FS as "Repository Files" participant Py as "Python 3.11 (validator)" GH->>Runner: trigger PR workflow (validate-profiles job) Runner->>FS: checkout repository Runner->>Py: setup Python 3.11 & pip install pyyaml Runner->>Py: run `test/validate-blueprint.py` Py->>FS: read `nemoclaw-blueprint/blueprint.yaml` and `.../openclaw-sandbox.yaml` Py-->>Py: validate declared vs defined profiles and required fields Py-->>Runner: exit 0 (pass) or 1 (fail) Runner-->>GH: report job result 

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 I nibble through YAML, hop by hop,
Counting profiles till the last line's top,
A tiny job hums in the CI light,
Four names checked — the pipeline's bright! 🌿

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'ci: add validate-profiles job to PR workflow' directly and clearly describes the main change: adding a new CI job to the GitHub Actions workflow.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (3)

.github/workflows/pr.yaml (3)

88-95: Consider validating that preset files actually exist.

If the presets directory is empty or contains no .yaml files, the script will print "Policy presets: 0 files OK" and pass silently. This could mask accidental deletion of all preset files. Adding a minimum count check would catch this regression.

🛡️ Proposed fix to add minimum file check

 preset_dir = 'nemoclaw-blueprint/policies/presets' count = 0 for f in sorted(os.listdir(preset_dir)): if f.endswith('.yaml'): data = yaml.safe_load(open(os.path.join(preset_dir, f))) assert data and 'network_policies' in data, f'{f}: missing network_policies key' count += 1 + assert count > 0, f'No policy preset YAML files found in {preset_dir}' print(f'Policy presets: {count} files OK')

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed. In @.github/workflows/pr.yaml around lines 88 - 95, The script currently counts YAML preset files in preset_dir ('nemoclaw-blueprint/policies/presets') but silently allows count == 0; update the logic after the loop (the block using preset_dir, count, and the final print) to validate that at least one .yaml file was found and fail loudly if none are present (e.g., raise an AssertionError or exit non‑zero with a clear message like "no preset files found in preset_dir"); keep the existing assertions for each file but add this minimum-count check so accidental deletion of all presets is detected. 

---

78-78: Unused sys import.

The sys module is imported but never used in the script.

🧹 Proposed fix

- import yaml, os, sys + import yaml, os

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed. In @.github/workflows/pr.yaml at line 78, The import line "import yaml, os, sys" contains an unused symbol sys; remove sys from the import so only used modules (yaml and os) are imported or split into separate imports as needed, ensuring no other references to sys exist in the script (search for "sys" and delete or replace any usages before removing the import). 

---

80-80: Consider using context managers for file handles.

The open() calls rely on garbage collection to close file handles. While this works in short scripts, using context managers (with open(...) as f) is more explicit and aligns with Python best practices.

♻️ Proposed refactor using context managers

- bp = yaml.safe_load(open('nemoclaw-blueprint/blueprint.yaml')) + with open('nemoclaw-blueprint/blueprint.yaml') as f: + bp = yaml.safe_load(f)

- data = yaml.safe_load(open(os.path.join(preset_dir, f))) + with open(os.path.join(preset_dir, fp)) as fh: + data = yaml.safe_load(fh)

- policy = yaml.safe_load(open('nemoclaw-blueprint/policies/openclaw-sandbox.yaml')) + with open('nemoclaw-blueprint/policies/openclaw-sandbox.yaml') as f: + policy = yaml.safe_load(f)

Also applies to: 92-92, 97-97

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed. In @.github/workflows/pr.yaml at line 80, Replace bare open() calls passed directly into yaml.safe_load and any other places using open('nemoclaw-blueprint/blueprint.yaml') with context-manager usage: open the file with "with open(... ) as f:" and then call yaml.safe_load(f) inside the block, doing the same for the other occurrences noted (the expressions like yaml.safe_load(open(...)) at the other lines). This ensures the file handle is closed immediately after use while preserving existing variable names (e.g., bp) and behavior. 

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed. Nitpick comments: In @.github/workflows/pr.yaml: - Around line 88-95: The script currently counts YAML preset files in preset_dir ('nemoclaw-blueprint/policies/presets') but silently allows count == 0; update the logic after the loop (the block using preset_dir, count, and the final print) to validate that at least one .yaml file was found and fail loudly if none are present (e.g., raise an AssertionError or exit non‑zero with a clear message like "no preset files found in preset_dir"); keep the existing assertions for each file but add this minimum-count check so accidental deletion of all presets is detected. - Line 78: The import line "import yaml, os, sys" contains an unused symbol sys; remove sys from the import so only used modules (yaml and os) are imported or split into separate imports as needed, ensuring no other references to sys exist in the script (search for "sys" and delete or replace any usages before removing the import). - Line 80: Replace bare open() calls passed directly into yaml.safe_load and any other places using open('nemoclaw-blueprint/blueprint.yaml') with context-manager usage: open the file with "with open(... ) as f:" and then call yaml.safe_load(f) inside the block, doing the same for the other occurrences noted (the expressions like yaml.safe_load(open(...)) at the other lines). This ensures the file handle is closed immediately after use while preserving existing variable names (e.g., bp) and behavior. 

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 816e9212-6bd4-49e9-9839-83d738d9e9eb

📥 Commits

Reviewing files that changed from the base of the PR and between dbfd78c and e305efb.

📒 Files selected for processing (1)

• .github/workflows/pr.yaml

[jayavenkatesh19]

Thanks for splitting this out! A few thoughts for future proofing this:

The policy preset validation (network_policies key exists on each preset) is already covered by test/policies.test.js which runs in the same PR workflow. It checks all 9 presets for network_policies, name, description, endpoints, and schema structure. So we can drop that part.

Hardcoding the profile list will be a maintenance issue. The check reads from blueprint.yaml but validates against a hardcoded ['default', 'ncp', 'nim-local', 'vllm']. If someone adds a new profile, they'd have to remember to update this list in the CI check too. The blueprint already declares its profiles at the top level (lines 9-13), so we can read from there instead.

We should check endpoint instead of model. ncp uses dynamic_endpoint: true so model may not always be required, but every profile needs provider_type and endpoint for inference routing to work.

It would also be good to move this out of inline Python for maintainability. A test/validate-blueprint.py or test/blueprint.test.js would be easier to maintain and test locally than Python embedded in YAML.

Also, test/e2e-test.sh Phase 3 already validates 3 of the 4 profiles, just missing ncp. Worth adding that as a one-liner there too.

I'm thinking about the check structure like this:

• read declared profiles from blueprint.yaml top-level 'profiles' key

• read defined profiles from components.inference.profiles

• for each declared profile:

  • has a definition in the inference section
  • has provider_type
  • has endpoint

• read base sandbox policy (openclaw-sandbox.yaml):

  • has version
  • has network_policies

I might be missing something else here, so would love to hear your thoughts about this!

[hulynn]

@jayavenkatesh19 Thanks for the thorough review! All great points — I agree with each one:

1. Will drop the policy preset validation since test/policies.test.js already covers it.
2. Will read declared profiles dynamically from blueprint.yaml instead of hardcoding.
3. Will validate provider_type + endpoint instead of model.
4. Will move the inline Python to a standalone test/validate-blueprint.py.
5. Will add ncp to Phase 3 in test/e2e-test.sh as well.

I'll push an updated commit shortly.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed. Inline comments: In `@test/validate-blueprint.py`: - Around line 31-33: The YAML root (bp) may be None or not a mapping, so before using bp.get(...) in the declarations for declared and defined, coerce bp to a dict when safe_load returns a non-mapping: after bp = yaml.safe_load(open(BLUEPRINT_PATH)) check if isinstance(bp, dict) (or use "if not isinstance(bp, dict): bp = {}") so declared = bp.get("profiles", []) and defined = bp.get("components", {}).get("inference", {}).get("profiles", {}) won't raise; apply the same defensive check where bp is used later (around the code that computes declared/defined at lines showing those variables). - Around line 41-43: The current loop in validate-blueprint.py only checks presence of keys (cfg and REQUIRED_PROFILE_FIELDS) but allows empty/null values; update the validation in the loop that iterates REQUIRED_PROFILE_FIELDS (and the check(...) calls) to assert each required field in cfg is non-empty/non-null (e.g., not None and not empty string/whitespace) — specifically ensure fields like 'provider_type' and 'endpoint' are validated for truthiness and meaningful content; use the existing check function to report a clear failure message including the profile name and field when a required value is missing or blank. 

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 443447d1-15d9-41f4-9c1c-4550fec84a15

📥 Commits

Reviewing files that changed from the base of the PR and between e305efb and 5955416.

📒 Files selected for processing (3)

• .github/workflows/pr.yaml
• test/e2e-test.sh
• test/validate-blueprint.py

🚧 Files skipped from review as they are similar to previous changes (1)

• .github/workflows/pr.yaml