Initial project documentation and status

story-so-far.md

@@ -0,0 +1,582 @@
+# MeshCentral ↔ OpenClaw Bridge — Story So Far
+
+**Project:** `meshcentral-openclaw-bridge`  
+**Project path:** `/home/ledadmin/projects/meshcentral-openclaw-bridge`  
+**Owner:** Rob  
+**Project manager:** Clawdia  
+**Write-up date:** 2026-08-26  
+**Current status:** **Paused / blocked after three failed, bounded read-only live probes**
+
+> This document is a narrative history of the work completed so far. It deliberately excludes the MeshCentral password, authentication headers, session tokens, and raw server-provided close/error text. The project credential remains outside the repository in a protected file.
+
+---
+
+## 1. The original goal
+
+The project began as an effort to give OpenClaw a carefully controlled interface to Rob's MeshCentral deployment. The intended long-term capability is for Rob to ask OpenClaw questions such as:
+
+- Which managed systems are offline?
+- What operating system, uptime, and agent status does a host report?
+- Can a narrowly defined diagnostic check be run against one approved system?
+- Later, can an explicitly confirmed and auditable remediation action be performed?
+
+The important design choice from the beginning was to treat this as a remote-management control plane, not as a convenience API. MeshCentral can ultimately expose shell access, file transfer, power operations, and other highly privileged actions, so the bridge must not become an unrestricted command channel.
+
+The delivery plan was split into explicit phases:
+
+1. Discovery
+2. Architecture and security design
+3. Read-only inventory/status MVP
+4. Allowlisted diagnostics
+5. Confirmation-gated reversible actions
+6. Hardening, deployment, rollback, and operational handoff
+
+Only the first three phases, plus local hardening and compatibility testing, have been pursued. Diagnostics remain simulated/mock-only, and no remote command or state-changing operation has been implemented or used.
+
+---
+
+## 2. Initial scope and safety boundaries
+
+The initial milestone was deliberately narrowed to:
+
+- MeshCentral server version/health
+- Device-group listing
+- Device inventory
+- Online/offline state
+- Safe device metadata
+- Structured results suitable for an OpenClaw integration
+
+The initial milestone explicitly excluded:
+
+- Arbitrary shell or command execution
+- Remote diagnostics that execute commands
+- File transfer
+- Router/tunnel operations
+- Reboots, shutdowns, or service changes
+- User/group administration
+- Autonomous remediation
+- Fleet-wide operations
+- Public exposure of the bridge
+- Access to the production Proxmox cluster through this project
+
+The first architecture decision, recorded as ADR-0001, was therefore to begin read-only. That allowed the project to validate connectivity, authentication, data mapping, logging, failure handling, and OpenClaw-facing contracts without changing an endpoint.
+
+A second decision was to prefer a local typed bridge over browser automation. Browser automation was retained as a possible fallback for discovery or occasional operations, but not as the primary integration architecture. A local adapter is easier to constrain, unit-test, audit, and maintain than a collection of UI selectors and browser sessions.
+
+---
+
+## 3. Discovery and integration research
+
+### 3.1 MeshCentral deployment facts
+
+The project documented the known deployment as follows:
+
+- Public/reverse-proxy name: `mesh.ledrew.me`
+- Internal MeshCentral address: `10.10.10.177:80`
+- Reported MeshCentral version: `1.2.5`
+- Location: LXC on PVE2
+- Installation method: Proxmox Community Scripts MeshCentral installer
+- Installation path: `/opt/meshcentral`
+- Data path: `/opt/meshcentral/meshcentral-data`
+- Service: `meshcentral.service`
+- Approved test device: `PVE3`
+- Approved MeshCentral group: `Servers`
+- Dedicated integration account: `sa_clawdia`
+- Intended external control endpoint: `wss://mesh.ledrew.me/control.ashx`
+
+The first live target was restricted to PVE3. PVE1 and PVE2 production infrastructure were explicitly kept out of scope.
+
+### 3.2 Options considered
+
+The research looked at several possible integration surfaces:
+
+1. **Official `meshctrl.js` control client** — selected as the practical starting point.
+2. **MeshCentral server internals or an in-process module** — rejected for the first version because it would couple the bridge to private implementation details, deployment paths, process state, and upgrade behavior.
+3. **Browser automation** — not selected as the primary path because it is more fragile and difficult to constrain.
+4. **Third-party MCP or MeshCentral adapters** — no clearly mature, maintained general-purpose adapter was found. A related automation-template project was noted, but it was not treated as a trusted dependency.
+
+The selected approach was a separate, typed local adapter that implements only a small read-only subset of MeshCentral's control-channel protocol. The raw protocol would remain behind the adapter boundary rather than being exposed directly as an OpenClaw tool API.
+
+### 3.3 Protocol and security observations from research
+
+The official MeshCentral control client uses a WebSocket connection to `/control.ashx` and exchanges JSON actions and responses. Relevant read-only actions include:
+
+- `serverversion`
+- `meshes`
+- `nodes`
+- `deviceinfo`
+
+The official client also supports more powerful operations, but the project intentionally did not pass those through.
+
+Research found two especially important security points:
+
+- MeshCentral's control protocol is an implementation-level contract, not a separately versioned stable REST API. Version pinning, fixtures, schema checks, and live compatibility verification are therefore necessary.
+- The upstream client source was observed to disable TLS certificate verification. The bridge intentionally did **not** copy that behavior. The bridge requires certificate validation and supports an explicit CA configuration path instead.
+
+---
+
+## 4. The first implementation iteration: mock-backed scaffold
+
+The first code iteration created a local TypeScript project under `implementation/`. It established the bridge boundary without requiring a live MeshCentral connection.
+
+### 4.1 Core structure
+
+The implementation introduced:
+
+- TypeScript source and build configuration
+- Zod-backed typed contracts
+- A `MeshCentralAdapter` interface
+- A mock adapter for offline development
+- Adapter factory logic
+- Structured bridge errors and error codes
+- Example configurations
+- Jest tests and coverage output
+- Build and lint scripts
+
+The mock adapter models device groups, devices, status, metadata, and diagnostic results. The mock data includes online and offline devices, group filtering, device lookups, and output truncation behavior.
+
+### 4.2 What the mock path proved
+
+The mock path established that the intended OpenClaw-facing abstraction could support:
+
+- Listing devices
+- Filtering by group ID
+- Getting one device
+- Listing groups
+- Getting one group
+- Returning online/offline/unknown state
+- Returning safe metadata such as OS, uptime, IPs, and agent version
+- Running simulated diagnostic checks
+- Reporting health
+- Returning structured errors for missing resources
+- Enforcing output limits and truncation metadata
+
+At this stage, the real adapter was either absent or only a scaffold. This is the state described by the earliest independent review: the mock path was functional, while the live path was not yet suitable for use.
+
+---
+
+## 5. The first independent review and its findings
+
+An independent review was performed against the early implementation. At that point the local suite was already passing, but the review correctly identified that passing mock tests did not make the real control path safe.
+
+The early review reported:
+
+- The real adapter was not yet fully implemented.
+- Credential validation and sanitization were insufficient.
+- Response schema validation was missing.
+- Error paths could potentially expose too much detail.
+- TLS behavior needed to be explicit and configurable.
+- There was no request rate limiting or bounded concurrency.
+- Device/group allowlist validation needed strengthening.
+- Structured audit logging was incomplete.
+- Separate connection/request timeouts and reconnect behavior were desirable future improvements.
+
+The early review's local test snapshot was 52 tests across 8 suites, with build and lint passing. Its conclusion was explicitly **do not enable live testing** until the high-severity findings were addressed.
+
+This review became the basis for the hardening pass rather than being discarded. Later documentation records which findings were fixed, which were deferred, and what residual risks remained.
+
+---
+
+## 6. Hardening iteration: making the local MVP defensible
+
+The next major iteration turned the scaffold into a real, but still tightly constrained, MeshCentral control-channel adapter.
+
+### 6.1 Real MeshCentral client
+
+`implementation/src/meshctrl-client.ts` was developed as a WebSocket client for MeshCentral's `/control.ashx` endpoint. It added:
+
+- WebSocket URL construction
+- Authentication header creation
+- Connection and request timeouts
+- Request/response correlation
+- Bounded retry behavior
+- Clean disconnect and cleanup
+- Response parsing
+- Protocol and action validation
+- Explicit TLS certificate validation
+- Optional custom CA support
+
+The client was designed to fail closed when the session, response shape, action, or connection state was ambiguous.
+
+### 6.2 Credential handling
+
+Credential handling was hardened so that:
+
+- Credentials are validated before encoding or storage.
+- Invalid usernames, passwords, and optional TOTP values are rejected.
+- User-facing configuration failures remain generic.
+- Credential metadata is summarized rather than printed.
+- Passwords, TOTP values, authentication headers, cookies, and tokens are not intentionally logged.
+- The live runner reads the password from a protected runtime file rather than accepting it as a command-line argument.
+
+The actual credential is not part of this repository or this write-up.
+
+### 6.3 TLS handling
+
+The bridge explicitly sets certificate validation on the WebSocket connection. It does not inherit the upstream client's certificate-verification bypass. A private CA bundle can be configured if needed, but disabling validation is prohibited.
+
+This distinction matters because the connection is a management control path. A successful WebSocket connection is not enough if the peer identity has not been verified.
+
+### 6.4 Protocol and response validation
+
+The client added several defensive checks:
+
+- Maximum response size of 1 MB
+- JSON parsing and protocol-envelope validation
+- Action-specific response schemas using Zod
+- Request/response action matching
+- Rejection of malformed or unexpected responses
+- Generic MeshCentral request errors instead of passing raw server error text through the bridge
+
+These controls address the early review's concern that a syntactically valid WebSocket message should not automatically be trusted as a valid response.
+
+### 6.5 Target protection
+
+The real adapter added device and group allowlists. It filters inventory results and checks requested targets against the configured allowlist.
+
+The intended live configuration is single-target:
+
+- MeshCentral group: `Servers`
+- Device: `PVE3`
+- No PVE1/PVE2 access
+
+The adapter also validates allowlist configuration rather than treating arbitrary IDs as trusted input.
+
+### 6.6 Rate limiting and concurrency
+
+The client added two protective controls:
+
+- A token-bucket rate limiter for outbound requests
+- A concurrency limiter/semaphore for simultaneous requests
+
+The safe default for concurrent requests is 5. Configuration values are bounded to prevent accidentally setting unreasonable limits. These controls are safety guardrails; they are not considered a substitute for proper capacity testing.
+
+### 6.7 Logging and audit controls
+
+Structured logging and audit utilities were added or expanded. The implementation records security-relevant events such as connection attempts, authentication outcomes, requests, target decisions, results, and failures without intentionally including secrets.
+
+Redaction handles:
+
+- Known secret keys
+- Password and token fields
+- Authentication headers
+- Token-like values
+- Base64-looking secret material
+- Hex-looking secret material
+- Nested objects and arrays
+
+A later logger cleanup also removed direct audit output from the client/rate-limiter paths in favor of the existing logger abstraction. This preserved redaction and made the lint gate clean.
+
+### 6.8 Diagnostics remain mock-only
+
+The adapter interface still contains diagnostic support for the mock implementation, but the real path does not execute commands. The live runner was specifically tested to ensure it contains only the approved read-only actions and does not expose a general-purpose command or diagnostic action.
+
+---
+
+## 7. Hardening verification milestones
+
+The test suite grew as the implementation gained controls.
+
+The recorded milestones were approximately:
+
+1. **Early scaffold:** 52 tests across 8 suites; build and lint passed.
+2. **Post-hardening checkpoint:** 56 tests; build, lint, and `tsc --noEmit` passed.
+3. **Expanded local MVP:** 90 tests across 11 suites; build passed; lint passed with no warnings.
+4. **Secure live-test runner:** 91 tests; build passed; lint passed.
+5. **Final runner/diagnostic and cleanup changes:** 12 suites and 91 tests; build passed; lint passed with no warnings.
+
+The project documentation also records a 90/90 local verification result immediately before the final live-test preparation. The small count differences reflect tests added during subsequent runner and cleanup work, not a reduction in coverage.
+
+The test categories include:
+
+- Contract/schema validation
+- Mock adapter behavior
+- Adapter factory behavior
+- Device status and offline handling
+- Group filtering
+- Full mock integration workflows
+- Timeout and output limits
+- Credential validation and redaction
+- Protocol response parsing and size limits
+- Action-specific response validation
+- Rate limiting
+- Concurrency limiting
+- Live-runner scope safety
+- MeshCtrl client behavior
+
+---
+
+## 8. Protocol compatibility correction
+
+The first live-compatible implementation was based on an incorrect assumption about parts of MeshCentral's wire format. The exact deployed-version-pinned `meshctrl.js` source was then compared against the bridge.
+
+The reference was MeshCentral `1.2.5`, associated with commit:
+
+`b09b1d60328de8e2d2f7387561ed51c64011c82a`
+
+The comparison produced `PROTOCOL-COMPARISON.md` and identified the following corrections.
+
+### 8.1 Authentication header format
+
+The earlier bridge assumption treated the credentials as one Base64-encoded combined string. The official client uses comma-separated Base64 fields for the username and password, with an optional third Base64 field for a TOTP token.
+
+The bridge was corrected to match the official wire format while retaining its own validation and redaction safeguards.
+
+### 8.2 Response correlation
+
+The earlier bridge used a numeric request ID model. The official control client uses a `responseid` value, with the response echoing that value.
+
+The bridge was changed to generate unique `responseid` values and correlate responses using the official field.
+
+### 8.3 Server-version response shape
+
+The bridge initially expected fields that did not match the official response shape. The compatibility work updated validation to accept the protocol envelope used by MeshCentral, including the expected result/tags-style fields.
+
+### 8.4 Handshake behavior
+
+The comparison found no special application-level handshake message required before the first request. The request is sent after the WebSocket `open` event.
+
+The bridge therefore did not add an unnecessary extra handshake. It retained the safer TLS behavior rather than copying the official client's certificate-verification weakness.
+
+### 8.5 Why this correction mattered
+
+The initial implementation could establish the WebSocket upgrade but was not fully wire-compatible for request/response correlation. The protocol correction was covered by local tests before the next live attempt.
+
+---
+
+## 9. Live-test runner and authorization
+
+A fixed-scope live runner was added as `implementation/src/live-readonly-test.ts`, exposed through:
+
+```text
+npm run live-test
+```
+
+The runner was intentionally designed not to accept arbitrary target, action, or credential command-line arguments. It contains fixed constants for:
+
+- MeshCentral URL
+- Integration username
+- Approved group name
+- Approved device name
+- Protected credential-file location
+
+Its operation sequence is:
+
+1. Read the protected password file and verify restrictive permissions.
+2. Connect using the real MeshCentral client.
+3. Request the server version.
+4. Locate the approved `Servers` group.
+5. Request nodes for that group.
+6. Resolve the approved `PVE3` device.
+7. Request device metadata for that device only.
+8. Validate scope and response shapes.
+9. Disconnect cleanly.
+
+The runner's safety test checks that it references only:
+
+- `serverversion`
+- `meshes`
+- `nodes`
+- `deviceinfo`
+
+It also checks that the source does not expose `runcommand`, general diagnostics, or `process.argv` handling.
+
+Rob authorized project-level read-only MeshCentral access on 2026-08-26. That authorization covers the documented compatibility probe only. It does not authorize commands, diagnostics, power operations, file transfer, administration, state changes, or any target other than PVE3.
+
+---
+
+## 10. Live testing: what happened
+
+### 10.1 What succeeded
+
+The bounded live probe demonstrated that:
+
+- The external WebSocket endpoint was reachable.
+- TLS negotiation completed.
+- Certificate validation succeeded.
+- The WebSocket upgrade/setup reached the MeshCentral service.
+
+This ruled out a simple DNS, network, certificate, or basic WebSocket-upgrade failure.
+
+### 10.2 What failed
+
+The server closed the control session before the first `serverversion` request could complete. The bridge's health check returned false. A direct client diagnostic showed the socket opening and then becoming unusable after the server-side close.
+
+The failure was classified at the safe `serverversion` stage as a protocol-or-connection failure. The exact server-provided close values were intentionally not preserved in project notes because server text can contain sensitive operational details.
+
+### 10.3 What did not happen
+
+No successful live operation progressed beyond the first server-version stage. Specifically, the live attempts did **not** complete:
+
+- Group enumeration
+- Device inventory
+- PVE3 metadata retrieval
+- Diagnostics
+- Commands
+- Shell access
+- File transfer
+- Power operations
+- Administrative changes
+- Any state change
+- Any access to PVE1 or PVE2
+
+No password, auth header, cookie, session token, or raw server error text was written to logs or project files.
+
+### 10.4 Three-attempt stop rule
+
+The live probe failed three consecutive times at the same safe stage. The project had a configured maximum of three consecutive live failures, so live testing stopped automatically at that point.
+
+The failure count must not be reset simply to retry the same hypothesis. A future attempt should only occur after the protocol/connection hypothesis has changed and the retry decision has been explicitly reviewed.
+
+---
+
+## 11. MeshCentral-side evidence collected
+
+The project recorded the following facts about the MeshCentral LXC and service:
+
+- `meshcentral.service` was active.
+- The main process was the expected Node.js MeshCentral process under `/opt/meshcentral`.
+- The service was running in production mode.
+- No debug flags were present in the service start command.
+- MeshCentral data files were present under the expected data directory.
+- A normal log file was not found through the provided search.
+- `journalctl` around the test window did not provide useful service entries.
+
+No MeshCentral service configuration was changed, and debug logging was not enabled. As a result, the server currently does not provide enough operational detail to identify exactly why the control channel closes before `serverversion`.
+
+---
+
+## 12. Worker/watchdog orchestration attempt
+
+Because this was being treated as a multi-step supervised project, a durable progress/watchdog arrangement was attempted.
+
+The project added:
+
+- `PROJECT-PROGRESS.json` as persisted authoritative state
+- `worker/run-worker.js`
+- `worker/worker.log`
+- A progress/watchdog cron arrangement
+
+The worker was intended to run local verification, execute the bounded probe, checkpoint status, and stop at the configured live-failure limit.
+
+There were problems with observability and orchestration:
+
+- One-shot subagent workers exited before completing all intended checkpoints.
+- A detached Node worker did execute verification and live probes, but child output was buffered and checkpoint logging was incomplete, leaving `worker.log` empty.
+- Watchdog cron setup had intermittent timeouts.
+- The persisted JSON state was therefore treated as more authoritative than the worker log.
+
+At the pause point, the progress state recorded three live failures, the `serverversion` blocker, and the stop-rule decision. The progress cron was disabled at Rob's request and preserved rather than deleted.
+
+---
+
+## 13. Documentation and review artifacts created
+
+The project now contains the following key records:
+
+- `README.md` — project vision, workflow, non-goals, and documentation map
+- `PROJECT.md` — goals, phases, scope boundaries, and multi-agent workflow
+- `SECURITY.md` — threat model, security requirements, and action classes
+- `DECISIONS.md` — architecture decisions
+- `TODO.md` — milestone and live-test checklist
+- `research/INTEGRATION-RESEARCH.md` — integration-surface research and protocol/security findings
+- `review/REVIEW.md` — early independent review of the under-hardened implementation
+- `review/LIVE-ADAPTER-REVIEW.md` — independent live-adapter security review
+- `review/FINAL-HARDENING-REVIEW.md` — reconciliation of early findings against current source
+- `PROTOCOL-COMPARISON.md` — exact MeshCentral 1.2.5 protocol comparison
+- `DEPLOYMENT.md` — deployment facts, secret handling, live gates, and rollback
+- `LIVE-TEST-PROCEDURE.md` — fixed-scope read-only test procedure and stop conditions
+- `RESUME-HANDOFF.md` — resume instructions from the earlier checkpoint
+- `PROJECT-STATUS-2026-08-25.md` — detailed project status snapshot
+- `SESSION-CHECKPOINT-2026-08-26.md` — final pause state and worker findings
+- `PROJECT-PROGRESS.json` — current machine-readable progress state
+- `implementation/README.md` — implementation usage, layout, and scope
+- `story-so-far.md` — this historical narrative
+
+---
+
+## 14. Current implementation state
+
+### Working locally
+
+- Typed contracts and Zod validation
+- Mock adapter
+- Real MeshCentral WebSocket client
+- Real read-only adapter methods
+- Credential validation
+- TLS certificate validation
+- Optional custom CA support
+- Protocol and response validation
+- Response-size limits
+- Request/response correlation using `responseid`
+- Device/group allowlists
+- Timeout and bounded retry behavior
+- Token-bucket rate limiting
+- Bounded concurrent requests
+- Structured logging and redaction
+- Audit event generation
+- Fixed-scope live runner
+- Automated tests/build/lint
+
+### Deliberately not implemented or not approved
+
+- Real diagnostics
+- Remote command execution
+- Shell access
+- File transfer
+- Power operations
+- Reboots/shutdowns
+- Service changes
+- User/group administration
+- Autonomous actions
+- Production-cluster targeting
+- Public bridge exposure
+- Automatic reconnect/session renewal for production fleet use
+
+The absence of automatic reconnect is currently safer for the first test because a dropped session fails instead of autonomously replaying requests. Separate connection/request timeout design and production-grade reconnect behavior remain deferred items.
+
+---
+
+## 15. Current status at pause
+
+The local code is verification-passing, but live MeshCentral compatibility is unresolved.
+
+The authoritative current state is:
+
+- **Project:** `meshcentral-openclaw-bridge`
+- **Status:** blocked / paused
+- **Local tests:** passing at the latest recorded checkpoint
+- **Build:** passing
+- **Lint:** passing with no warnings
+- **Live failures:** 3 consecutive
+- **Failure stage:** `serverversion`
+- **Failure category:** protocol-or-connection
+- **Live testing:** stopped by the configured failure-limit rule
+- **Diagnostics:** mock-only
+- **Remote commands/state changes:** not implemented and prohibited
+- **Approved live target:** PVE3 only
+- **Production PVE1/PVE2:** off-limits
+- **Progress cron:** disabled at Rob's request
+- **Worker:** no active worker at the checkpoint
+
+The project is not ready to claim a successful live integration. It is accurate to call it a hardened local read-only MVP with an unresolved MeshCentral 1.2.5 control-channel compatibility problem.
+
+---
+
+## 16. What needs to happen next
+
+When Rob explicitly asks to resume, the next work should be:
+
+1. Review this document, `SESSION-CHECKPOINT-2026-08-26.md`, `PROJECT-PROGRESS.json`, `RESUME-HANDOFF.md`, `PROTOCOL-COMPARISON.md`, `review/FINAL-HARDENING-REVIEW.md`, and `LIVE-TEST-PROCEDURE.md`.
+2. Inspect the exact sanitized close/error behavior without exposing raw sensitive server text.
+3. Compare the deployed MeshCentral 1.2.5 server behavior and bundled client more deeply, especially authentication acceptance and the first control-channel request.
+4. Repair the worker/checkpoint logging if durable background execution is needed again.
+5. Add or update local protocol fixtures for the revised hypothesis.
+6. Run the full local tests, TypeScript build, and lint.
+7. Do not reset the three-failure count or retry the live probe until the hypothesis has materially changed and the retry policy is reviewed.
+8. If a revised live test is authorized, keep it limited to `serverversion`, `meshes`, `nodes`, and `deviceinfo` for PVE3 only.
+9. Do not add diagnostics, commands, or state-changing actions as a way to troubleshoot the current connection problem.
+
+The central unresolved question is: **why does MeshCentral accept the TLS/WebSocket setup but close the control session before completing the first `serverversion` request?**
+
+That is where the project currently stands.