Pulse/docs/MULTI_TENANT.md

Multi-Tenant Feature Documentation

Status: Disabled by Default

This feature is gated behind a feature flag and license check. It will not affect existing users unless explicitly enabled.

---

How to Enable

Requirements

1. Feature flag: Set environment variable

   PULSE_MULTI_TENANT_ENABLED=true
   

2. License: Enterprise license with multi_tenant feature enabled

Behavior Without Enablement

Condition	HTTP Response	WebSocket Response
Feature flag disabled	501 Not Implemented	501 Not Implemented
Flag enabled, no license	402 Payment Required	402 Payment Required
Flag enabled + licensed	Normal operation	Normal operation

The "default" organization always works regardless of feature flag or license status.

---

What's Implemented

Tenant Isolation

Component	Status	Details
State/Monitor	✅	Each org gets its own Monitor instance via MultiTenantMonitor
WebSocket	✅	Clients bound to tenant, broadcasts filtered by org
Audit Logs	✅	LogAuditEventForTenant() writes to per-org audit DB
Resources	✅	Per-tenant resource stores with PopulateFromSnapshotForTenant()
Persistence	✅	MultiTenantPersistence provides per-org config directories

Gating & Authorization

Component	Status	Details
Feature flag	✅	PULSE_MULTI_TENANT_ENABLED env var (default: false)
License check	✅	Requires multi_tenant feature in Enterprise license
HTTP middleware	✅	TenantMiddleware extracts org ID, validates access
WebSocket gating	✅	MultiTenantChecker validates before upgrade
Token authorization	✅	AuthorizationChecker.TokenCanAccessOrg()
User authorization	✅	AuthorizationChecker.UserCanAccessOrg() via org membership

Tenant-Aware Endpoints

All user-facing data endpoints use getTenantMonitor(ctx):

• /api/state
• /api/charts
• /api/storage/{id}
• /api/backups, /api/backups/pve, /api/backups/pbs
• /api/snapshots
• /api/resources/*
• /api/metrics/*

---

Storage Layout and Migration

• The default org uses the root data dir for backward compatibility.
• Non-default orgs store data in /orgs/<org-id>/.
• When multi-tenant is enabled, legacy single-tenant data is migrated into /orgs/default/ and symlinks are created in the root data dir for compatibility.
• Organization metadata is stored in org.json inside each org directory.

---

Intentionally Global (Admin-Level)

These endpoints show system-wide data regardless of tenant context:

Endpoint	Rationale
/api/health	System uptime, not tenant-specific
/api/scheduler/health	Process-level scheduler status
/api/diagnostics/*	Admin diagnostics for full system

Also global:

• security_setup_fix.go - Clears unauthenticated agents on default monitor

---

Architecture

Key Files

File	Purpose
internal/api/middleware_tenant.go	Extracts org ID, validates access, injects context
internal/api/middleware_license.go	Feature flag, license check, 501/402 responses
internal/api/authorization.go	AuthorizationChecker interface, token/user access checks
internal/monitoring/multi_tenant_monitor.go	Per-org monitor instances
internal/config/multi_tenant.go	Per-org persistence (config directories)
internal/websocket/hub.go	Tenant-aware client tracking, MultiTenantChecker
pkg/server/server.go	Wires up org loader, multi-tenant checker

Request Flow

Request
  │
  ├─► TenantMiddleware
  │     ├─► Extract org ID (header/cookie/default)
  │     ├─► Feature flag check (501 if disabled)
  │     ├─► License check (402 if unlicensed)
  │     ├─► Authorization check (403 if denied)
  │     └─► Inject org ID into context
  │
  ├─► Handler
  │     └─► getTenantMonitor(ctx) → org-specific Monitor
  │
  └─► Response (org-scoped data)

Org ID Sources (Priority Order)

1. X-Pulse-Org-ID header (API clients/agents)
2. pulse_org_id cookie (browser sessions)
3. Fallback: "default"

---

Data Model

Organization

type Organization struct {
    ID          string
    DisplayName string
    OwnerUserID string               // Creator/owner
    Members     []OrganizationMember // User membership
}

type OrganizationMember struct {
    UserID  string
    Role    string // "owner", "admin", "member"
    AddedAt time.Time
    AddedBy string
}

API Token Binding

type APITokenRecord struct {
    // ... existing fields ...
    OrgID  string   // Single org binding
    OrgIDs []string // Multi-org access (MSP tokens)
}

Legacy tokens (empty OrgID) have wildcard access during migration period.

---

TODO / Deferred Items

High Priority (Before GA)

• UI integration: Org switcher, org management screens

Medium Priority

• Org CRUD endpoints: Create/update/delete organizations via API
• Member management: Add/remove users from organizations

Low Priority / Policy Decisions

• Decide if diagnostics should be org-scoped or super-admin only
• Decide if security_setup_fix.go agent cleanup should be org-scoped

---

Testing Checklist

Unit Tests

# Tenant middleware tests
go test ./internal/api -run TestTenantMiddleware

# WebSocket multi-tenant tests
go test ./internal/websocket -run TestHandleWebSocket_MultiTenant

Manual Testing

1. Default behavior (flag disabled)

   • Start Pulse without PULSE_MULTI_TENANT_ENABLED
   • Verify normal operation
   • Attempt X-Pulse-Org-ID: test-org header → expect 501

2. Flag enabled, no license

   • Set PULSE_MULTI_TENANT_ENABLED=true
   • No Enterprise license
   • Attempt non-default org → expect 402

3. Full multi-tenant

   • Enable flag + Enterprise license
   • Create org "test-a" with PVE node A
   • Create org "test-b" with PVE node B
   • Open browser tabs for each org
   • Verify each sees only their nodes
   • Verify WebSocket updates are isolated
   • Attempt header spoofing with wrong token → expect 403

Integration Test Script

# 1. Verify default org works without flag
curl -u admin:admin http://localhost:7655/api/state
# → 200 OK

# 2. Verify non-default org blocked without flag
curl -u admin:admin -H "X-Pulse-Org-ID: test-org" http://localhost:7655/api/state
# → 501 Not Implemented

# 3. With flag enabled but no license
export PULSE_MULTI_TENANT_ENABLED=true
curl -u admin:admin -H "X-Pulse-Org-ID: test-org" http://localhost:7655/api/state
# → 402 Payment Required

---

Rollout

Verification Status

Component	Status	Method	Notes
Feature Flag	✅ Verified	Unit Test	Flag disables/enables multi-tenant access correctly
Licensing	✅ Verified	Unit Test	Unlicensed access blocked with 402 Payment Required
Migration	✅ Verified	Unit Test	Legacy data moves to default org; symlinks created
Isolation	✅ Verified	Unit Test	API State, WebSockets, and Resources respect tenant context
Security	✅ Verified	Code Audit	API Tokens and Audit Logs enforce tenant binding

Readiness Checklist

• Enterprise license loaded for the orgs that will access multi-tenant features
• PULSE_MULTI_TENANT_ENABLED=true configured in the runtime environment
• Config migration has run on startup (verify tenant layout exists in data dir)
• Org membership loader is available for session users
• API tokens for non-default orgs are bound to the org(s)
• Per-tenant audit logging is enabled (tenant audit DBs present and writable)
• Tenant config loading uses per-org nodes and credentials (no shared secrets)

Rollout Steps

1. Enable the feature flag in staging
2. Confirm enterprise license activation for a test org
3. Create a non-default org and bind a test API token to it
4. Validate:
   • 501/402 behavior for disabled/unlicensed org access
   • Success for licensed access (HTTP + WebSocket)
   • Data isolation across orgs
5. Roll out to production with monitoring for 4xx/5xx spikes

Rollback

Disable PULSE_MULTI_TENANT_ENABLED to revert non-default org access (default org unaffected).

---

Response Codes Reference

Code	Meaning	When
200	Success	Valid org access
400	Bad Request	Invalid org ID format
402	Payment Required	Feature enabled but not licensed
403	Forbidden	Token/user not authorized for org
501	Not Implemented	Feature flag disabled

---

Changelog

• 2024-01: Initial implementation
  • Feature flag gating
  • License enforcement
  • Per-tenant state isolation
  • WebSocket tenant binding
  • Audit log isolation
  • Authorization framework