The installer was adding node hostnames (and accidentally the header "Name")
to allowed_nodes in addition to IPs. This caused:
1. Invalid entries like "Name", "minipc", "delly" in config
2. These are not valid for SSH temperature collection
Only IPs should be in allowed_nodes since that's what the proxy uses for SSH.
Removed the loop that added CLUSTER_NODE_NAMES to the array.
Also fixed: Removed extraction of CLUSTER_NODE_NAMES since it's no longer used.
Problem:
Cleanup script uses systemd-run with both --wait and
--property="After=pulse-sensor-cleanup.service", creating a circular
dependency:
- cleanup.service runs and waits for uninstaller to complete
- uninstaller has After=cleanup.service, so it waits for cleanup to finish
- Result: Both services stuck waiting for each other
Fix:
Remove the --property="After=pulse-sensor-cleanup.service" line. The
Conflicts=pulse-sensor-proxy.service is sufficient to ensure the proxy
stops before uninstallation. The cleanup script doesn't need to finish
before the uninstaller starts.
Testing:
Cleanup now completes successfully, removing all artifacts:
- Systemd units removed
- Binaries deleted from /opt/pulse/sensor-proxy/
- Data directory /var/lib/pulse-sensor-proxy/ removed
- SSH keys cleaned from authorized_keys
- pulse-monitor user and API tokens deleted
- LXC bind mounts removed from container configs
Related to #605 (temperature monitoring cleanup)
**Missed Migration**:
- Line 2204 still used /usr/local/bin/pulse-sensor-wrapper.sh in fallback path
- Updated to use /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh
**Backward Compatibility**:
- When pushing SSH keys to cluster nodes, installer now checks if remote node
has old installation (/usr/local/bin wrapper exists but /opt path doesn't)
- Automatically creates symlink on remote nodes to maintain compatibility
- Prevents temperature collection failures when cluster has mixed old/new installs
**Root Cause**:
When installer runs on upgraded node (delly), it pushes SSH keys with new forced
command path to all cluster nodes. If remote node (minipc) has old installation,
the forced command fails because wrapper doesn't exist at new path.
This fix ensures "it works straight out the box" by bridging old and new paths
automatically during SSH key deployment.
Rewrote AWK state machine to correctly handle:
- Multiple allowed_nodes sections (removes all of them)
- Comment lines immediately preceding allowed_nodes (discards them)
- Empty lines within allowed_nodes section
- Indented list items and comments
The function now:
1. Buffers comment lines that might precede allowed_nodes
2. When allowed_nodes: is detected, discards buffered comments
3. Skips all content until hitting a non-indented, non-comment line
4. Flushes buffered comments when hitting non-comment content
This ensures running the installer multiple times won't create duplicate
allowed_nodes sections in config.yaml.
Tested with script that verifies duplicate sections are removed correctly.
**LXC Bind Mount Removal**:
- Changed from sed to `pct set -delete` for safer mount removal
- Validates syntax and prevents breaking container configs
- Finds mount points by grepping for pulse-sensor-proxy, extracts mp number
**API Token Parsing** (three-tier fallback):
1. Try `pveum --output-format json` with python3 JSON parsing
2. Fall back to `pvesh get /access/users/pulse-monitor@pam/token` JSON API
3. Last resort: parse table output with improved filtering (handles more Unicode chars)
**Retry Logic**:
- Rename cleanup-request.json to .processing instead of deleting immediately
- Allows retry on failure (processing file persists if script crashes)
- Remove .processing file only on successful completion
- Prevents loops while enabling failure recovery
These complete all 8 issues identified by Codex review.
**Host Detection**:
- Now detects localhost by hostname and FQDN, not just IP
- Fixes issue where nodes configured as https://hostname:8006 would skip
localhost cleanup (API tokens, bind mounts, service removal)
**Systemd Sandbox**:
- Added /etc/pve and /etc/systemd/system to ReadWritePaths
- Allows cleanup script to modify Proxmox configs and systemd units
**Uninstaller Improvements**:
- Use UUID for transient unit names (prevents same-second collisions)
- Added --purge flag for complete removal
- Added --wait and --collect flags to capture exit code
- Now fails cleanup if uninstaller exits non-zero
**Path Migration**:
- Fixed all /usr/local references to use /opt/pulse/sensor-proxy
- Updated forced command in SSH authorized_keys
- Updated self-heal script installer path
- Updated Go backend removal helpers (supports both new and legacy paths)
These fixes address Codex findings: hostname detection, sandbox permissions,
transient unit collisions, incomplete purging, and incomplete path migration.
Related to cleanup implementation testing.
After relocating binaries to INSTALL_ROOT, the SHARE_DIR variable was removed
but one reference remained in cache_installer_for_self_heal() causing
'unbound variable' error.
Changed to use INSTALL_ROOT directly since that's where the cached installer
is stored (STORED_INSTALLER=${INSTALL_ROOT}/install-sensor-proxy.sh).
The installer was trying to write binaries to /opt/pulse/sensor-proxy/bin/
before creating the directory structure, causing 'No such file or directory'
errors on fresh installs.
Moved directory creation for INSTALL_ROOT and bin/ to before binary installation
section (before line 657), ensuring directories exist before use.
Related to cleanup implementation testing.
Extends cleanup script to completely remove Pulse footprint from hosts
when nodes are removed, not just SSH keys. Now removes: SSH keys, proxy
service, binaries, API tokens, pulse-monitor user, and LXC bind mounts.
Key improvements:
1. **flock Serialization**: Prevents concurrent cleanup runs
- Acquires exclusive lock on cleanup.lock file
- Prevents race conditions and cleanup loops
2. **Immediate Request File Deletion**: Delete cleanup-request.json
before any long-running operations to prevent re-triggering
3. **API Token Cleanup**: Removes all pulse-monitor@pam API tokens
- Tries JSON output first (Proxmox 7.0+)
- Falls back to table parsing with proper filtering (no decoration chars)
- Deletes pulse-monitor@pam user after removing all tokens
4. **LXC Bind Mount Removal**: Scans all container configs and removes
pulse-sensor-proxy bind mount entries
5. **Process Isolation for Uninstaller**: Uses systemd-run to spawn
isolated transient unit that won't be killed when proxy service stops
- Unit name: pulse-uninstall-{timestamp}
- Properties: Type=oneshot, Conflicts=pulse-sensor-proxy.service
- Runs non-blocking so cleanup service can exit cleanly
- Falls back to direct call if systemd-run unavailable
6. **Complete Service/Binary Removal**: Calls installer's --uninstall
- Stops and disables pulse-sensor-proxy.service
- Removes all systemd units
- Deletes all binaries from /opt/pulse/sensor-proxy/
- Removes configuration files
- Cleans up directories
Changes to cleanup script logic:
- Added LOCKFILE and INSTALLER_PATH configuration
- Acquire flock before processing (prevents concurrent runs)
- Delete request file immediately after reading
- Full localhost cleanup: SSH keys → API tokens → bind mounts → uninstall
- Remote cleanup still SSH-key-only (can't orchestrate uninstall remotely)
- Better error handling with appropriate log levels
Updated cleanup service unit:
- ExecStart now uses ${CLEANUP_SCRIPT_PATH} variable (new /opt location)
- Changed heredoc from 'SERVICE_EOF' to SERVICE_EOF for variable expansion
Addresses all issues documented in CLEANUP_TODO.md:
- ✅ Read-only filesystem (binaries now in /opt, removable)
- ✅ Process isolation (systemd-run transient unit)
- ✅ Cleanup loops (flock + immediate file deletion)
- ✅ API token parsing (JSON first, filtered table fallback)
The UI message is now accurate: "Removing this proxmox ve node also
scrubs the Pulse footprint on the host — the proxy service, SSH key,
API token, and bind mount are all cleaned up automatically."
Part of: CLEANUP_TODO.md Phase 2-4
Supersedes: ed65fda74 (original cleanup attempt with process issues)
Depends on: b192c60e9 (binary relocation to /opt)
Moves all sensor-proxy binaries and scripts from /usr/local/bin to
/opt/pulse/sensor-proxy/bin to ensure they can be removed during cleanup
even on systems with read-only /usr (hardened Proxmox setups).
Changes:
- INSTALL_ROOT=/opt/pulse/sensor-proxy (new writable location)
- Binary path: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy
- Wrapper script: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh
- Cleanup script: /opt/pulse/sensor-proxy/bin/pulse-sensor-cleanup.sh
- Selfheal script: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy-selfheal.sh
- Installer storage: /opt/pulse/sensor-proxy/install-sensor-proxy.sh
Updated:
- Directory creation to include ${INSTALL_ROOT}/bin
- Systemd service ExecStart paths to use ${BINARY_PATH}
- Self-heal service ExecStart to use ${SELFHEAL_SCRIPT}
- Changed heredoc delimiters from 'EOF' to EOF for variable expansion
Rationale:
Proxmox VE can mount /usr as read-only in hardened configurations.
The previous /usr/local/bin location prevented complete uninstallation
on these systems, violating Pulse's correctness principle. The /opt
location is guaranteed writable and appropriate for third-party software.
This is Phase 1 of implementing full cleanup functionality per
CLEANUP_TODO.md. Subsequent commits will add process isolation,
API token deletion, and bind mount removal.
Part of: #CLEANUP_TODO.md Phase 1
Related: ed65fda74 (original cleanup attempt)
The update_allowed_nodes function was changing ownership of the temp file
before all writes were complete, causing 'Permission denied' errors when
appending the allowed_nodes section.
Root cause:
- mktemp creates file owned by script runner (root)
- chown changed ownership to pulse-sensor-proxy:pulse-sensor-proxy
- Subsequent append (>>) failed because root can't write to the file
Fix: Defer chown until after all writes complete and file is moved to
final location. Ownership is still correctly set on the final config file.
When a Proxmox node is removed from Pulse, the cleanup now performs full uninstallation:
- SSH keys removal (existing functionality)
- Uninstalls pulse-sensor-proxy service
- Removes LXC bind mounts from container configs
- Deletes Proxmox API tokens
- Removes pulse-monitor@pam user
This aligns with security best practices and user expectations - "remove node"
should completely sever trust with that machine, not leave credentials and
privileged services behind.
The cleanup script now calls the uninstaller (--uninstall) and uses pveum
to remove API tokens. This prevents leftover artifacts if the host is
repurposed or compromised.
Related: config_handlers.go triggerPVEHostCleanup() at node deletion
- Change mktemp to use /tmp/pulse-config.XXXXXXXXXX template
- Prevents conflicts with stale temp files from previous runs
- Fixes 'Permission denied' errors when script re-runs
- Add trap to remove temp file on function return (success or failure)
- Add error check for mv command with descriptive message
- Ensure config file has proper permissions after update
This prevents orphaned temp files when errors occur and provides
better diagnostics when file operations fail.
The all_nodes arrays were declared with 'local' keyword outside of
functions, causing bash syntax error:
'local: can only be used in a function'
Fixed by removing 'local' keyword - arrays in main script scope don't
need it and it's actually invalid syntax.
The awk logic was removing allowed_nodes sections but leaving their
comment headers behind. When multiple sections existed, comments would
accumulate.
New approach:
- Buffer all comment lines encountered outside sections
- When a non-comment line is found, flush buffered comments
- When allowed_nodes is found, discard buffered comments (they belonged
to the section we're removing)
- This cleanly removes section headers like:
'# Cluster nodes (auto-discovered during installation)'
'# These nodes are allowed to request...'
Tested with config containing duplicate allowed_nodes sections - now
correctly produces clean output with all duplicates and headers removed.
The installer was only creating base config.yaml in standalone mode,
but update_allowed_nodes() is also called in LXC mode. When the config
didn't exist, update_allowed_nodes() would create an empty file and only
add the allowed_nodes section, missing required fields like
allowed_peer_uids, metrics_address, rate_limit, etc.
This caused the proxy to fail when it tried to parse the incomplete config.
Now creates a proper base config with all required fields if the file
doesn't exist, before any mode-specific configuration is added.
The install-sensor-proxy.sh script was blindly appending allowed_nodes
sections to the config file without checking if they already existed.
When the script was re-run or if the initial config already had an
allowed_nodes section, this created duplicate YAML keys that caused
the proxy service to fail with parse errors.
Changes:
- Add update_allowed_nodes() helper function that safely updates the
allowed_nodes section by removing any existing ones first
- Replace all three cat >> config.yaml heredocs with calls to the
helper function (cluster nodes, standalone mode, pvecm fallback)
- Uses awk to properly parse and remove multi-line YAML sections
This makes the installer idempotent and prevents config corruption on
re-runs.
Fixes issue where proxy service crashed with:
'mapping key "allowed_nodes" already defined at line X'
The HTTP mode installer now includes 127.0.0.1/32 in allowed_source_subnets
to permit self-monitoring queries from localhost. This fixes 403 Forbidden
errors when nodes query their own sensor-proxy instance.
Related to HTTP mode implementation for external PVE hosts.
## HTTP Server Fixes
- Add source IP middleware to enforce allowed_source_subnets
- Fix missing source subnet validation for external HTTP requests
- HTTP health endpoint now respects subnet restrictions
## Installer Improvements
- Auto-configure allowed_source_subnets with Pulse server IP
- Add cluster node hostnames to allowed_nodes (not just IPs)
- Fix node validation to accept both hostnames and IPs
- Add Pulse server reachability check before installation
- Add port availability check for HTTP mode
- Add automatic rollback on service startup failure
- Add HTTP endpoint health check after installation
- Fix config backup and deduplication (prevent duplicate keys)
- Fix IPv4 validation with loopback rejection
- Improve registration retry logic with detailed errors
- Add automatic LXC bind mount cleanup on uninstall
## Temperature Collection Fixes
- Add local temperature collection for self-monitoring nodes
- Fix node identifier matching (use hostname not SSH host)
- Fix JSON double-encoding in HTTP client response
Related to #XXX (temperature monitoring fixes)
When cluster IPC validation fails (due to systemd hardening), the proxy
falls back to allowlist-based validation. The installer now automatically
populates allowed_nodes with:
- Cluster mode: All discovered cluster member IPs
- Standalone mode: localhost IP addresses (including 127.0.0.1/localhost)
- Fallback mode: localhost IPs when pvecm unavailable
This ensures out-of-the-box temperature monitoring works on fresh installs
without manual configuration.
Root cause: pulse-sensor-proxy runs with strict systemd hardening that prevents
access to Proxmox corosync IPC (abstract UNIX sockets). When pvecm fails with
IPC errors, the code incorrectly treated it as "standalone mode" and only
discovered localhost addresses, rejecting legitimate cluster members and external
nodes.
Changes:
1. **Distinguish IPC failures from true standalone mode**
- Detect ipcc_send_rec and access control list errors specifically
- These indicate a cluster exists but isn't accessible (LXC, systemd restrictions)
- Return error to disable cluster validation instead of misusing standalone logic
2. **Graceful degradation when cluster validation fails**
- When cluster IPC is unavailable, fall through to permissive mode
- Log debug message suggesting allowed_nodes configuration
- Allows requests to proceed rather than blocking all temperature monitoring
3. **Improve local address discovery for true standalone nodes**
- Use Go's native net.Interfaces() instead of shelling out to 'ip addr'
- More reliable and works with AF_NETLINK restrictions
- Add helpful logging when only hostnames are discovered
4. **Systemd hardening adjustments**
- Add AF_NETLINK to RestrictAddressFamilies (for net.Interfaces())
- Remove RemoveIPC=true (attempted fix for corosync, insufficient)
- Add ReadWritePaths=-/run/corosync (optional path, corosync uses abstract sockets anyway)
Result: Temperature monitoring now works in:
- Clustered Proxmox hosts (falls back to permissive when IPC blocked)
- LXC containers (correctly detects IPC failure, allows requests)
- Standalone nodes (proper local address discovery with IPs)
Workaround for maximum security: Configure allowed_nodes in /etc/pulse-sensor-proxy/config.yaml
when cluster validation cannot be used.
Root cause: The systemd service hardening blocked AF_NETLINK sockets,
preventing IP address discovery on standalone nodes. The proxy could
only discover hostnames, causing node_not_cluster_member rejections
when users configured Pulse with IP addresses.
Changes:
1. Add AF_NETLINK to RestrictAddressFamilies in all systemd services
- pulse-sensor-proxy.service
- install-sensor-proxy.sh (both modes)
- pulse-sensor-cleanup.service
2. Replace shell-based 'ip addr' with Go native net.Interfaces() API
- More reliable and doesn't require external commands
- Works even with strict systemd restrictions
- Properly filters loopback, link-local, and down interfaces
3. Improve error logging and user guidance
- Warn when no IP addresses can be discovered
- Provide clear instructions about allowed_nodes workaround
- Include address counts in logs for debugging
This fix ensures standalone Proxmox nodes can properly validate
temperature requests by IP address without requiring manual
allowed_nodes configuration.
- Replace unreliable git fetch --dry-run check
- Use git rev-parse to compare local and remote commits
- Prevents false warnings about diverged branches
- Check VERSION file matches before triggering workflow
- Validate working directory is clean
- Confirm on main branch and up to date
- Load release notes from /tmp/release_notes_X.Y.Z.md
- Prevents wasting CI time on misconfigured releases
Users were abandoning Pulse due to catastrophic temperature monitoring setup failures. This commit addresses the root causes:
**Problem 1: Silent Failures**
- Installations reported "SUCCESS" even when proxy never started
- UI showed green checkmarks with no temperature data
- Zero feedback when things went wrong
**Problem 2: Missing Diagnostics**
- Service failures logged only in journald
- Users saw "Something going on with the proxy" with no actionable guidance
- No way to troubleshoot from error messages
**Problem 3: Standalone Node Issues**
- Proxy daemon logged continuous pvecm errors as warnings
- "ipcc_send_rec" and "Unknown error -1" messages confused users
- These are expected for non-clustered/LXC setups
**Solutions Implemented:**
1. **Health Gate in install.sh (lines 1588-1629)**
- Verify service is running after installation
- Check socket exists on host
- Confirm socket visible inside container via bind mount
- Fail loudly with specific diagnostics if any check fails
2. **Actionable Error Messages in install-sensor-proxy.sh (lines 822-877)**
- When service fails to start: dump full systemctl status + 40 lines of logs
- When socket missing: show permissions, service status, and remediation command
- Include common issues checklist (missing user, permission errors, lm-sensors, etc.)
- Direct link to troubleshooting docs
3. **Better Standalone Node Detection in ssh.go (lines 585-595)**
- Recognize "Unknown error -1" and "Unable to load access control list" as LXC indicators
- Log at INFO level (not WARN) since this is expected behavior
- Clarify message: "using localhost for temperature collection"
**Impact:**
- Eliminates "green checkmark but no temps" scenario
- Users get immediate actionable feedback on failures
- Standalone/LXC installations work silently without error spam
- Reduces support burden from #571 (15+ comments of user frustration)
Related to #571
Snap-installed Docker does not automatically create a docker group,
causing permission denied errors when the pulse-docker service user
tries to access /var/run/docker.sock.
Changes:
- Auto-detect Snap Docker installations
- Create docker group if missing when Snap Docker is detected
- Restart Snap Docker after group creation to refresh socket ACLs
- Add socket access validation before starting the service
- Handle symlinked Docker sockets in systemd unit ReadWritePaths
- Document troubleshooting steps in DOCKER_MONITORING.md
