Diagnose and resolve common issues encountered during CosmicAC staging server deployment, including PM2 restarts, missing status files, and permission errors.

Troubleshooting CosmicAC Deployment

Common Issues

PM2 Processes Keep Restarting

# Check logs for errors
pm2 logs --err

# Check specific process logs
pm2 logs app-node-0 --lines 100

Status Files Not Found

Ensure workers are started in the correct order. The status files are created when workers initialize:

# Check if status files exist
ls -la ~/cosmicac-wrk-ork/status/
ls -la ~/cosmicac-app-node/status/

Node Version Issues

# Verify you're using Node 20
node --version

# If wrong version, switch
nvm use 20

Permission Denied

# Ensure you're running as cosmicac user
whoami

# If not, switch to cosmicac
sudo -u cosmicac -i

Log Locations

PM2 logs are stored in:

~/.pm2/logs/

View all available logs:

ls -la ~/.pm2/logs/

Health Check Script

Create ~/health-check.sh:

#!/bin/bash

echo "=== CosmicAC Health Check ==="
echo ""
echo "PM2 Status:"
pm2 jlist | jq -r '.[] | "\(.name): \(.pm2_env.status)"'
echo ""
echo "Status Files:"
for dir in cosmicac-wrk-ork cosmicac-app-node cosmicac-wrk-server-k8s-nvidia cosmicac-proxy-inference; do
  if [ -d "$HOME/$dir/status" ]; then
    echo "  ✓ $dir/status exists"
  else
    echo "  ✗ $dir/status missing"
  fi
done
echo ""
echo "Ports in use:"
netstat -tlnp 2>/dev/null | grep -E ':(3000|8000)' || echo "  No relevant ports found"

chmod +x ~/health-check.sh
./health-check.sh

Troubleshooting CosmicAC Deployment

On this page