# WSP Daily Sync — Deployment on satra (Docker)

The WSP scraper deploys as a Docker container on satra. The container exits
after each run; a cron entry triggers it daily at 06:00.

## One-time setup

### 1. Sync code + cert from orchi
```bash
rsync -av --exclude='.venv' --exclude='__pycache__' --exclude='*.log' \
  /home/orchestrator/Code/gov-agreg/services/seap-scraper/ \
  satra:/opt/vreaudigital/services/seap-scraper/
```

### 2. Create env file on satra
```bash
ssh satra "sudo mkdir -p /opt/wsp && sudo chown bulibasa:bulibasa /opt/wsp && chmod 700 /opt/wsp"

# From orchi (don't echo values to logs):
( echo "DATABASE_URL=$(ssh satra 'grep ^DATABASE_URL /opt/architools/.env | cut -d= -f2-' | sed 's|?schema=[^&]*||;s|?$||')" && \
  source ~/Code/claude-dotfiles/load-infisical-path.sh /seap >/dev/null 2>&1 && \
  echo "SEAP_USER=$SEAP_USER" && \
  echo "SEAP_PASS=$SEAP_PASS" && \
  echo "SEAP_CERT_KEY=$SEAP_CERT_KEY" \
) | ssh satra "tee /opt/wsp/.env >/dev/null && chmod 600 /opt/wsp/.env"
```

### 3. Build the image on satra
```bash
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
  docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env build'
```

### 4. Test run (manual)
```bash
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
  docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
  python -m wsp.runner status'
```

### 5. Install cron entry
```bash
ssh satra 'echo "0 6 * * * bulibasa cd /opt/vreaudigital/services/seap-scraper && \
  docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
  >> /var/log/wsp-incremental.log 2>&1" | sudo tee /etc/cron.d/wsp-incremental'
ssh satra 'sudo chmod 644 /etc/cron.d/wsp-incremental'
```

## Manual operation

### Check status
```bash
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
  docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
  python -m wsp.runner status'
```

### Run incremental for one op
```bash
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
  docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
  python -m wsp.runner incremental SU_CaNotices'
```

### Refresh materialized views (after sync)
```bash
ssh satra 'docker exec architools_postgres psql -U architools_user -d architools_db \
  -c "SELECT seap.refresh_wsp_views()"'
```

## Backfill (one-time, large)

Run from **orchi** (5 workers, 12 months, 1-2h):
```bash
. /tmp/wsp_env.sh && cd ~/Code/gov-agreg/services/seap-scraper && \
./.venv/bin/python -m wsp.runner backfill SU_CaNotices --start 2025-05-06 --end 2026-05-06 --workers 5
```

Or from satra container:
```bash
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
  docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
  python -m wsp.runner backfill SU_CaNotices --start 2025-05-06 --end 2026-05-06 --workers 5'
```