initial: split from gov-agreg — vreau.digital standalone platform
Moved from gov-agreg/src/pages/achizitii/* to root (drop prefix). - 22 pages migrated, 127 files total - All internal links: /achizitii/X → /X (176 occurrences fixed) - AchizitiiLayout subnav rewritten: /X paths, top-right link to vreaudigital.ro hub - BaseLayout new (vreau.digital branding, OG tags, site URL) - astro.config.mjs: site https://vreau.digital, server output (was static) - docker-compose: port 5096 (vreaudigital is 5095), container vreau-digital - deploy.sh: paths /opt/vreau-digital, log /var/log/vreau-digital-deploy.log Backend shared with gov-agreg: - PostgreSQL satra (same schemas: seap, firms, anaf, anre, ...) - Photon, Martin tiles - Infisical /vreaudigital path (DATABASE_URL etc. shared) build: PASS (npx astro check 0 errors, npm run build 5s vite + 10s server)
This commit is contained in:
@@ -0,0 +1,84 @@
|
||||
# WSP Daily Sync — Deployment on satra (Docker)
|
||||
|
||||
The WSP scraper deploys as a Docker container on satra. The container exits
|
||||
after each run; a cron entry triggers it daily at 06:00.
|
||||
|
||||
## One-time setup
|
||||
|
||||
### 1. Sync code + cert from orchi
|
||||
```bash
|
||||
rsync -av --exclude='.venv' --exclude='__pycache__' --exclude='*.log' \
|
||||
/home/orchestrator/Code/gov-agreg/services/seap-scraper/ \
|
||||
satra:/opt/vreaudigital/services/seap-scraper/
|
||||
```
|
||||
|
||||
### 2. Create env file on satra
|
||||
```bash
|
||||
ssh satra "sudo mkdir -p /opt/wsp && sudo chown bulibasa:bulibasa /opt/wsp && chmod 700 /opt/wsp"
|
||||
|
||||
# From orchi (don't echo values to logs):
|
||||
( echo "DATABASE_URL=$(ssh satra 'grep ^DATABASE_URL /opt/architools/.env | cut -d= -f2-' | sed 's|?schema=[^&]*||;s|?$||')" && \
|
||||
source ~/Code/claude-dotfiles/load-infisical-path.sh /seap >/dev/null 2>&1 && \
|
||||
echo "SEAP_USER=$SEAP_USER" && \
|
||||
echo "SEAP_PASS=$SEAP_PASS" && \
|
||||
echo "SEAP_CERT_KEY=$SEAP_CERT_KEY" \
|
||||
) | ssh satra "tee /opt/wsp/.env >/dev/null && chmod 600 /opt/wsp/.env"
|
||||
```
|
||||
|
||||
### 3. Build the image on satra
|
||||
```bash
|
||||
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
|
||||
docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env build'
|
||||
```
|
||||
|
||||
### 4. Test run (manual)
|
||||
```bash
|
||||
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
|
||||
docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
|
||||
python -m wsp.runner status'
|
||||
```
|
||||
|
||||
### 5. Install cron entry
|
||||
```bash
|
||||
ssh satra 'echo "0 6 * * * bulibasa cd /opt/vreaudigital/services/seap-scraper && \
|
||||
docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
|
||||
>> /var/log/wsp-incremental.log 2>&1" | sudo tee /etc/cron.d/wsp-incremental'
|
||||
ssh satra 'sudo chmod 644 /etc/cron.d/wsp-incremental'
|
||||
```
|
||||
|
||||
## Manual operation
|
||||
|
||||
### Check status
|
||||
```bash
|
||||
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
|
||||
docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
|
||||
python -m wsp.runner status'
|
||||
```
|
||||
|
||||
### Run incremental for one op
|
||||
```bash
|
||||
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
|
||||
docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
|
||||
python -m wsp.runner incremental SU_CaNotices'
|
||||
```
|
||||
|
||||
### Refresh materialized views (after sync)
|
||||
```bash
|
||||
ssh satra 'docker exec architools_postgres psql -U architools_user -d architools_db \
|
||||
-c "SELECT seap.refresh_wsp_views()"'
|
||||
```
|
||||
|
||||
## Backfill (one-time, large)
|
||||
|
||||
Run from **orchi** (5 workers, 12 months, 1-2h):
|
||||
```bash
|
||||
. /tmp/wsp_env.sh && cd ~/Code/gov-agreg/services/seap-scraper && \
|
||||
./.venv/bin/python -m wsp.runner backfill SU_CaNotices --start 2025-05-06 --end 2026-05-06 --workers 5
|
||||
```
|
||||
|
||||
Or from satra container:
|
||||
```bash
|
||||
ssh satra 'cd /opt/vreaudigital/services/seap-scraper && \
|
||||
docker compose -f wsp-docker-compose.yml --env-file /opt/wsp/.env run --rm wsp-incremental \
|
||||
python -m wsp.runner backfill SU_CaNotices --start 2025-05-06 --end 2026-05-06 --workers 5'
|
||||
```
|
||||
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=SEAP WSP daily incremental sync
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=bulibasa
|
||||
Group=bulibasa
|
||||
WorkingDirectory=/opt/vreaudigital/services/seap-scraper
|
||||
ExecStart=/opt/vreaudigital/services/seap-scraper/wsp/cron.sh
|
||||
Nice=10
|
||||
TimeoutStartSec=2h
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=SEAP WSP daily incremental sync (06:00)
|
||||
Requires=wsp-incremental.service
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 06:00:00
|
||||
RandomizedDelaySec=15min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user