Compare commits
9 Commits
f49fdb1da0
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
 Claude VM
|
aa246c2d91 | ||
|
Claude VM
|
9b66dd6452 | ||
|
Claude VM
|
ffad5bb96d | ||
|
Claude VM
|
50165d2369 | ||
|
Claude VM
|
c9f1219eaa | ||
|
Claude VM
|
1c8d7ea59c | ||
|
Claude VM
|
5ad8870dc5 | ||
|
Claude VM
|
b62132ab9e | ||
|
Claude VM
|
28c870fb12 |
@@ -0,0 +1,285 @@
|
||||
# Plan 001 — DB GIS Central Multi-App
|
||||
|
||||
**Status:** DRAFT v1 — neaprobat, de discutat
|
||||
**Autor:** Claude + Marius Tarau
|
||||
**Data:** 2026-04-18
|
||||
**Scope:** Arhitectură unificată GIS pentru ArchiTools, eTerra.live, SmartCity360, Planhub ERP + servicii viitoare
|
||||
|
||||
---
|
||||
|
||||
## 1. Context & motivație
|
||||
|
||||
Date GIS (cadastru ANCPI, enrichment, PUG/PUZ, UTR) sunt folosite de multiple platforme:
|
||||
- **ArchiTools** — portal intern Beletage (există, producție)
|
||||
- **eTerra.live** — SaaS topografi (proxy eTerra, repo nou gitadmin/eterra-live)
|
||||
- **SmartCity360** — portal GIS primării (în arhitectură)
|
||||
- **Planhub ERP** — ERP arhitecți cu geoportal (în arhitectură)
|
||||
- Viitoare servicii
|
||||
|
||||
Nevoie: **single source of truth GIS** cu acces controlat, sync centralizat, cache/CDN partajat, izolare multi-tenant.
|
||||
|
||||
---
|
||||
|
||||
## 2. Topologie
|
||||
|
||||
**All on-prem.** Nimic în cloud public.
|
||||
- Servere: satra (10.10.10.166), shop (10.10.10.84), proxy (10.10.10.199), orchi (10.10.40.60 — CJ2 via VPN)
|
||||
- Gateway public: Traefik pe proxy, Sophos DNAT 80/443
|
||||
- Cloudflare doar pentru domenii SaaS public-facing (eTerra.live, SmartCity360)
|
||||
|
||||
---
|
||||
|
||||
## 3. DB Central — Shop server (10.10.10.84)
|
||||
|
||||
**Resurse disponibile (verificat 2026-04-18):**
|
||||
- Xeon Gold 6430, 128 vCPU, 251GB RAM, 3.5TB NVMe (1% ocupat)
|
||||
- Rulează deja: Supabase stack, WordPress, Tegola
|
||||
- Load avg ~0.5, headroom masiv
|
||||
|
||||
**Cluster Postgres nou, dedicat:**
|
||||
- Container `postgres-gis` (separat de Supabase existent — izolare critică)
|
||||
- Postgres 16 + PostGIS 3.4 + pg_cron
|
||||
- SRID nativ 3844 (Stereo70)
|
||||
- Port intern, expus doar pe 10.10.10.0/24 + VPN CJ2 (10.10.40.0/24)
|
||||
- NICIODATĂ expus direct pe internet
|
||||
|
||||
### Schema split
|
||||
|
||||
| Schema | Conținut | Access |
|
||||
|---|---|---|
|
||||
| `gis_core` | Cadastru național ANCPI: UAT, terenuri, clădiri, administrativ | Public read, write doar sync daemon |
|
||||
| `gis_urban` | PUG, PUZ, PUD, UTR, zone protecție, zone reglementate | Read public (layer public), write per tenant primărie (RLS) |
|
||||
| `gis_enrichment` | CF, proprietari, adrese, categorii folosință — **GDPR sensibil** | Auth obligatoriu, RLS pe rol |
|
||||
| `gis_meta` | Sync runs, rules, status, audit | Internal only |
|
||||
| `tenants` | Primării, firme arh, clienți SaaS | Control plane |
|
||||
|
||||
### Roluri Postgres
|
||||
|
||||
- `gis_sync_rw` — write pe `gis_core` + `gis_meta` (sync daemons)
|
||||
- `gis_public_ro` — read pe `gis_core` + layer public din `gis_urban` (Martin public tiles)
|
||||
- `gis_app_ro` — read authenticated cu RLS pe `gis_urban` privat + `gis_enrichment`
|
||||
- `gis_urban_rw` — write pe `gis_urban` doar pentru tenant-ul propriu
|
||||
|
||||
### RLS pe multi-tenant
|
||||
|
||||
```sql
|
||||
-- exemplu pentru gis_urban tabele private
|
||||
CREATE POLICY tenant_isolation ON gis_urban.puz_private
|
||||
USING (siruta = ANY(current_setting('app.allowed_sirutas', true)::text[]));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Layer acces — public/auth endpoints
|
||||
|
||||
| Subdomain | Serviciu | Auth | Scop |
|
||||
|---|---|---|---|
|
||||
| `tiles.beletage.ro` | Martin (instanță dedicată) | Niciun auth pe layer public | Tile server public cadastru + PUG public |
|
||||
| `pmtiles.beletage.ro` | nginx static | Niciun auth | Overview Romania, PMTiles pre-generated |
|
||||
| `gis-api.beletage.ro` | Next.js API sau PostgREST | JWT Authentik | Enrichment, query spațial, mutations |
|
||||
| `sync.beletage.ro` | Orchestrator UI | Authentik admin | Control panel sync central |
|
||||
| `eterra.live` | SaaS topografi | Authentik | Portal topografi |
|
||||
| `planhub.beletage.ro` | ERP arhitecți | Authentik | Geoportal + PM |
|
||||
| `<primarie>.smartcity360.ro` | Portal primărie | Cetățeni anonim + admin Authentik | Subdomain per tenant |
|
||||
|
||||
**Traefik** — reverse proxy unic, SSL auto. **Cloudflare** în față pe domenii SaaS public (eTerra.live, SmartCity360) cu rate limit + CDN cache tile-uri (TTL 24h pentru UAT).
|
||||
|
||||
---
|
||||
|
||||
## 5. ACL Enrichment (per rol)
|
||||
|
||||
| Rol | Vede enrichment |
|
||||
|---|---|
|
||||
| `topograf` (eTerra.live) | Full — CF, proprietari, adrese |
|
||||
| `architect` (Planhub ERP) | Full + PUG complet |
|
||||
| `admin_primaria` (SmartCity360) | Full pentru UAT-ul primăriei + date PUG private |
|
||||
| `cetatean` (SmartCity360 public) | Doar nr. cadastral + suprafață + UTR |
|
||||
| `admin` (ArchiTools) | Tot |
|
||||
|
||||
Implementat prin RLS + `current_setting('app.role')`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Shared enrichment pool — insight cheie
|
||||
|
||||
**De-duplicare globală.** Dacă topograf A cere enrichment pe parcela X:
|
||||
1. Salvat în `gis_enrichment` central
|
||||
2. Topograf B cere aceeași parcelă → **hit instant**, zero request la eTerra
|
||||
3. TTL default 90 zile → după expire auto-refresh
|
||||
|
||||
Beneficii:
|
||||
- Economii masive de sesiuni eTerra
|
||||
- Viteză maximă pentru request-uri repetate
|
||||
- Baza de date crește cu datele "cele mai cerute" de topografi (natural prioritization)
|
||||
|
||||
**TBD:** TTL fix 90 zile vs decizie manuală a topografului pentru "force refresh"?
|
||||
|
||||
---
|
||||
|
||||
## 7. Sync workers — centralizat
|
||||
|
||||
**Extrage din ArchiTools într-un serviciu standalone** `gis-sync-orchestrator`:
|
||||
- Un singur scheduler (nu duplicat per app)
|
||||
- Workers:
|
||||
- `eterra-sync` — delta sync cadastru (nocturn automat + on-demand per UAT)
|
||||
- `enrichment-worker` — pulls CF/owners pe cerere
|
||||
- `pug-ingest` (viitor) — import PUG din shapefiles/PDFs
|
||||
- **Queue:** `pg-boss` (Postgres-backed, zero infra extra) sau BullMQ+Redis
|
||||
- **Delta sync** folosește `gis_uats.last_updated_dtm` deja existent
|
||||
- **Notificare apps** la sync complete: `Postgres LISTEN/NOTIFY` + webhook → fiecare app se abonează
|
||||
- **PMTiles rebuild** trigger-uit de orchestrator după sync major (webhook existent satra:9876)
|
||||
|
||||
### Session pool eTerra per-user
|
||||
|
||||
- Tabel `eterra_sessions` keyed by `user_id` (fiecare topograf = sesiune proprie)
|
||||
- Sync on-demand eTerra.live folosește credențialele topografului logat
|
||||
- Sync nocturn automat folosește cont service (actualul hardcoded)
|
||||
- Avantaj: toți topografii + sistemul automat contribuie la menținerea DB-ului actual
|
||||
|
||||
---
|
||||
|
||||
## 8. Consumers
|
||||
|
||||
| App | Schemas | Write | DB user | Deploy |
|
||||
|---|---|---|---|---|
|
||||
| ArchiTools | core + urban + enrichment | enrichment | `archi_rw` | satra |
|
||||
| eTerra.live | core + enrichment (ACL topograf) | enrichment via sync | `eterra_ro` + `eterra_sync` | satra/shop TBD |
|
||||
| SmartCity360 | core + urban (filter siruta) | urban (PUG/PUZ) | `sc_rw` | TBD |
|
||||
| Planhub ERP | core + urban | proiecte proprii (schemă `erp`) | `erp_rw` | TBD |
|
||||
|
||||
Fiecare app:
|
||||
- Own Authentik OIDC app
|
||||
- Own DB user + permisiuni minimale
|
||||
- Own MinIO bucket
|
||||
- Own Infisical project pentru secrete
|
||||
|
||||
---
|
||||
|
||||
## 9. Securitate
|
||||
|
||||
- **DB niciodată expus direct pe internet** → tot prin Martin / API cu auth
|
||||
- **Enrichment** (CF, proprietari) → NU în tile-uri publice, doar via API auth'd
|
||||
- **Rate limit Cloudflare** pe `tiles.*` (ex: 100 req/s per IP)
|
||||
- **Audit log** pe `gis_urban` în `gis_meta.audit` — cine modifică PUG primăriei
|
||||
- **Secrete** în Infisical, proiecte separate per app (zero reuse)
|
||||
- **Backup:** `pg_basebackup` + WAL archiving zilnic → MinIO satra + replică async CJ2 (PITR 14 zile)
|
||||
|
||||
---
|
||||
|
||||
## 10. Performanță
|
||||
|
||||
- Tabele simplificate pe zoom (deja există pentru UAT: z0/5/8/12) — extinde pe `gis_terenuri` dacă nevoie
|
||||
- **Materialized views** pentru query-uri grele cross-tenant (refresh nocturn)
|
||||
- **Indexe GIST** pe toate `geom`, **BRIN** pe `created_at` pentru audit
|
||||
- Martin pool: 16 conexiuni per instanță, 2 instanțe behind Traefik LB
|
||||
- **Read replica Postgres** când QPS > 500/s (streaming replication)
|
||||
- **PMTiles** pentru tot ce nu se schimbă zilnic (UAT, administrativ) — mult mai rapid decât Martin live
|
||||
|
||||
---
|
||||
|
||||
## 11. Scale target an 1
|
||||
|
||||
- ~50 arhitecți (Planhub ERP)
|
||||
- ~100 topografi (eTerra.live)
|
||||
- ~2 primării × 5 admin = 10 admin_primaria (SmartCity360)
|
||||
- Citizen public access (SmartCity360) — nedeterminat
|
||||
|
||||
**Concluzie:** Shop server (128 vCPU, 251GB RAM) = zero probleme la scale-ul ăsta. Rezerve pentru 10x creștere.
|
||||
|
||||
---
|
||||
|
||||
## 12. Migrare graduală — fără downtime ArchiTools
|
||||
|
||||
1. Setup `postgres-gis` container pe shop, config de bază
|
||||
2. Replicare initial snapshot `architools_db` → `postgres-gis` (pg_dump + restore)
|
||||
3. Creează schemas noi (`gis_urban`, `tenants`), mută tabele existente (`ALTER TABLE SET SCHEMA`)
|
||||
4. Creează roluri + RLS policies, testează cu user nou
|
||||
5. Extrage sync în serviciu standalone — paritate funcțională cu ArchiTools
|
||||
6. Lansează Martin dedicat pe `tiles.beletage.ro`
|
||||
7. ArchiTools trece pe endpoint nou (feature flag)
|
||||
8. Verify paritate + performanță → cutover final
|
||||
9. eTerra.live + SmartCity360 + Planhub ERP conectate la DB central
|
||||
|
||||
---
|
||||
|
||||
## 13. Date GIS existente (autoritative)
|
||||
|
||||
| Tabel | Conținut | Features aprox |
|
||||
|---|---|---|
|
||||
| `gis_uats` | Limite UAT Romania (+ views z0/5/8/12 simplified) | 3181 UAT-uri |
|
||||
| `gis_administrativ` | Intravilan + arii speciale ANCPI | — |
|
||||
| `gis_terenuri` | Parcele cadastrale eTerra | Cluj 774K (extrapolare ~25M total Romania) |
|
||||
| `gis_cladiri` | Clădiri cadastrale eTerra | Cluj — (similar magnitude) |
|
||||
| `gis_terenuri_status` / `gis_cladiri_status` | Parcele/clădiri + flag enrichment/legal | — |
|
||||
| `gis_features` | Raw eTerra (attributes JSON + enrichment JSON) | — |
|
||||
| `gis_sync_rules`, `gis_sync_runs` | Orchestrare sync | — |
|
||||
|
||||
Status actual: Cluj complet sincronizat (81 UAT, ~8.3h), restul României de populat (estimare 15-25h inițial, 1-2h delta).
|
||||
|
||||
---
|
||||
|
||||
## 14. Date GIS de adăugat pentru SmartCity360 + ERP
|
||||
|
||||
**Urbanism PUG:**
|
||||
- `gis_urban.utr` — Unități Teritoriale Referință (code, zone_type, regulation JSON cu POT/CUT/H_max, geom)
|
||||
- `gis_urban.zone_reglementate` — zone funcționale PUG
|
||||
- `gis_urban.zone_protectie` — monumente, arheologie, peisaj, sanitară, hidrologică
|
||||
|
||||
**PUZ/PUD:**
|
||||
- `gis_urban.puz` / `gis_urban.pud` — doc_ref, HCL_number, approval_date, status, geom perimetru
|
||||
|
||||
**Administrativ suplimentar:**
|
||||
- `gis_urban.strazi`
|
||||
- `gis_urban.numerotare`
|
||||
- `gis_urban.dotari_publice` (școli, spitale, parcuri)
|
||||
|
||||
**Fază 2 (opțional):**
|
||||
- Rețele utilități (apă, canal, gaz, electric, telecom)
|
||||
- Cadastru verde (arbori, scuaruri)
|
||||
|
||||
---
|
||||
|
||||
## 15. Întrebări deschise — TBD
|
||||
|
||||
### Tehnice
|
||||
|
||||
1. **Supabase vs Postgres raw pe shop** — schemă nouă în Supabase existent (câștig: RLS auto, PostgREST gratis, auth) sau container Postgres separat (izolare totală, simplu)? **Recomandare curentă:** separat.
|
||||
|
||||
2. **Domain SaaS** — `eterra.live` deja deținut? `smartcity360.ro` de cumpărat? Subdomain per primărie (`cluj-napoca.smartcity360.ro`) vs path-based (`smartcity360.ro/cluj-napoca`)?
|
||||
|
||||
3. **Portal cetățeni SmartCity** — acces complet anonim (zero login, doar rate limit) vs email/captcha pentru abuse prevention?
|
||||
|
||||
4. **TTL enrichment** — 90 zile auto-refresh implicit? Topograf poate forța refresh manual?
|
||||
|
||||
5. **Backup offsite** — replică Postgres async la CJ2 (10.10.40.x) via VPN? Sau doar MinIO snapshot zilnic?
|
||||
|
||||
6. **Migrare ArchiTools** — DB-ul actual rămâne paralel cu noul central, apoi cutover? Sau migrare directă cu downtime controlat?
|
||||
|
||||
7. **Martin shared vs per-app** — o singură instanță Martin publică pentru toate app-urile, sau instanțe separate per app (branding URL + izolare)?
|
||||
|
||||
### Business / organizaționale
|
||||
|
||||
8. **Billing / metering** — eTerra.live SaaS plătit: how measured? Per sync on-demand? Per enrichment request? Per user/lună?
|
||||
|
||||
9. **Data ownership** — dacă topograf X face enrichment pe parcela Y, datele sunt ale lui privat sau contribuie automat la pool-ul central public? (răspuns parțial: shared pool, dar GDPR?)
|
||||
|
||||
10. **PUG private pentru primării** — ce date specific sunt "private" vs "public"? (ex: observații interne, documentație anexă)
|
||||
|
||||
11. **SLA public tiles** — uptime target? Recovery time după outage?
|
||||
|
||||
---
|
||||
|
||||
## 16. Concluzii curente
|
||||
|
||||
- Arhitectura e fezabilă pe infrastructura existentă fără hardware nou
|
||||
- Shop server = candidat ideal DB central
|
||||
- Shared enrichment pool = insight major cu ROI imediat
|
||||
- Migrarea se poate face gradual fără downtime
|
||||
- Securitate + multi-tenancy rezolvate cu RLS + Authentik per-app
|
||||
|
||||
**Next steps când e aprobat:**
|
||||
1. Răspuns la întrebările deschise (secțiunea 15)
|
||||
2. Design detaliat schema `gis_urban` (DDL concret)
|
||||
3. Design API `gis-api.beletage.ro` (endpoints + auth flow)
|
||||
4. Plan migrare ArchiTools pas-cu-pas
|
||||
5. POC postgres-gis pe shop + test Martin + 1 app consumer
|
||||
@@ -0,0 +1,822 @@
|
||||
# Plan 001 v2 — DB GIS Central Multi-App
|
||||
|
||||
**Status:** FINAL APPROVED — 2026-04-20 (replaces v1 DRAFT)
|
||||
**Autor:** Marius Tarau + Claude Code
|
||||
**Scope:** Arhitectură unificată GIS pentru ArchiTools, eterra.live, pug.digital, planhub.ro + viitoare servicii
|
||||
**Execution trigger:** user sends "go" in new session → orchestrator begins Sprint 1
|
||||
|
||||
---
|
||||
|
||||
## 0. Cum recunoaște Claude "GO"
|
||||
|
||||
În sesiunea următoare, la cuvântul **"go"** singular:
|
||||
1. Re-read `/home/orchestrator/Code/ArchiTools/docs/plans/001-v2-central-gis-db-multi-app.md` complet
|
||||
2. Read `project_plan001_GO_sequence.md` din memorii
|
||||
3. Verifică secrete pre-loaded în Infisical (listă la §20)
|
||||
4. Pornește **Sprint 1 — Day 1** (§16)
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive summary
|
||||
|
||||
**Ce construim:** platformă GIS centralizată pe server shop (10.10.10.84) care servește 4 produse (ArchiTools intern, eterra.live SaaS topografi, pug.digital SaaS primării, planhub.ro SaaS arhitecți) + viitoare servicii.
|
||||
|
||||
**Durată:** 5 săptămâni dev + 1 weekend cutover + 1 săptămână cleanup = ~7 săptămâni total.
|
||||
|
||||
**Echipă:** 1 dev (Marius) + Claude Code pair.
|
||||
|
||||
**Rezultat:**
|
||||
- DB unică (PostgreSQL 18.3 + PostGIS 3.6.3) schemă split, RLS multi-tenant
|
||||
- Tile serving unificat (Martin + TiTiler + PMTiles + Cloudflare CDN)
|
||||
- Sync orchestrator cu multi-account ANCPI shuffle
|
||||
- Backup multi-tier (NAS + B2 + viitor Azure + viitor CJ2)
|
||||
- Dashboard unificat admin eterra.live
|
||||
- MDLPA export compliance pentru PUG/PUZ/PUD/PMUD
|
||||
|
||||
---
|
||||
|
||||
## 2. State curent (baseline)
|
||||
|
||||
### Infrastructură
|
||||
- **Shop** (10.10.10.84): Xeon Gold 6430, 128 vCPU, 251GB RAM, 3.3TB NVMe free
|
||||
- **Satra** (10.10.10.166): Ubuntu, 194GB disk (61% plin), architools_postgres curent
|
||||
- **Proxy** (10.10.10.199): Traefik v3.6.8
|
||||
- **NAS NewAmun** (10.10.10.10): NETGEAR ReadyNAS, 40.88TB free, btrfs
|
||||
|
||||
### App-uri curente
|
||||
- ArchiTools (tools.beletage.ro) — Next.js, Prisma, conectat architools_postgres satra
|
||||
- eterra.live — Next.js v5 NextAuth, per-user AES-256-GCM, același DB
|
||||
- Martin tile server v1.4.0 pe satra port 3010
|
||||
|
||||
### POC postgres-gis (2026-04-19)
|
||||
- Deployed pe shop la `/home/dnz/postgres-gis/`
|
||||
- PG 18.3 + PostGIS 3.6.3 + pgBouncer v1.24.1-p1
|
||||
- `architools_db` restored from satra: 16GB, 24 tabele, 100% paritate row counts
|
||||
- Performance test: **9.4× faster** ST_Intersects vs satra
|
||||
- Status: **RUNNING sandbox**, zero impact production
|
||||
|
||||
### Securitate rotate 2026-04-19
|
||||
- AUTHENTIK_CLIENT_SECRET ✅
|
||||
- NEXTAUTH_SECRET ✅
|
||||
- DB_PASS Postgres ✅
|
||||
- MINIO_ACCESS/SECRET_KEY ✅ (user nou gen-hex)
|
||||
- NOTIFICATION_CRON_SECRET ✅
|
||||
- ETERRA_USERNAME/PASSWORD ✅ (mutate Infisical)
|
||||
|
||||
### Backup infrastructure testată 2026-04-20
|
||||
- NAS rsync via SSH key RSYNC-ONLY flag, `gisbkp/` → `/HDD/gisbkp` (symlink), 50 MB/s LAN
|
||||
- Backblaze B2 bucket `beletage-gis-backups` (EU, Object Lock Compliance, SSE-B2), key scoped
|
||||
|
||||
---
|
||||
|
||||
## 3. Decizii arhitecturale (finalizate)
|
||||
|
||||
### Golden rule (aplicat întreg planul)
|
||||
Preferă investment upfront pentru câștiguri pe termen lung în: **safety, speed, resources, future-proof**.
|
||||
|
||||
### Tehnologii
|
||||
- **PostgreSQL 18.3** (async I/O, skip scans, temporal constraints, UUID v7)
|
||||
- **PostGIS 3.6.3** (ST_RemoveIrrelevantPointsForView, SFCGAL modern)
|
||||
- **pgBouncer 1.24.1-p1** (transaction mode, 1000 max clients, 50 default pool)
|
||||
- **Martin 1.5.0** tile server (vector MVT)
|
||||
- **TiTiler** (raster COG)
|
||||
- **Next.js 16 + Hono + Prisma + Zod** pentru gis-api
|
||||
- **pg-boss** queue (NU Redis)
|
||||
- **pgBackRest** backup engine
|
||||
- **Docker Compose** (NU Kubernetes)
|
||||
- **MinIO** object storage
|
||||
- **Authentik** OIDC (existent)
|
||||
|
||||
### Versioning policy
|
||||
- Latest stable ≥3 luni în producție + ≥1 patch
|
||||
- Pin exact în compose (NU `:latest`)
|
||||
- Manual upgrades după test pe snapshot
|
||||
- Periodic review trimestrial
|
||||
|
||||
### Domains (brand strategy)
|
||||
| Domain | Rol |
|
||||
|---|---|
|
||||
| `gis.ac` | Infra neutru — tiles, api, pmtiles, s3 |
|
||||
| `eterra.live` | SaaS topografi |
|
||||
| `pug.digital` | SaaS primării (fost SmartCity360) |
|
||||
| `planhub.ro` | SaaS multi-tenant arhitecți |
|
||||
| `app.beletage.ro` | Beletage tenant CNAME rebrand |
|
||||
| `vreau.digital` | Rezervă — portal cetățeni |
|
||||
| `buildini.ai` | Rezervă — AI design |
|
||||
| `puz.digital` | SEO redirect → pug.digital |
|
||||
| `beletage.ro` | Intern + admin |
|
||||
|
||||
---
|
||||
|
||||
## 4. Target stack shop (13 containere)
|
||||
|
||||
```
|
||||
shop (10.10.10.84) — /home/dnz/postgres-gis/ + /opt/gis-stack/:
|
||||
|
||||
├── postgres-gis PG 18.3 + PostGIS 3.6.3 [✅ POC]
|
||||
├── postgres-gis-pgbouncer transaction mode, 1000 clients [✅ POC]
|
||||
├── martin-public vector tiles publice, zero auth [Sprint 1]
|
||||
├── martin-private vector tiles auth via JWT ForwardAuth [Sprint 1]
|
||||
├── titiler raster COG + DEM proxy [Sprint 1]
|
||||
├── gis-api Next.js 16 + Hono + Prisma + Zod [Sprint 2]
|
||||
├── gis-sync-orchestrator pg-boss workers + session pool [Sprint 3]
|
||||
├── nginx-pmtiles static serve MinIO pmtiles [Sprint 1]
|
||||
├── minio-gis buckets: pmtiles, cog, cf-pdfs, dxf, backups [Sprint 1]
|
||||
├── pgadmin intern doar (pgadmin.beletage.ro) [Sprint 1]
|
||||
├── prometheus scrape metrics [Sprint 4]
|
||||
├── grafana dashboards [Sprint 4]
|
||||
└── alertmanager alerts → n8n webhook [Sprint 4]
|
||||
```
|
||||
|
||||
**Resource utilization estimat:** ~8% CPU shop, ~15% RAM shop. Loaded la plural, idle majority.
|
||||
|
||||
---
|
||||
|
||||
## 5. Schema DB split
|
||||
|
||||
```
|
||||
postgres-gis / gis (database):
|
||||
|
||||
├── schema: gis_core Cadastru ANCPI (public readable)
|
||||
│ ├── terenuri Parcele (25M estimate)
|
||||
│ ├── cladiri Clădiri cadastrale
|
||||
│ ├── uats 3,186 UAT România
|
||||
│ ├── administrativ Intravilan + arii protejate
|
||||
│ └── uats_z0/5/8/12 Simplified views per zoom
|
||||
│
|
||||
├── schema: gis_urban PUG/PUZ/PUD (multi-tenant RLS per siruta)
|
||||
│ ├── plan_spatial MDLPA-compliant root (SIRUTA, Judet, HCL, stadiu)
|
||||
│ ├── zf_existenta, zf_propusa Zone funcționale + HILUCS codes
|
||||
│ ├── zona_reglementare Zone protecție (monumente, sanitară, etc.)
|
||||
│ ├── utr Unități Teritoriale Referință + regulament JSONB
|
||||
│ ├── puz, pud Sub-plans
|
||||
│ ├── cai_comunicatie Drumuri
|
||||
│ ├── retele_edilitare Apă, canal, gaz, electric, telecom
|
||||
│ ├── echipare_edilitara Hidranți, stații (puncte)
|
||||
│ ├── regulament_local Text regulament
|
||||
│ ├── avize_acorduri PDF metadata
|
||||
│ ├── *_drafts Versiuni work-in-progress arhitecți
|
||||
│ └── puz_in_avizare Public preview în timpul avizării
|
||||
│
|
||||
├── schema: gis_enrichment GDPR sensitive (CF, proprietari, adrese)
|
||||
│ ├── cf_extracts Criptat, RLS strict
|
||||
│ ├── proprietari Nume + CNP
|
||||
│ ├── adrese Normalizare nomenclator
|
||||
│ └── shared_pool GDPR-safe shareable (nr_cad, suprafață, UTR)
|
||||
│
|
||||
├── schema: gis_meta Orchestrare + audit
|
||||
│ ├── sync_runs Log sync eTerra
|
||||
│ ├── sync_rules Planning sync
|
||||
│ ├── audit ENCRYPTED cu pgcrypto (audit metadata)
|
||||
│ ├── eterra_sessions Session pool persistent
|
||||
│ ├── eterra_accounts Multi-account ANCPI shuffle
|
||||
│ ├── eterra_account_usage_log Per-action audit
|
||||
│ └── raster_sources COG registry (upload metadata)
|
||||
│
|
||||
├── schema: tenants Control plane multi-tenancy
|
||||
│ ├── tenants id, name, tenant_type, is_beletage_group, siruta_scope[]
|
||||
│ ├── members user → tenant mapping
|
||||
│ └── enrichment_scopes per-user flag (none/basic/full)
|
||||
│
|
||||
├── schema: eterra App-specific eterra.live
|
||||
├── schema: pug App-specific pug.digital
|
||||
├── schema: archi App-specific ArchiTools
|
||||
├── schema: planhub App-specific planhub.ro (multi-tenant)
|
||||
├── schema: queue pg-boss queue
|
||||
└── schema: public Extensii (postgis, pg_stat_statements, pgcrypto, pg_trgm)
|
||||
```
|
||||
|
||||
### Super-tenant Beletage group
|
||||
Tabela `tenants.is_beletage_group BOOLEAN`. Tenants marked (Beletage SRL, Studii de Teren, Urban Switch, Cubitron, + viitoare firme asociate) au acces NERESTRICȚIONAT la toate datele. Controlat prin admin Beletage.
|
||||
|
||||
---
|
||||
|
||||
## 6. Roluri Postgres + RLS
|
||||
|
||||
### DB users
|
||||
- `gis_app_rw` — gis-api main
|
||||
- `gis_sync_rw` — orchestrator worker (UNICUL scriitor în gis_core)
|
||||
- `gis_public_ro` — Martin public (read-only)
|
||||
- `gis_private_ro` — Martin-private (read-only cu RLS)
|
||||
- `gis_titiler_ro` — TiTiler raster
|
||||
- `gis_admin_dba` — pgAdmin + Marius (NO BYPASSRLS — Golden)
|
||||
|
||||
### Session variables setate la fiecare tranzacție
|
||||
```sql
|
||||
SET LOCAL app.user_id = ...;
|
||||
SET LOCAL app.tenant_id = ...;
|
||||
SET LOCAL app.is_beletage_group = 'true'|'false';
|
||||
SET LOCAL app.enrichment_scope = 'none'|'basic'|'full';
|
||||
SET LOCAL app.allowed_sirutas = 'csv,list';
|
||||
SET LOCAL app.roles = 'csv,list';
|
||||
```
|
||||
|
||||
### RLS policy patterns
|
||||
- `gis_core`: zero RLS, public read
|
||||
- `gis_urban.*_drafts`: tenant-isolated (own tenant_id OR Beletage group)
|
||||
- `gis_urban.*_aprobate`: public read, write admin_primaria sau Beletage
|
||||
- `gis_enrichment.cf_extracts`:
|
||||
```sql
|
||||
USING (
|
||||
current_setting('app.is_beletage_group', true)::boolean = true
|
||||
OR fetched_by_user_id = current_setting('app.user_id', true)::uuid
|
||||
OR fetched_by_tenant_id = current_setting('app.tenant_id', true)::uuid
|
||||
OR (is_shareable = true AND current_setting('app.enrichment_scope', true) IN ('basic', 'full'))
|
||||
)
|
||||
```
|
||||
- `FORCE ROW LEVEL SECURITY` pe toate tabelele RLS (fallback deny)
|
||||
|
||||
---
|
||||
|
||||
## 7. Auth flow (Authentik OIDC)
|
||||
|
||||
### JWT custom claims
|
||||
```json
|
||||
{
|
||||
"sub": "user-uuid",
|
||||
"email": "...",
|
||||
"tenant_id": "tenant-uuid",
|
||||
"is_beletage_group": true,
|
||||
"roles": ["topograf", "firm_admin"],
|
||||
"enrichment_scope": "full",
|
||||
"allowed_sirutas": ["54975", "155243"]
|
||||
}
|
||||
```
|
||||
|
||||
### User attributes sync
|
||||
- **Webhook HMAC-signed** din planhub/eterra.live la user create/update
|
||||
- Idempotency key dedupe 24h
|
||||
- Exponential backoff retry (6 tentative: 1,2,4,8,16,32s)
|
||||
- Dead Letter Queue în pg-boss `auth-sync-dlq`
|
||||
|
||||
### Super-tenant check
|
||||
**Dynamic DB query** la fiecare JWT emit (Authentik custom expression), cache 60s.
|
||||
|
||||
### JWT expiry
|
||||
- Access: 1h
|
||||
- Refresh: 30d
|
||||
|
||||
---
|
||||
|
||||
## 8. Sync orchestrator + multi-account ANCPI
|
||||
|
||||
### Componente
|
||||
- **Queue:** pg-boss (schema `queue`), max 8 workers, retention 30d
|
||||
- **Workers:**
|
||||
- `eterra-sync-worker` (delta/full UAT sync)
|
||||
- `enrichment-worker` (CF fetch + GDPR pool split)
|
||||
- `pug-ingest-worker` (GPKG upload → gis_urban drafts)
|
||||
- `pmtiles-rebuild-worker` (tippecanoe → MinIO)
|
||||
- `cache-invalidate-worker` (Cloudflare purge API)
|
||||
|
||||
### Multi-account ANCPI shuffle
|
||||
```sql
|
||||
gis_meta.eterra_accounts (
|
||||
id, username, password_encrypted (AES-256-GCM, key GIS_SYNC_AES_KEY),
|
||||
account_type, quota_per_hour DEFAULT 500,
|
||||
usage_current_hour, last_reset_at,
|
||||
status, notes
|
||||
)
|
||||
```
|
||||
Round-robin pe conturi active cu quota disponibilă. Failover auto dacă blocked. Distribution load ANCPI = zero detection singular-account patterns.
|
||||
|
||||
### Session pool persistent (TTL 20 min)
|
||||
Înlocuiește in-memory din eterra-live + hardcoded din ArchiTools.
|
||||
|
||||
### LISTEN/NOTIFY events cross-app
|
||||
- `sync:uat:done` → apps refresh UI
|
||||
- `pmtiles:rebuild:done` → cache invalidate
|
||||
- `enrichment:new` → user UI notify
|
||||
|
||||
### Scheduler (pg-boss cron)
|
||||
- Delta sync: Lu-Vi 2:00
|
||||
- Deep sync: Weekend 23:00
|
||||
- PMTiles rebuild: Lu-Vi 4:00
|
||||
- Cleanup sessions: 6h
|
||||
|
||||
### ArchiTools/eterra-live post-migration
|
||||
- **ArchiTools:** DELETE eterra-client.ts + setInterval. Read-only consumer + trigger sync via API.
|
||||
- **eterra-live:** Păstrează UI eterra login (per-user), trigger sync prin orchestrator API.
|
||||
- **Orchestrator = UNICUL scriitor gis_core.**
|
||||
|
||||
---
|
||||
|
||||
## 9. Tile serving (4 subdomenii)
|
||||
|
||||
### `tiles.gis.ac` (PUBLIC)
|
||||
Martin 1 instanță, zero auth, CF rate limit 100 req/s/IP.
|
||||
|
||||
Layer groups:
|
||||
- `/cadastru`, `/uat`, `/pug-public`, `/zone-protectie`, `/drumuri`, `/retele-publice`
|
||||
- `/puz-aprobate`, `/pud-aprobate`, `/analize-publice`, `/puz-in-avizare`
|
||||
|
||||
### `tiles-private.gis.ac` (AUTH)
|
||||
Martin-private container + Traefik ForwardAuth JWT validation. Zero cache.
|
||||
|
||||
Layer groups:
|
||||
- `/pug-drafts/{tenant}`, `/puz-drafts/{tenant}`, `/pud-drafts/{tenant}`
|
||||
- `/observatii-primarie/{siruta}`, `/analize-interne/{tenant}`
|
||||
|
||||
### `raster.gis.ac` + `dem.gis.ac`
|
||||
TiTiler (NU Martin — Martin only vector).
|
||||
- `raster.gis.ac` serves COG din MinIO bucket
|
||||
- `dem.gis.ac` = **proxy cu cache CF** la ANCPI MNT (geoportal.ancpi.ro/maps/rest/services/ANCPI/MNT)
|
||||
- Hillshade + contour generate local din DEM
|
||||
|
||||
### `pmtiles.gis.ac`
|
||||
nginx static serve din MinIO bucket `pmtiles`. CF cache 7 zile.
|
||||
|
||||
### CF TTL config
|
||||
| Path | TTL |
|
||||
|---|---|
|
||||
| tiles.gis.ac/uat | 30 zile |
|
||||
| tiles.gis.ac/cadastru | 6h + auto-purge la sync |
|
||||
| tiles.gis.ac/pug-public | 24h |
|
||||
| tiles.gis.ac/puz-in-avizare | 1h |
|
||||
| tiles.gis.ac/puz-aprobate, pud | 24h |
|
||||
| pmtiles.gis.ac | 7 zile |
|
||||
| tiles-private.gis.ac | zero |
|
||||
| raster.gis.ac, dem.gis.ac | 30 zile |
|
||||
|
||||
Bot Fight Mode: Medium default.
|
||||
|
||||
### PUZ lifecycle visibility
|
||||
| Stadiu | Vizibil | Cache |
|
||||
|---|---|---|
|
||||
| Draft intern | tiles-private (tenant only) | zero |
|
||||
| Avizare publică | tiles.gis.ac/puz-in-avizare | 1h |
|
||||
| Aprobat | tiles.gis.ac/puz-aprobate (overlay pug-public) | 24h |
|
||||
|
||||
### Raster Library module (admin eterra.live)
|
||||
- Upload TIFF/GeoTIFF → background gdal_translate → COG + overviews
|
||||
- Preview thumbnail 512px
|
||||
- Public/private toggle
|
||||
- Delete/replace
|
||||
- DB: `gis_meta.raster_sources`
|
||||
|
||||
---
|
||||
|
||||
## 10. API layer (api.gis.ac)
|
||||
|
||||
### Stack
|
||||
Next.js 16 + Hono router intern + Prisma + Zod + Authentik JWT middleware.
|
||||
|
||||
### Endpoint exemple
|
||||
```
|
||||
POST /enrichment/parcela auth: tier≥basic
|
||||
GET /parcela/{id} auth: optional
|
||||
POST /pug/zone-functionala auth: admin_primaria
|
||||
POST /sync/uat auth: admin Beletage
|
||||
POST /raster/upload auth: architect cu scope
|
||||
GET /search?q=... auth: optional
|
||||
```
|
||||
|
||||
### Rate limiting (Redis per JWT sub)
|
||||
| Tier | Req/oră | Enrichments/zi |
|
||||
|---|---|---|
|
||||
| Free | 100 | 10 |
|
||||
| Basic | 1000 | 100 |
|
||||
| Pro | 5000 | unlimited |
|
||||
| Admin | nelimitat | nelimitat |
|
||||
|
||||
Managed prin admin eterra.live.
|
||||
|
||||
---
|
||||
|
||||
## 11. MDLPA compliance (Ordin 904/2023)
|
||||
|
||||
**Abordare Golden:** schema internă flexibilă + export layer separat conform MDLPA.
|
||||
|
||||
### Internal schema gis_urban.plan
|
||||
Include ALL MDLPA required fields + extras (history, drafts, workflows, comments).
|
||||
|
||||
### PG18 temporal constraint
|
||||
```sql
|
||||
ALTER TABLE gis_urban.plan_spatial ADD CONSTRAINT no_overlap_plan
|
||||
EXCLUDE USING gist (siruta WITH =, doc_type WITH =,
|
||||
daterange(data_aprob, data_exp, '[)') WITH &&);
|
||||
```
|
||||
Garantează zero PUG-uri suprapuse pe aceeași UAT.
|
||||
|
||||
### Export worker
|
||||
- `pug-export-worker` (pg-boss job)
|
||||
- Input: `{plan_id}`
|
||||
- Output: ZIP cu 5 subdirectoare + GPKG MDLPA-compliant + PDF-uri
|
||||
- Tooling: ogr2ogr (GDAL) pentru GPKG, custom orchestrator Node
|
||||
- Drop Z/M dimensions la export
|
||||
- Default HILUCS_N1 dacă missing intern
|
||||
- ZIP naming: `^[A-Z][A-Z]_.*?_\d{1,10}_(PUG|PUZ|PATJ|PMUD)_[0-9]{8}`
|
||||
|
||||
### Validator MDLPA
|
||||
**Faza 1:** manual pre-submit by user (download Validator_2.0.5, run local).
|
||||
**Faza 2:** Marius primește repo validator de la friends MDLPA → containerizăm.
|
||||
**Faza 3:** custom pre-validation 80% reguli (blocker-i obvious) înaintea ZIP download.
|
||||
|
||||
### Doc types în scope
|
||||
PUG (prio 1), PUZ (prio 1), PUD (prio 2), PMUD (da), PATJ (opțional, max 1 județ).
|
||||
|
||||
---
|
||||
|
||||
## 12. Backup strategy (multi-tier)
|
||||
|
||||
### Tier 1 — NAS local (LAN fast)
|
||||
- Path: `gis-backup@10.10.10.10:gisbkp/` (symlink → `/HDD/gisbkp`)
|
||||
- btrfs + COW + compression
|
||||
- Retention: 30 daily + 12 monthly + 5 yearly snapshots
|
||||
- Protocol: rsync over SSH, RSYNC-ONLY key flag
|
||||
|
||||
### Tier 2 — Backblaze B2 (offsite hot)
|
||||
- Bucket: `beletage-gis-backups` (region EU)
|
||||
- Object Lock Compliance mode 90d + SSE-B2
|
||||
- Weekly full + monthly archive encrypted AES-256 client-side
|
||||
|
||||
### Tier 3 — Azure (deferred, credit €2400/year)
|
||||
- Azure PostgreSQL Flexible Server (DR cloud fallback, ~50-80€/lună)
|
||||
- Azure CDN + Blob Hot (PMTiles secondary, ~20€/lună)
|
||||
- When: scale 10k+ users sau geo-redundanță
|
||||
|
||||
### Tier 4 — CJ2 standby (deferred ~oct 2026)
|
||||
- Streaming replication async, RPO~0, RTO<5min
|
||||
- Aștept workstation AI nou + disk dedicat
|
||||
|
||||
### pgBackRest setup
|
||||
- 2 repos: NAS + B2
|
||||
- PITR window: **30 zile**
|
||||
- Continuous WAL archive
|
||||
- Compress zstd + encrypt AES-256
|
||||
- Parallel restore workers
|
||||
|
||||
### DR drill
|
||||
- **Trimestrial** manual full recovery simulation
|
||||
- **Săptămânal** smoke test (read-only integrity check)
|
||||
- **Lunar** test restore automation (ephemeral container, 30 min, 20GB temp)
|
||||
|
||||
### Backup Dashboard
|
||||
Modul în admin.eterra.live, tab "Backup & Recovery":
|
||||
- Cards pentru fiecare tier (NAS, B2, Azure placeholder, CJ2 placeholder)
|
||||
- Plugin architecture `BackupTarget` interface — extensibil
|
||||
- Slot-uri viitoare: email archival, Gitea repos, workstations, configs
|
||||
- Alerting Golden low-noise (email doar la red)
|
||||
|
||||
---
|
||||
|
||||
## 13. Monitoring (Dashboard admin eterra.live)
|
||||
|
||||
### 6 cards simple (RO tooltip clar)
|
||||
1. Hărți (req/s)
|
||||
2. Baza de date (query latency)
|
||||
3. Spațiu disc (%)
|
||||
4. Topografi activi
|
||||
5. Sincronizare (ultim)
|
||||
6. Backup (ultim)
|
||||
|
||||
### Culori: verde/galben/roșu
|
||||
### Alert doar la roșu (zero spam)
|
||||
### Grafana full = collapsed "Detalii avansate"
|
||||
|
||||
### Prometheus metrics scraped
|
||||
- Martin (/metrics), Postgres (pg_stat_statements + exporter), pgBouncer, MinIO, gis-api, orchestrator
|
||||
|
||||
---
|
||||
|
||||
## 14. Securitate — posture final
|
||||
|
||||
### Authentication
|
||||
- Authentik OIDC pentru toate apps
|
||||
- JWT 1h access + 30d refresh
|
||||
- JWKS cache 1h
|
||||
|
||||
### Authorization
|
||||
- RLS multi-tenant (siruta + tenant_id + is_beletage_group)
|
||||
- Session variables via SET LOCAL (zero leak)
|
||||
- Zero Postgres role cu BYPASSRLS permanent
|
||||
|
||||
### Secrets management
|
||||
- Infisical single source of truth
|
||||
- Rotate schedule: annual pentru keys critice, automatic pentru JWT
|
||||
- Encryption keys: Infisical + paper print sealed Beletage safe (YubiKey viitor)
|
||||
|
||||
### Network
|
||||
- PG 5433 bind 127.0.0.1 (intern shop)
|
||||
- pgBouncer 6432 bind 127.0.0.1
|
||||
- Martin/TiTiler/gis-api prin Traefik only
|
||||
- Sophos DNAT doar 80/443 către proxy
|
||||
|
||||
### Audit
|
||||
- `gis_meta.audit` ENCRYPTED cu pgcrypto (key AUDIT_ENCRYPTION_KEY)
|
||||
- Retention: 1 an hot PG + 5 ani archive MinIO
|
||||
- Decryption doar la query admin explicit
|
||||
|
||||
### Data at rest
|
||||
- Backups encrypted (pgBackRest + client-side AES pentru B2)
|
||||
- ENCRYPTION_SECRET rotate pending (re-encrypt 15 vault entries)
|
||||
|
||||
---
|
||||
|
||||
## 15. Ce DISPARE post-cutover
|
||||
|
||||
- `setInterval` cron-in-cod din ArchiTools
|
||||
- `eterra-client.ts` din ArchiTools (mut în orchestrator)
|
||||
- In-memory session pool eterra-live (persistent DB)
|
||||
- PG 5432 exposed 0.0.0.0 satra (shop bind LAN)
|
||||
- PMTiles webhook broken (înlocuit de pmtiles-rebuild-worker)
|
||||
- systemd pmtiles-webhook parole în args (container nou)
|
||||
- Hardcoded ETERRA_USERNAME/PASSWORD ArchiTools (shared pool)
|
||||
- AES IV=16 eterra-live (upgrade 12)
|
||||
- Rate limit in-memory eterra-live (Redis)
|
||||
|
||||
---
|
||||
|
||||
## 16. Migration timeline (6 săptămâni)
|
||||
|
||||
### Sprint 1 — Foundation stack (săpt 1)
|
||||
**Day 1** (GO trigger):
|
||||
- Deploy `martin-public` container pe shop
|
||||
- Deploy `martin-private` container + Traefik ForwardAuth middleware
|
||||
- Deploy `titiler` container
|
||||
- Deploy `nginx-pmtiles` container
|
||||
- Deploy `minio-gis` container + bucket setup
|
||||
|
||||
**Day 2:**
|
||||
- Deploy `pgadmin` (intern pgadmin.beletage.ro)
|
||||
- Traefik routes pentru toate subdomeniile GIS
|
||||
|
||||
**Day 3-4:**
|
||||
- Cloudflare zones: add `gis.ac`, `eterra.live`, `pug.digital`, `planhub.ro`
|
||||
- DNS records + Page Rules TTL + Rate Limits + Bot Fight Medium
|
||||
- Test tile serving basic
|
||||
|
||||
**Day 5:**
|
||||
- Schema split pe shop:
|
||||
- `CREATE SCHEMA gis_core, gis_urban, gis_enrichment, gis_meta, tenants, eterra, pug, archi, planhub, queue`
|
||||
- `ALTER TABLE public.X SET SCHEMA gis_core` pentru tabele existente
|
||||
- Create views + roles
|
||||
|
||||
### Sprint 2 — Auth + data layer (săpt 2)
|
||||
**Day 1-2:**
|
||||
- Authentik custom property mappings (tenant_id, is_beletage_group, enrichment_scope, allowed_sirutas)
|
||||
- Webhook signed HMAC din eterra.live → Authentik sync
|
||||
- PG roluri aplicație + grants
|
||||
|
||||
**Day 3-4:**
|
||||
- RLS policies per tabel (gis_urban, gis_enrichment)
|
||||
- FORCE ROW LEVEL SECURITY pe toate
|
||||
- gis-api boilerplate (Next.js 16 + Hono + Prisma + Zod)
|
||||
- Auth middleware JWT verify + SET LOCAL
|
||||
|
||||
**Day 5:**
|
||||
- Cypress test suite RLS (6 scenarii obligatorii)
|
||||
|
||||
### Sprint 3 — Sync orchestrator (săpt 3)
|
||||
**Day 1-2:**
|
||||
- `gis-sync-orchestrator` container
|
||||
- pg-boss setup schema queue
|
||||
- eTerra client extract în shared lib
|
||||
|
||||
**Day 3:**
|
||||
- Session pool persistent `gis_meta.eterra_sessions` + multi-account `gis_meta.eterra_accounts`
|
||||
- Workers: eterra-sync, enrichment, pug-ingest
|
||||
|
||||
**Day 4:**
|
||||
- Workers: pmtiles-rebuild, cache-invalidate (CF API)
|
||||
- LISTEN/NOTIFY events implementation
|
||||
|
||||
**Day 5:**
|
||||
- Admin UI module eterra.live — sync jobs control panel
|
||||
|
||||
### Sprint 4 — Backup + Dashboard (săpt 4)
|
||||
**Day 1-2:**
|
||||
- pgBackRest deploy shop
|
||||
- 2 repos: NAS (rsync push via cron) + B2 (direct S3)
|
||||
- Scheduled jobs: daily incremental, weekly full, monthly archive
|
||||
- WAL archive continuous
|
||||
|
||||
**Day 3:**
|
||||
- Backup Dashboard UI (admin eterra.live tab "Backup & Recovery")
|
||||
- Plugin architecture `BackupTarget` interface
|
||||
- Cards: NAS, B2, Azure placeholder, CJ2 placeholder
|
||||
|
||||
**Day 4:**
|
||||
- DR runbook scripts
|
||||
- Monthly restore test automation (ephemeral container cron)
|
||||
- Prometheus + Grafana + Alertmanager deploy
|
||||
|
||||
**Day 5:**
|
||||
- Grafana dashboards (GIS Overview, Martin Perf, PG Health, API Rate Limits, Raster Library)
|
||||
- 6 simple cards admin eterra.live
|
||||
- Alertmanager → n8n webhook (email + Telegram on red)
|
||||
|
||||
### Sprint 5 — Test & parity (săpt 5)
|
||||
**Day 1-2:**
|
||||
- Shadow sync: orchestrator rulează în paralel cu satra setInterval (both read from ANCPI)
|
||||
- Compare row counts ambele, checksum-uri, latency
|
||||
|
||||
**Day 3:**
|
||||
- Load test: 1000 concurrent API requests, 500 QPS tile server
|
||||
- Performance comparisons satra vs shop end-to-end
|
||||
|
||||
**Day 4:**
|
||||
- Security audit final: TLS scan, dependency audit, pentest basic
|
||||
- CORS, CSP, rate limits verify
|
||||
|
||||
**Day 5:**
|
||||
- GO/NO-GO decision meeting
|
||||
- Final checklist review
|
||||
|
||||
### Weekend cutover (if GO)
|
||||
**Friday 22:00:**
|
||||
- Notify users (eterra.live + ArchiTools) maintenance 4h
|
||||
- Traefik maintenance page
|
||||
- Stop ArchiTools + eterra-live containers
|
||||
- pg_dump satra final snapshot → shop (delta apply)
|
||||
- Diff check paritate
|
||||
- Switch DATABASE_URL containere → shop
|
||||
- Deploy ArchiTools nou (cu eterra-client DELETED)
|
||||
- Cloudflare DNS update: tiles.gis.ac → shop Martin
|
||||
- Start orchestrator cron jobs
|
||||
- Smoke test full flow
|
||||
|
||||
**Saturday 06:00:**
|
||||
- Monitor dashboard
|
||||
- On-call standby
|
||||
|
||||
**Saturday 20:00:**
|
||||
- All green → GO production
|
||||
- satra architools_postgres READ-ONLY (fallback 30 zile safety)
|
||||
|
||||
**Sunday:**
|
||||
- Observation + bug bash
|
||||
|
||||
### Cleanup (săpt 6)
|
||||
- git filter-repo ArchiTools (rescrie istoria, notify re-clone)
|
||||
- Structural fix compose (env_file + stack.env in .gitignore)
|
||||
- Remove satra containers (după 30 zile)
|
||||
- ENCRYPTION_SECRET rotate + re-encrypt 15 vault entries
|
||||
- Documentation final în busc-infra
|
||||
|
||||
---
|
||||
|
||||
## 17. Runbooks (DR scenarios)
|
||||
|
||||
### R1 — Corruption parcela X la 14:30
|
||||
```bash
|
||||
pgbackrest --stanza=gis --type=time --target="2026-04-20 14:29:00" restore
|
||||
# Verify integrity + promote
|
||||
```
|
||||
|
||||
### R2 — Crash shop total → failover CJ2 (când activ)
|
||||
1. Promote CJ2 standby to primary
|
||||
2. Update DNS: tiles.gis.ac, api.gis.ac → CJ2 IP
|
||||
3. Restart apps pointing CJ2
|
||||
4. RTO target: <5 min
|
||||
|
||||
### R3 — Ransomware
|
||||
- B2 Object Lock Compliance = fișiere immutable
|
||||
- Restore latest clean snapshot pre-ransomware
|
||||
- Wipe compromised + rebuild from B2
|
||||
|
||||
### R4 — Accidental DROP TABLE
|
||||
```bash
|
||||
pgbackrest restore --type=time --target="<moment înainte de DROP>"
|
||||
# Replay WAL până exact înainte de comanda distructivă
|
||||
```
|
||||
|
||||
### R5 — Disaster regional (shop + CJ2 down)
|
||||
- Restore din B2 arhive lunar
|
||||
- Bootstrap new cluster pe alt host
|
||||
- RTO: 4-8 ore
|
||||
|
||||
---
|
||||
|
||||
## 18. Next features post-production
|
||||
|
||||
### Imediat după stable (month 1-2)
|
||||
- MDLPA validator containerizat (după repo de la friends MDLPA)
|
||||
- Geopackage export pipeline complet
|
||||
- Raster Library module + upload UI
|
||||
|
||||
### Medium term (month 3-6)
|
||||
- PMUD workflow (rețele transport, piste, parcare)
|
||||
- SmartCity360 / pug.digital launch (primul tenant primărie)
|
||||
- planhub.ro SaaS pivot multi-tenant (primii arhitecți externi)
|
||||
- CJ2 standby activation (după workstation AI nou)
|
||||
|
||||
### Long term (month 6-12)
|
||||
- PATJ suport (1 județ pilot)
|
||||
- Azure activation (Azure PostgreSQL DR + Blob Hot CDN)
|
||||
- Cross-region geo-redundanță dacă scale justifies
|
||||
|
||||
---
|
||||
|
||||
## 19. Risk register
|
||||
|
||||
| Risc | Probabilitate | Impact | Mitigare |
|
||||
|---|---|---|---|
|
||||
| Cutover fail | Low | High | Rollback DNS → satra, 30d safety window |
|
||||
| Data loss cutover window | Very Low | High | Shadow sync săpt 5, delta final |
|
||||
| Orchestrator bugs prod | Medium | Medium | Săpt 5 shadow prinde majoritatea |
|
||||
| ANCPI block multi-account | Low | High | Shuffle + quota + alerting |
|
||||
| B2 cloud outage | Very Low | Medium | NAS + CJ2 (când activ) alternative |
|
||||
| Marius unavailable | Medium | High | Claude Code has full plan + memory |
|
||||
|
||||
---
|
||||
|
||||
## 20. Secrete pre-loaded în Infisical (verificate 2026-04-20)
|
||||
|
||||
### Active (ArchiTools production)
|
||||
- AUTHENTIK_CLIENT_ID, AUTHENTIK_CLIENT_SECRET
|
||||
- NEXTAUTH_SECRET
|
||||
- DB_PASS (satra Postgres user, will rotate at cutover)
|
||||
- ETERRA_USERNAME, ETERRA_PASSWORD
|
||||
- NOTIFICATION_CRON_SECRET
|
||||
- BREVO_BELETAGE_API_KEY
|
||||
|
||||
### Shop stack (pre-loaded pentru sprint 1)
|
||||
- POSTGRES_GIS_SUPERUSER_PASSWORD
|
||||
- POSTGRES_GIS_ARCHITOOLS_PASS
|
||||
- POSTGRES_GIS_ETERRA_PASS
|
||||
- POSTGRES_GIS_MARTIN_PASS
|
||||
- POSTGRES_GIS_SYNC_PASS
|
||||
- POSTGRES_GIS_PUG_PASS
|
||||
- POSTGRES_GIS_PLANHUB_PASS
|
||||
- MINIO_GIS_ROOT_USER
|
||||
- MINIO_GIS_ROOT_PASSWORD
|
||||
- PGADMIN_DEFAULT_PASSWORD
|
||||
|
||||
### Security/Crypto
|
||||
- GIS_SYNC_AES_KEY
|
||||
- AUTHENTIK_WEBHOOK_HMAC_SECRET
|
||||
- AUDIT_ENCRYPTION_KEY
|
||||
- BACKUP_DASHBOARD_ADMIN_TOKEN
|
||||
|
||||
### Backup
|
||||
- BACKUP_ARCHIVE_KEY (AES-256 B2 client-side)
|
||||
- PGBACKREST_REPO1_CIPHER_PASS
|
||||
- STANDBY_REPLICATION_PASS (pentru CJ2 viitor)
|
||||
- B2_APPLICATION_KEY_ID
|
||||
- B2_APPLICATION_KEY
|
||||
- B2_BUCKET_NAME (beletage-gis-backups)
|
||||
- B2_ENDPOINT (s3.eu-central-003.backblazeb2.com)
|
||||
|
||||
### De generat la moment (viitor)
|
||||
- POSTGRES_GIS_TITILER_PASS (day 1 sprint 1)
|
||||
- POSTGRES_GIS_APP_RW_PASS (day 1 sprint 2)
|
||||
- GRAFANA_ADMIN_PASSWORD (sprint 4)
|
||||
- PROMETHEUS_BASIC_AUTH (sprint 4)
|
||||
- AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_SAS (când activăm)
|
||||
|
||||
---
|
||||
|
||||
## 21. Hardware/infrastructure check-list pre-GO
|
||||
|
||||
- ✅ Shop 10.10.10.84 reachable SSH (user dnz, groups docker+wheel)
|
||||
- ✅ Shop docker 29.2.1 instalat
|
||||
- ✅ Shop disk 3.3TB free
|
||||
- ✅ Shop RAM 225GB avail
|
||||
- ✅ Shop ports 5433, 6432 libere (POC ocupate)
|
||||
- ✅ Satra SSH (user bulibasa) reachable
|
||||
- ✅ Proxy Traefik v3.6.8 (10.10.10.199)
|
||||
- ✅ Sophos 80/443 only WAN (LAN internal safe)
|
||||
- ✅ NAS 10.10.10.10 rsync key deployed, symlink OK
|
||||
- ✅ Backblaze B2 bucket + scoped key tested
|
||||
- ✅ Authentik auth.beletage.ro v2025.2.4 operational
|
||||
- ✅ Infisical toate secretele pre-loaded
|
||||
- ✅ Cloudflare API token disponibil
|
||||
|
||||
---
|
||||
|
||||
## 22. Acceptance criteria (Definition of Done)
|
||||
|
||||
Production considerat stable când:
|
||||
- [ ] Toate 13 containere shop running + healthy
|
||||
- [ ] Cypress RLS suite 100% pass
|
||||
- [ ] Load test 500 QPS tiles + 100 QPS api sub 200ms p95
|
||||
- [ ] Backup test restore successful (monthly)
|
||||
- [ ] ArchiTools + eterra.live live pe shop DATABASE_URL
|
||||
- [ ] Orchestrator UNICUL scriitor gis_core
|
||||
- [ ] Zero error logs 72h continuous
|
||||
- [ ] Dashboard admin eterra.live arată toate verde
|
||||
- [ ] DR runbook validat (scenario R1 + R4 testat)
|
||||
- [ ] Grafana dashboards populated
|
||||
- [ ] Users notificați + feedback pozitiv
|
||||
|
||||
---
|
||||
|
||||
## 23. Contacts & ownership
|
||||
|
||||
- **Owner:** Marius Tarau (m.tarau@beletage.ro)
|
||||
- **Pair:** Claude Code (via Anthropic Opus 4.7 1M context)
|
||||
- **Escalation:** m.tarau@beletage.ro + (telefon Beletage)
|
||||
- **On-call during cutover weekend:** Marius primary, Claude assist
|
||||
|
||||
---
|
||||
|
||||
## 24. Changelog
|
||||
|
||||
- **v1 DRAFT** (2026-04-18) — initial propunere
|
||||
- **v2 FINAL** (2026-04-20) — approved, includes all decisions from Pas 1-7, POC results, security rotation status, multi-account ANCPI shuffle, super-tenant Beletage group, Azure credit alloc, NAS + B2 tested, Backup Dashboard plugin architecture
|
||||
|
||||
---
|
||||
|
||||
**END OF PLAN.**
|
||||
Next action: when Marius sends "go" → Sprint 1 Day 1 begins.
|
||||
@@ -0,0 +1,178 @@
|
||||
# Plan 002 — ArchiTools thin-client migration: review feedback from eterra.live side
|
||||
|
||||
**Date:** 2026-05-17
|
||||
**Reviewer:** Claude session in `/home/orchestrator/Code/eterra-live` (Marius)
|
||||
**Context:** Reviews the "Faza 0..7" plan proposed by the architools claude that routes architools enrichment through eterra.live.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — change of direction
|
||||
|
||||
The original plan made **eterra.live** the gateway for architools/planhub enrichment via `POST /api/internal/enrich`. **This is the wrong boundary.** The correct gateway is **`api.gis.ac`** (the `gis-api` service on shop), which is already designed for cross-app consumption (JWT + RLS + scope filtering, Hono on `10.10.10.84:3100`, fronted by Traefik).
|
||||
|
||||
eterra.live is a single-instance Next.js end-user product. Putting architools' geoportal behind it makes eterra.live a SPOF — any deploy/restart breaks architools cadastre. The right model treats both architools AND eterra.live as **peer consumers** of api.gis.ac.
|
||||
|
||||
eterra.live already migrated to api.gis.ac for `CfExtract` (Sprint 2 Day 3-4, 2026-04-20). Same migration pattern applies to architools, just bigger surface area.
|
||||
|
||||
---
|
||||
|
||||
## What's already in place (don't reinvent)
|
||||
|
||||
`api.gis.ac` exists in production:
|
||||
|
||||
- Repo: `gitadmin/gis-api`, local clone `~/Code/gis-api`
|
||||
- Stack: Next.js 16.1 + Hono 4 + Prisma 6.19 + Zod 4 + jose 6
|
||||
- Auth: HS256 JWT today (`GIS_API_DEV_JWT_SECRET`) — **migrating to Authentik OIDC before architools cutover** (decision locked 2026-05-17)
|
||||
- Claims: `{ sub, org_ids[], is_beletage_group, enrichment_scope: "none"|"basic"|"full", email? }` — `tenant` claim being added
|
||||
- RLS: `withUserContext()` sets `app.user_id / org_ids / is_beletage_group / enrichment_scope` per Postgres tx
|
||||
- Live endpoints (under `/api/v1/`):
|
||||
- `GET /me` — claims echo
|
||||
- `GET /parcela/:id` — full `GisFeature` read
|
||||
- `GET /search?q=&limit=` — UAT + cadastralRef text search
|
||||
- `GET/POST/PATCH /enrichment/cf*` — CfExtract CRUD with RLS
|
||||
- `GET /enrichment/catalog/:nrCadastral` — catalog metadata
|
||||
- `GET/POST /enrichment/cf/:id/pdf` — MinIO PDF stream
|
||||
- RLS test suite: 6 scenarios, asserts BOTH HTTP and direct-SQL paths (CI: `.gitea/workflows/rls-test.yml`)
|
||||
|
||||
What it does NOT have yet (gaps to close before architools cutover):
|
||||
|
||||
1. **Authentik OIDC** (HS256 still in use)
|
||||
2. **Live-fetch proxy endpoints** to orchestrator (parcel/tech, parcel/units, building/tech, building/condo-owners, imm-apps)
|
||||
3. **Scope-based field filtering** on `/parcela/:id` (currently leaks `enrichment.PROPRIETARI` regardless of scope)
|
||||
4. **`tenant` claim** in JWT
|
||||
5. **CORS allowlist** for cross-origin app calls
|
||||
6. **Per-tenant rate limiting**
|
||||
7. **Audit log table** (`gis_meta.api_audit`)
|
||||
|
||||
These will be added in a dedicated gis-api session before architools starts consuming. Full spec is in gis-api project memory (`~/.claude/projects/-home-orchestrator-Code-gis-api/memory/`):
|
||||
- `project_architools_planhub_plan.md` — endpoints to add, in priority order
|
||||
- `feedback_jwt_authentik_decision.md` — Authentik OIDC spec (scope mapping from LDAP groups)
|
||||
- `reference_orchestrator_endpoints.md` — exact contracts for the 5 LAN endpoints to proxy
|
||||
- `reference_cross_app_architecture.md` — the diagram + consumer responsibility matrix
|
||||
|
||||
---
|
||||
|
||||
## Revised plan (replaces architools claude's Faza 0..7)
|
||||
|
||||
### Faza A — Wait for gis-api gaps to close (managed in gis-api repo, ~3 days)
|
||||
|
||||
Driven from `~/Code/gis-api` in a separate session. NOT architools' work to do, but architools depends on it. Deliverables:
|
||||
|
||||
1. **PR1 (gis-api):** Authentik OIDC support alongside HS256 (`src/lib/auth.ts` issuer-based dual path). Add `tenant` claim. Eterra.live keeps working on HS256 during the overlap. Provisioning needed in Authentik:
|
||||
- New OAuth2 application + provider with slug `gis-api`
|
||||
- Property mappings for `enrichment_scope` derived from LDAP groups:
|
||||
- `beletage-staff` → `enrichment_scope=full`, `is_beletage_group=true`
|
||||
- `planhub-pro` → `full`
|
||||
- `planhub-free` / no match → `basic`
|
||||
- missing claim → REJECT
|
||||
- `tenant` claim from `aud` (application slug)
|
||||
2. **PR2 (gis-api):** 5 proxy endpoints + scope filter on `/parcela/:id` + CORS allowlist:
|
||||
```
|
||||
POST /api/v1/parcel/tech → orchestrator POST /api/v1/parcel/tech
|
||||
POST /api/v1/parcel/units/fetch → orchestrator POST /api/v1/parcel-units/fetch
|
||||
POST /api/v1/parcel/imm-apps → orchestrator POST /api/v1/imm-apps
|
||||
POST /api/v1/building/tech → orchestrator POST /api/v1/building/tech
|
||||
POST /api/v1/building/condo-owners → orchestrator POST /api/v1/building/condo-owners
|
||||
```
|
||||
All require `enrichment_scope >= basic`. Rewrite `correlationId` server-side to `${tenant}:${user_sub[:8]}:${requestId}`.
|
||||
3. **PR3 (gis-api):** Per-tenant rate limit + `gis_meta.api_audit` migration.
|
||||
|
||||
When PR1+PR2 are green and deployed, signal back to this plan and proceed with Faza B.
|
||||
|
||||
### Faza B — ArchiTools secrets + Authentik client (½ day)
|
||||
|
||||
- In Authentik: create OAuth2 client for architools (slug `architools-app`). Configure redirect URI for its NextAuth flow.
|
||||
- Infisical adds (prod env, path `/`):
|
||||
- `AUTHENTIK_CLIENT_ID` / `AUTHENTIK_CLIENT_SECRET` / `AUTHENTIK_ISSUER` / `AUTHENTIK_JWKS_URL` for architools
|
||||
- `GIS_API_URL=https://api.gis.ac`
|
||||
- `NEXT_PUBLIC_MARTIN_URL=https://tiles.gis.ac`
|
||||
- `NEXT_PUBLIC_PMTILES_URL=https://pmtiles.gis.ac/overview.pmtiles`
|
||||
- NextAuth on architools: add Authentik OIDC provider. Store `access_token` in session. ⚠️ DO NOT mint HS256 JWTs server-side anymore — pass the Authentik `access_token` directly as Bearer to gis-api.
|
||||
|
||||
### Faza C — Rip-out (1 day, ONE PR, feature-flagged)
|
||||
|
||||
`NEXT_PUBLIC_USE_GIS_AC=1` flag controls cutover. With flag on:
|
||||
|
||||
- Geoportal map sources flip to `pmtiles.gis.ac` + `tiles.gis.ac`
|
||||
- All `src/app/api/eterra/*` and `src/app/api/ancpi/*` route handlers replaced by thin wrappers that forward to api.gis.ac with the user's access_token
|
||||
- All `src/modules/parcel-sync/**` and parcel-sync UI removed
|
||||
- `src/config/{modules,navigation,flags}.ts` cleanup
|
||||
|
||||
With flag off: old behavior retained. **Do not drop Prisma tables (GisFeature, GisUat, GisSyncRun, GisSyncRule, CfExtract) in this PR.** Keep them as dead columns for 2-3 days post-cutover so rollback stays cheap. Drop in a follow-up PR after validation in prod.
|
||||
|
||||
Stop `martin` container on satra ONLY after flag-on is verified working in prod. The PMTiles webhook on satra:9876 — **does not exist**; the architools claude's plan had this wrong. The actual PMTiles trigger is the orchestrator cron at 03:00 EEST + admin button in eterra.live, hitting `gis-tippecanoe-builder` on **shop** at port 9876.
|
||||
|
||||
### Faza D — Thin client (1-2 days)
|
||||
|
||||
- `src/lib/gis-api-client.ts` — fetch wrapper with the Authentik access_token from session
|
||||
- `gisApi.parcela.get(id)` → `GET /api/v1/parcela/:id`
|
||||
- `gisApi.search(q, limit)` → `GET /api/v1/search`
|
||||
- `gisApi.parcel.tech({siruta, cadastralRef, force})` → `POST /api/v1/parcel/tech`
|
||||
- `gisApi.parcel.unitsFetch(...)` → `POST /api/v1/parcel/units/fetch`
|
||||
- `gisApi.parcel.immApps(...)` → `POST /api/v1/parcel/imm-apps`
|
||||
- `gisApi.building.tech(...)` → `POST /api/v1/building/tech`
|
||||
- `gisApi.building.condoOwners(...)` → `POST /api/v1/building/condo-owners`
|
||||
- `gisApi.enrichment.cf.list/get/create/patch/uploadPdf/getPdf/getCatalog` (mirror eterra.live's existing usage)
|
||||
- NO separate `scope-mapper.ts` — gis-api resolves scope from Authentik claims itself.
|
||||
- NO separate `gis-api-token.ts` — Authentik access_token is the JWT, NextAuth session already has it.
|
||||
|
||||
### Faza E — Geoportal rewrite (2 days)
|
||||
|
||||
- `map-viewer.tsx`: sources `pmtiles://pmtiles.gis.ac/overview.pmtiles` + Martin `tiles.gis.ac/{view}/{z}/{x}/{y}`. Drop satra Martin entirely.
|
||||
- `search-bar.tsx` → `gisApi.search()`.
|
||||
- `feature-info-panel.tsx` → `gisApi.parcela.get()` + (on missing enrichment) `gisApi.parcel.tech()` + `gisApi.parcel.unitsFetch()` for buildings.
|
||||
- `basemap-switcher.tsx`: same config as eterra.live (liberty/dark/satellite/orto/topo50). Source files are public CSS/tile URLs; literal copy is OK.
|
||||
- Decision for `boundary-check`, `cf-status`, `export`, `pad`, `piz`: if architools keeps them, rewrite as thin proxies to api.gis.ac equivalents (which may need new endpoints — defer that scoping until each is actually needed).
|
||||
|
||||
### Faza F — ePay / CF ordering (½ day)
|
||||
|
||||
- Architools UI for "Comandă extras CF" calls `gisApi.enrichment.cf.create({nrCadastral, type, ...})` directly.
|
||||
- Status + download list: `gisApi.enrichment.cf.list({...})` and `getPdf(id)`.
|
||||
- The `tenant=architools` claim in the access_token tells gis-api which billing account to attribute the order to (future-work for billing routing, but the field is in place from Faza A).
|
||||
- Delete `src/app/api/ancpi/*` entirely.
|
||||
|
||||
### Faza G — Test E2E + deploy (½ day)
|
||||
|
||||
- `/geoportal` in architools renders tiles from gis.ac, click parcel → enrichment via api.gis.ac → live-fetch fallback triggers orchestrator.
|
||||
- Verify zero references to `architools_postgres.GisFeature` remain in code AND that no DB query goes to the deprecated tables.
|
||||
- Type-check + lint + build clean.
|
||||
- Commit + push + Portainer redeploy.
|
||||
- 24h grace period: old stack (Martin satra, deprecated parcel-sync code) kept around. After validation, follow-up PR drops Prisma tables + stops martin satra container.
|
||||
|
||||
### Faza H — Planhub (separate PR, post-architools)
|
||||
|
||||
- Same pattern. Different scope mapping: `planhub-free` → `basic`, `planhub-pro` → `full`.
|
||||
- Already covered by gis-api's tenant claim from Faza A.
|
||||
|
||||
---
|
||||
|
||||
## Specific corrections to the original plan
|
||||
|
||||
1. **`POST /api/internal/enrich` on eterra.live** — REMOVE. Use `api.gis.ac` directly.
|
||||
2. **Bearer `$ARCHITOOLS_INTERNAL_TOKEN` static token** — REMOVE. Use Authentik OIDC access_token.
|
||||
3. **`GIS_API_JWT_SECRET (shared)` in Infisical** — REMOVE. With Authentik OIDC, gis-api fetches JWKS from `https://auth.beletage.ro/application/o/gis-api/jwks/` — no shared secrets between apps.
|
||||
4. **`scope-mapper.ts` in architools** — REMOVE. Scope is in the JWT claims; architools doesn't compute it.
|
||||
5. **PMTiles webhook satra:9876** — does not exist. The builder is `gis-tippecanoe-builder` on **shop:9876**, fired by orchestrator cron + eterra.live admin button.
|
||||
6. **Single-PR rip-out** — replace with feature-flagged migration (`NEXT_PUBLIC_USE_GIS_AC=1`), 2-3 day overlap, drop dead code in a separate follow-up PR after prod validation.
|
||||
7. **eterralive-coordinator** terminology — drop. The pool is "owned" by `gis-sync-orchestrator` (LAN-only on shop:3101). gis-api is the auth/scope/audit gateway in front of it.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Authentik OIDC client provisioning** — needs to be done first; until then, no architools migration. Marius or a sys-admin session needs to set up the OAuth2 provider in Authentik UI.
|
||||
2. **Audit table schema** — does `gis_meta` already have an `api_audit` table? Probably no. Migration needed in same PR as proxy endpoints.
|
||||
3. **Rate-limit defaults** — what's the cap per tenant? 100 req/min seems safe; configurable per tenant via Infisical JSON env. Confirm with Marius.
|
||||
4. **Architools' `archi.*` schema writes** — currently architools writes Canvas, Job, RegistryAudit, RgiSearchTemplate directly via its own Prisma. Migration here is OPTIONAL for the cadastre work; could be done in a later sprint. Don't bundle it.
|
||||
|
||||
---
|
||||
|
||||
## Reference docs
|
||||
|
||||
- gis-api project memory: `~/.claude/projects/-home-orchestrator-Code-gis-api/memory/MEMORY.md`
|
||||
- Plan 001 v2 master: `~/Code/ArchiTools/docs/plans/001-v2-central-gis-db-multi-app.md`
|
||||
- Tile data sources / cache / gis.ac rule: `~/Code/busc-infra/references/TILE-DATA-SOURCES.md`
|
||||
- Orchestrator workers + schema: `~/Code/busc-infra/references/SPRINT3.md`
|
||||
- GIS stack infrastructure: `~/Code/busc-infra/references/GIS-STACK.md`
|
||||
- Authentik configuration: `~/Code/busc-infra/references/AUTHENTIK.md`
|
||||
- Sprint 2 (gis-api delivery): `~/Code/busc-infra/references/SPRINT2.md`
|
||||
@@ -0,0 +1,257 @@
|
||||
# Plan 003 — ArchiTools cutover execution (post-PR1+PR2)
|
||||
|
||||
**Date:** 2026-05-17
|
||||
**Author:** gis-api session (Claude, `/home/orchestrator/Code/gis-api`)
|
||||
**Supersedes Faza A–H of:** `002-architools-thin-client-review.md`
|
||||
**Starting brief for:** next ArchiTools session
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — what changed since 002 was written
|
||||
|
||||
PR1 (Authentik OIDC dual-support, `ad1825a` deployed 22:25) and PR2 (5 proxy endpoints + scope filter + CORS + rate limit + audit + RLS smoke test, `fc459e0` deployed 22:55) are both live on `api.gis.ac`. Migration 008-api-audit.sql applied to gis DB. RLS leak from 2026-05-17 patched; `relrowsecurity` enforced via CI smoke test.
|
||||
|
||||
**Skip Faza A entirely** — its full content is done.
|
||||
|
||||
**One pre-cutover gis-api blocker** discovered (PR2.1 below) — must ship before architools' first request hits the rate limiter or audit log starts collecting unreadable 40-char tenant strings.
|
||||
|
||||
Everything else in 002 (Faza B–G) is executable against the deployed api.gis.ac. Faza H (Planhub) deferred — separate PR after architools is stable.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ PR2.1 — Tenant-slug translation (gis-api blocker)
|
||||
|
||||
**Status:** NOT YET SHIPPED. ~30 min of work in gis-api repo.
|
||||
|
||||
**Problem:** PR2 lifts `aud` → `claims.tenant` verbatim. For Authentik authorization_code AND client_credentials flows, `aud` is the OAuth client_id (40-char opaque string like `V59GMiYle87yd9VZOgUmdSmzYQALqNsKVAUR6QMi`), not the friendly slug `architools`. Two real consequences:
|
||||
|
||||
1. **Rate limiter mis-keyed:** `GIS_API_TENANT_RATE_LIMITS` env JSON uses friendly slugs (`{"architools":500,"planhub":120,...}`). Bucket lookup on `claims.tenant` = the 40-char client_id → miss → falls to `_default` (100 rpm). architools would silently run at 5× lower throughput than configured.
|
||||
2. **Audit log unreadable:** `gis_meta."ApiAudit".tenant` stores the 40-char string. Ops queries (`SELECT … WHERE tenant = 'architools'`) return zero rows. Future billing routing has the same problem.
|
||||
|
||||
**Fix shape (small, surgical):**
|
||||
|
||||
- New env: `GIS_API_TENANT_MAP` — JSON `{"<client_id>":"<slug>", ...}`. Missing entry → fall back to verbatim aud. Missing env → no translation (current behavior).
|
||||
- In `src/lib/auth.ts`, after `audToTenant(payload.aud)`, look up in the map.
|
||||
- Unit test that adds a case to `tests/auth/oidc.test.ts` (or new `tenant-map.test.ts`).
|
||||
- `.env.example` entry. Infisical `/gis-api` adds the secret.
|
||||
|
||||
**Who ships this:** gis-api session (next firing or this one if Marius approves). Architools cutover is gated on this being deployed first.
|
||||
|
||||
**Required values:** ArchiTools client_id (confirmed in user message: `V59GMiYle87yd9VZOgUmdSmzYQALqNsKVAUR6QMi` → `architools`). Planhub + eterra-live client_ids: TBD as those apps onboard.
|
||||
|
||||
---
|
||||
|
||||
## Faza B — ArchiTools secrets + Authentik client (½ day)
|
||||
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| **Owner** | ArchiTools session + Marius (Infisical) |
|
||||
| **Repo** | ArchiTools (NextAuth config) + Infisical (secrets) + Authentik (no changes — provider pk=6 already exists with `gis-api enrichment scope` mapping attached, per `reference-authentik-provisioned-state` memory) |
|
||||
| **gis-api involvement** | Zero. Just confirm: architools provider pk=6 in Authentik issues tokens with `iss = https://auth.beletage.ro/application/o/architools/`, `aud = V59GMiYle87yd9VZOgUmdSmzYQALqNsKVAUR6QMi`. **NOTE:** gis-api's current OIDC path validates against `GIS_API_AUTHENTIK_ISSUER = https://auth.beletage.ro/application/o/gis-api/` — architools tokens will NOT match this issuer and will fall through to HS256 path → 401. See **"Multi-issuer JWT support"** below — this is a second gis-api blocker that PR2.1 (or PR2.2) must address. |
|
||||
|
||||
**Infisical adds** (`/architools` folder, prod env):
|
||||
- `AUTHENTIK_CLIENT_ID` — value from Authentik provider pk=6
|
||||
- `AUTHENTIK_CLIENT_SECRET` — from provider pk=6
|
||||
- `AUTHENTIK_ISSUER` = `https://auth.beletage.ro/application/o/architools/`
|
||||
- `AUTHENTIK_JWKS_URL` = `https://auth.beletage.ro/application/o/architools/jwks/`
|
||||
- `GIS_API_URL` = `https://api.gis.ac`
|
||||
- `NEXT_PUBLIC_MARTIN_URL` = `https://tiles.gis.ac` *(verify infrastructure live first)*
|
||||
- `NEXT_PUBLIC_PMTILES_URL` = `https://pmtiles.gis.ac/overview.pmtiles` *(same)*
|
||||
|
||||
**ArchiTools NextAuth changes:**
|
||||
- Add Authentik OIDC provider. Request `scope: "openid profile email enrichment"` so the `enrichment_scope` claim is in the issued access_token (per the scope mapping attached to provider pk=6).
|
||||
- Session callback stores `access_token` on the session object (not the JWT — different things; Authentik issues the JWT we forward).
|
||||
- **Remove** any HS256 minting code (`gis-api-token.ts` and similar — per 002 corrections).
|
||||
|
||||
**Verification before moving to Faza C:**
|
||||
- `architools` can mint an Authentik authorization_code token end-to-end (login flow works).
|
||||
- Decoded token includes `enrichment_scope`, `is_beletage_group`, `org_ids` claims.
|
||||
- `curl -H "Authorization: Bearer <token>" https://api.gis.ac/api/v1/me` returns 200 with those claims. **This curl will FAIL today** until multi-issuer support ships — see PR2.2 below.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ PR2.2 — Multi-issuer JWT support (second gis-api blocker)
|
||||
|
||||
**Status:** NOT YET SHIPPED.
|
||||
|
||||
**Problem:** PR1's `src/lib/auth.ts` validates `iss === GIS_API_AUTHENTIK_ISSUER` (the gis-api provider's issuer). architools tokens carry `iss = https://auth.beletage.ro/application/o/architools/` — different issuer, different JWKS endpoint. Current code falls through to HS256 → 401.
|
||||
|
||||
**Fix shape:**
|
||||
- Multi-issuer config: env `GIS_API_ACCEPTED_ISSUERS` = JSON array `["https://auth.beletage.ro/application/o/gis-api/","https://auth.beletage.ro/application/o/architools/", …]`.
|
||||
- For each accepted issuer, derive its JWKS URL via the well-known discovery path (`{iss}/.well-known/openid-configuration`) at boot, or accept a parallel `GIS_API_JWKS_URLS` JSON map keyed by issuer. The latter is simpler + avoids a discovery call at first-token-time.
|
||||
- Cache one `createRemoteJWKSet` per issuer at module scope.
|
||||
- Pre-parse `iss` from the token (already done) and route to the matching JWKS getter.
|
||||
|
||||
**Bundle PR2.1 + PR2.2** as one gis-api PR ("PR3" in the cutover flow): they touch the same file (`auth.ts`), share a config layout (per-issuer + per-tenant maps), and architools needs both before it can call.
|
||||
|
||||
**Who ships:** gis-api session, ~1 hour total.
|
||||
|
||||
**Required values (need to be in Infisical /gis-api):**
|
||||
- `GIS_API_ACCEPTED_ISSUERS` = `["https://auth.beletage.ro/application/o/gis-api/","https://auth.beletage.ro/application/o/architools/"]` (add planhub+eterra-live later)
|
||||
- `GIS_API_JWKS_URLS` = `{"https://auth.beletage.ro/application/o/gis-api/":"https://auth.beletage.ro/application/o/gis-api/jwks/","https://auth.beletage.ro/application/o/architools/":"https://auth.beletage.ro/application/o/architools/jwks/"}`
|
||||
- `GIS_API_TENANT_MAP` = `{"V59GMiYle87yd9VZOgUmdSmzYQALqNsKVAUR6QMi":"architools"}` (PR2.1)
|
||||
|
||||
---
|
||||
|
||||
## Faza C — Rip-out (1 day, feature-flagged)
|
||||
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| **Owner** | ArchiTools session |
|
||||
| **Repo** | ArchiTools only |
|
||||
| **gis-api involvement** | None |
|
||||
|
||||
`NEXT_PUBLIC_USE_GIS_AC=1` flag controls cutover. Plan-as-written stands.
|
||||
|
||||
**One verification gis-api can pre-confirm:** CORS allowlist on api.gis.ac includes `architools.beletage.ro` (deployed in PR2 `src/lib/hono-app.ts`). Browser preflight from architools origin → 204 with `Access-Control-Allow-Origin` echo (smoke-tested 22:55).
|
||||
|
||||
**Do not drop Prisma tables** (GisFeature, GisUat, GisSyncRun, GisSyncRule, CfExtract) in this PR — keep as dead columns 2-3 days. Drop in follow-up PR after prod validation.
|
||||
|
||||
---
|
||||
|
||||
## Faza D — Thin client (1-2 days)
|
||||
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| **Owner** | ArchiTools session |
|
||||
| **Repo** | ArchiTools only |
|
||||
| **gis-api involvement** | None — all endpoints exist as specified |
|
||||
|
||||
**Endpoint contract confirmation** (each verified against PR2 deploy):
|
||||
|
||||
| ArchiTools method | gis-api endpoint | Auth | Scope |
|
||||
|---|---|---|---|
|
||||
| `gisApi.me()` | `GET /api/v1/me` | Bearer | any |
|
||||
| `gisApi.parcela.get(id)` | `GET /api/v1/parcela/:id` | Bearer | `>= basic`; `none` → 403; `basic` redacts PROPRIETARI / PROPRIETARI_VECHI / NR_CF / DOC |
|
||||
| `gisApi.search(q, limit)` | `GET /api/v1/search?q=&limit=` | Bearer | any |
|
||||
| `gisApi.parcel.tech(body)` | `POST /api/v1/parcel/tech` | Bearer | `>= basic` |
|
||||
| `gisApi.parcel.unitsFetch(body)` | `POST /api/v1/parcel/units/fetch` | Bearer | `>= basic` |
|
||||
| `gisApi.parcel.immApps(body)` | `POST /api/v1/parcel/imm-apps` | Bearer | `>= basic` |
|
||||
| `gisApi.building.tech(body)` | `POST /api/v1/building/tech` | Bearer | `>= basic` |
|
||||
| `gisApi.building.condoOwners(body)` | `POST /api/v1/building/condo-owners` | Bearer | `>= basic` |
|
||||
| `gisApi.enrichment.cf.list({...})` | `GET /api/v1/enrichment/cf?…` | Bearer | RLS-filtered |
|
||||
| `gisApi.enrichment.cf.get(id)` | `GET /api/v1/enrichment/cf/:id` | Bearer | RLS-filtered |
|
||||
| `gisApi.enrichment.cf.create(body)` | `POST /api/v1/enrichment/cf` | Bearer | RLS-owned write |
|
||||
| `gisApi.enrichment.cf.patch(id, body)` | `PATCH /api/v1/enrichment/cf/:id` | Bearer | RLS-owned write |
|
||||
| `gisApi.enrichment.cf.uploadPdf(id, buf)` | `POST /api/v1/enrichment/cf/:id/pdf` | Bearer + `Content-Type: application/pdf` | RLS-owned write |
|
||||
| `gisApi.enrichment.cf.getPdf(id)` | `GET /api/v1/enrichment/cf/:id/pdf` | Bearer | RLS-filtered |
|
||||
| `gisApi.enrichment.catalog(nrCadastral)` | `GET /api/v1/enrichment/catalog/:nrCadastral` | Bearer | `>= basic` |
|
||||
|
||||
**Proxy request payload shape** (parcel/tech, parcel/units/fetch, building/tech, building/condo-owners):
|
||||
```ts
|
||||
{ siruta: string /* /^\d{3,7}$/ */, cadastralRef: string /* 1..200 */, force?: boolean }
|
||||
```
|
||||
For `parcel/imm-apps` add `layerId?: "TERENURI_ACTIVE" | "CLADIRI_ACTIVE"`.
|
||||
|
||||
**Do NOT** pass `correlationId` from the client — gis-api rewrites it server-side. Anything you send is silently overwritten.
|
||||
|
||||
**Response shape:** orchestrator response forwarded verbatim. Per `reference-orchestrator-endpoints` memory: `{status: "ok", data: {...}}` on success or `{error: "...", code}` on failure.
|
||||
|
||||
**Rate limit headers** on every response: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` (seconds). On exhaustion: `429 { error: "rate_limited", retryAfterSec }`.
|
||||
|
||||
**No correlationId echoed** in proxy response today (see `project-audit-correlation-echo` memory). If architools wants to correlate, generate a client-side request ID and log it alongside; gis-api's `gis_meta."ApiAudit"` row has the server-side traceId in format `${tenant_slug}:${sub.slice(0,8)}:${reqId}` indexed by `(tenant, ts DESC)`.
|
||||
|
||||
---
|
||||
|
||||
## Faza E — Geoportal rewrite (2 days)
|
||||
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| **Owner** | ArchiTools session |
|
||||
| **Repo** | ArchiTools only |
|
||||
| **Prerequisite to verify before starting** | `tiles.gis.ac` (Martin) and `pmtiles.gis.ac/overview.pmtiles` are live. NOT a gis-api concern — infrastructure on shop. The 002 brief assumed they exist; confirm with infra team before ripping out the satra Martin source. |
|
||||
| **gis-api involvement** | Confirm via curl `curl -I https://tiles.gis.ac/` + `curl -I https://pmtiles.gis.ac/overview.pmtiles`. If either 404s, that's an infrastructure blocker for Faza E (not gis-api). |
|
||||
|
||||
`map-viewer.tsx`, `search-bar.tsx`, `feature-info-panel.tsx`, `basemap-switcher.tsx` rewrites stand as-written in 002.
|
||||
|
||||
For `feature-info-panel.tsx` calling `gisApi.parcel.tech()` on missing enrichment: that path requires `enrichment_scope >= basic`. Beletage staff (Arhitecti LDAP group) get `enrichment_scope=full` from the scope mapping → no friction. Anonymous / non-Arhitecti users get `none` → 403 → panel must handle gracefully (probably hide enrichment section + show "log in for details").
|
||||
|
||||
`boundary-check / cf-status / export / pad / piz`: defer scoping per 002. If architools keeps any, evaluate whether a thin proxy to api.gis.ac is enough OR if a new endpoint is needed. New endpoints = new gis-api PR — out of scope for this cutover.
|
||||
|
||||
---
|
||||
|
||||
## Faza F — ePay / CF ordering (½ day)
|
||||
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| **Owner** | ArchiTools session |
|
||||
| **Repo** | ArchiTools only |
|
||||
| **gis-api involvement** | None — endpoints already exist |
|
||||
|
||||
UI flow: "Comandă extras CF" button calls `gisApi.enrichment.cf.create({nrCadastral, type: "epay", …})`. Status + download list via `enrichment.cf.list` + `enrichment.cf.getPdf(id)`.
|
||||
|
||||
**Tenant claim for billing routing:** post-PR2.1, `claims.tenant === "architools"` (human-readable). The CfExtract row will be RLS-owned by the user's `sub`, but ops can attribute billing via the audit log + `tenant` join. Future billing PR will pick this up.
|
||||
|
||||
Delete `src/app/api/ancpi/*` entirely.
|
||||
|
||||
---
|
||||
|
||||
## Faza G — Test E2E + deploy (½ day)
|
||||
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| **Owner** | ArchiTools session |
|
||||
| **Repo** | ArchiTools only |
|
||||
| **gis-api involvement** | Read-only: watch container logs + `gis_meta."ApiAudit"` for first architools traffic |
|
||||
|
||||
Test plan:
|
||||
1. `/geoportal` in architools renders tiles from `gis.ac` (no satra requests).
|
||||
2. Click a parcel → `gisApi.parcela.get(id)` returns full enrichment (Beletage staff scope=full).
|
||||
3. Click parcel with no enrichment → `gisApi.parcel.tech()` triggers orchestrator live-fetch → enriched response returns.
|
||||
4. Place a CF order → `gisApi.enrichment.cf.create()` → row appears in `gis_meta."CfExtract"` with `userId = <Arhitecti user sub>`.
|
||||
5. Audit log: `SELECT tenant, endpoint, statusCode, ts FROM gis_meta."ApiAudit" WHERE tenant='architools' ORDER BY ts DESC LIMIT 20;` shows architools traffic with human-readable tenant.
|
||||
6. Rate limit: 600 rapid calls (architools is configured for 500 rpm) → ~500 succeed, rest 429 with `X-RateLimit-Reset` ≤ 60.
|
||||
7. Zero references to `architools_postgres.GisFeature` remain in code.
|
||||
8. Type-check + lint + build clean.
|
||||
9. Portainer redeploy.
|
||||
10. **24h grace period:** keep old stack (Martin satra, deprecated parcel-sync code) until validation. Follow-up PR drops dead Prisma tables + stops martin satra container.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites checklist (in execution order)
|
||||
|
||||
Mark these done before opening the architools session:
|
||||
|
||||
- [ ] **gis-api PR2.1 (tenant translation) + PR2.2 (multi-issuer JWT)** shipped + deployed on api.gis.ac. Verified with: mint an architools-issuer token via client_credentials → call `https://api.gis.ac/api/v1/me` → 200 + `claims.tenant === "architools"`.
|
||||
- [ ] **Infisical `/architools`** populated: `AUTHENTIK_CLIENT_ID`, `AUTHENTIK_CLIENT_SECRET`, `AUTHENTIK_ISSUER`, `AUTHENTIK_JWKS_URL`, `GIS_API_URL`, `NEXT_PUBLIC_MARTIN_URL`, `NEXT_PUBLIC_PMTILES_URL`.
|
||||
- [ ] **Infisical `/gis-api`** populated: `GIS_API_ACCEPTED_ISSUERS`, `GIS_API_JWKS_URLS`, `GIS_API_TENANT_MAP` (PR2.1/PR2.2 inputs).
|
||||
- [ ] **tiles.gis.ac + pmtiles.gis.ac** live (curl 200 / non-404). Infra confirms.
|
||||
- [ ] **architools-session Claude** has read this plan + `MEMORY.md` of `~/.claude/projects/-home-orchestrator-Code-gis-api/memory/`.
|
||||
|
||||
---
|
||||
|
||||
## Items resolved from 002's "Open questions"
|
||||
|
||||
| 002 question | Resolution |
|
||||
|---|---|
|
||||
| **1. Authentik OIDC client provisioning** | DONE. architools provider pk=6, scope mapping `41b23bc3-bdd8-4a61-b975-6e0eff56df72` attached. See `reference-authentik-provisioned-state`. |
|
||||
| **2. Audit table schema exists?** | DONE. `gis_meta."ApiAudit"` created by busc-infra `0693457` (008-api-audit.sql). Mirrored in `prisma/schema.prisma` (`model ApiAudit`). |
|
||||
| **3. Rate-limit defaults** | DONE. `GIS_API_TENANT_RATE_LIMITS` JSON env, defaults to 100 rpm. Deployed seed: `{"architools":500,"planhub":120,"eterra-live":300,"_default":100}`. ⚠️ but mis-keyed until PR2.1 ships. |
|
||||
| **4. archi.* schema writes (Canvas, Job, etc.)** | DEFERRED. Out of scope for this cutover. Later sprint. |
|
||||
|
||||
---
|
||||
|
||||
## Items obsolete in 002
|
||||
|
||||
- **Faza A** entire content — done.
|
||||
- **"PR3 (gis-api): Per-tenant rate limit + audit migration"** — done as part of actual PR2 (rolled together).
|
||||
- **"`scope-mapper.ts` in architools"** correction — already noted; reaffirm: architools must NOT compute scope client-side.
|
||||
- **"`GIS_API_JWT_SECRET (shared)` in Infisical"** correction — still valid; do not pull this into architools NextAuth.
|
||||
|
||||
---
|
||||
|
||||
## What this gis-api session will produce next (if Marius approves)
|
||||
|
||||
**PR2.1 + PR2.2** (bundled into one gis-api commit): `src/lib/auth.ts` multi-issuer + tenant-map, `.env.example` adds, unit tests. ~50–80 LOC, ~1 hour including typecheck+build+push. Migration of architools onto api.gis.ac is gated on this landing.
|
||||
|
||||
After that, this gis-api session is done — architools cutover is owned by the architools session.
|
||||
|
||||
---
|
||||
|
||||
## Reference
|
||||
|
||||
- gis-api memory: `~/.claude/projects/-home-orchestrator-Code-gis-api/memory/MEMORY.md`
|
||||
- gis-api recent commits: `b34c58e` (PR1) → `ad1825a` (PR1.1) → `a43673a..fc459e0` (PR2 = 4 commits)
|
||||
- busc-infra: `0693457` (migration 008)
|
||||
- Authentik provider pks: gis-api pk=8, architools pk=6; scope mapping pk=41b23bc3-bdd8-4a61-b975-6e0eff56df72
|
||||
- Live container: `gis-api` on shop, image at `fc459e0`, healthcheck `https://api.gis.ac/api/healthz`
|
||||
@@ -0,0 +1,143 @@
|
||||
# Plan 006 — ePay CF-Extract as a Multi-Tenant Service
|
||||
|
||||
**Status:** design (executable). **Author:** deep-dive 2026-06-04 + hardening 2026-06-05.
|
||||
**Prereq reading:** plans 002/003 (architools thin-client cutover), `project_epay_cf_roadmap_2026_06` memory.
|
||||
|
||||
## Why this exists
|
||||
|
||||
The CF-extract capability (ANCPI ePay paid extracts + the free `cf-intern` copycf
|
||||
circuit) will be offered beyond internal use — multi-tenant (ArchiTools, eterra-live,
|
||||
Planhub, external paying customers). Today it runs as an **in-process queue inside
|
||||
ArchiTools** (`src/modules/parcel-sync/services/epay-*`). That path is now hardened
|
||||
(commit `f49fdb1`: cart hygiene, auth/IDOR gates, single-page fetch, parallel
|
||||
downloads, recover-by-extractId) and is **billing-safe and correct for the internal
|
||||
tool** — but it is the wrong *shape* for a service:
|
||||
|
||||
- Queue + ePay session are in-memory globals → die on redeploy mid-batch.
|
||||
- One serial cart per process → no multi-tenant throughput.
|
||||
- No catalog dedup on the paid path → the same parcel is paid for repeatedly.
|
||||
- `EPAY_ORDERING_VIA_GIS_AC=false` because **gis-api `POST /enrichment/cf` inserts a
|
||||
pending row that nothing fulfills** — the orchestrator has no ePay worker. Plan 003
|
||||
Faza F ("endpoints already exist") is wrong: the fulfiller is the unwritten keystone.
|
||||
|
||||
This plan is the path from the hardened-internal state to a real service. Each phase
|
||||
is independently shippable; do them in order, validate, then flip the flag per-tenant.
|
||||
|
||||
## Invariants carried over from the hardened internal path (do NOT regress)
|
||||
|
||||
These were learned the hard way (2026-06-04 incident, order 10009605). The new worker
|
||||
MUST preserve every one:
|
||||
|
||||
1. **Submit is timeout-resilient.** A slow `EditCartSubmit` that ANCPI completes must
|
||||
never be marked failed. Resolve the order via `findNewOrderId(previous, known)` which
|
||||
never adopts a stale/known id. (`SUBMIT_TIMEOUT_MS`, today's fix.)
|
||||
2. **Cart hygiene invariant.** ePay has ONE global cart per account; `EditCartSubmit`
|
||||
checks out everything in it. After N adds a clean cart reports exactly N items — any
|
||||
excess = orphan from a crash → wipe + abort, never submit a cart you didn't build.
|
||||
3. **CF-number matching is authoritative; index fallback is `review`, not `completed`.**
|
||||
4. **`%PDF` magic-byte check** on every download (expired session returns login HTML).
|
||||
5. **Single-page order fetch** via `itemsPerPage` (5/page default silently drops docs).
|
||||
6. **Recover is idempotent** (re-poll + re-download an already-paid order, no new charge).
|
||||
|
||||
## Phase A — DB-backed fulfiller worker (`eterra.cf-epay`) — THE KEYSTONE
|
||||
|
||||
A pg-boss worker in `gis-sync-orchestrator` (next to `enrichment-drainer`, cron 1–2 min).
|
||||
**The CfExtract row IS the work item** — no in-memory queue.
|
||||
|
||||
- **Claim:** `SELECT … FROM gis_enrichment."CfExtract" WHERE status='pending' AND
|
||||
type='epay' [AND account-compatible] ORDER BY "createdAt" FOR UPDATE SKIP LOCKED LIMIT N;
|
||||
UPDATE → status='claimed', claimedAt=now()`. SKIP LOCKED → two instances never grab the
|
||||
same rows.
|
||||
- **State machine** (each transition = one UPDATE = a precise resumable marker):
|
||||
`pending → claimed → cart → submitted_unconfirmed → polling → downloading → completed |
|
||||
review | failed | cancelled`. Extend gis-api's `ExtractStatus` enum
|
||||
(`gis-api/src/routes/enrichment.ts:9`) with `claimed`, `submitted_unconfirmed`, `review`.
|
||||
- **Crash recovery:** a boot **reaper** requeues rows stuck in a transitional state past a
|
||||
heartbeat TTL. `submitted_unconfirmed` rows are resolved via the recover pattern (find
|
||||
the order at ANCPI, never re-charge). This structurally eliminates the in-memory-queue
|
||||
orphan class (criticals C2).
|
||||
- **Idempotent submit:** before `EditCartSubmit`, persist on the claimed rows the account's
|
||||
current latest orderId + the intended `nrCadastral` set. On timeout/crash, resume
|
||||
re-runs `findNewOrderId` against that snapshot — never adopts a stale id.
|
||||
- Port the hardened `epay-client` here (see Phase G — shared package).
|
||||
|
||||
## Phase B — `epay_accounts` pool with one-batch-per-account lock
|
||||
|
||||
Mirror `gis_meta.eterra_accounts` (busc-infra migration 004): AES-256-GCM creds,
|
||||
`status active/blocked/retired`, `blocked_reason`, `credits_cached`, optional hourly cap,
|
||||
`in_flight_batch_id`.
|
||||
|
||||
- `pickEpayAccount`: `FOR UPDATE SKIP LOCKED`, but because ePay's cart is **global per
|
||||
account**, atomically set `in_flight_batch_id` (status `busy`) so no second batch can
|
||||
touch that account's cart. This is the structural fix for cart contamination (C1) in the
|
||||
pooled world.
|
||||
- Refuse to claim a batch larger than the account's cached credits. ePay credits are a
|
||||
**hard consumable (real money)** — unlike the soft eTerra quota, the credit cap is
|
||||
mandatory, not advisory.
|
||||
|
||||
## Phase C — Catalog dedup (largest recurring economic win)
|
||||
|
||||
`CfExtractCatalog` is written **only** on the `cf-intern` path today; nothing writes it
|
||||
when a paid ePay order completes → a paid extract by tenant A is never "fresh" for tenant
|
||||
B, so the 30-day money-saver is structurally unrealized.
|
||||
|
||||
- On ePay completion, `upsert CfExtractCatalog(nrCadastral, latestId,
|
||||
expiresAt=documentDate+30d, isFresh=true)`.
|
||||
- `POST /enrichment/cf/claim {nrCadastral}`: on a catalog hit, create a B-owned row
|
||||
`type='catalog', status='completed', creditsUsed=0` pointing at the shared MinIO object
|
||||
(or a copy). This turns today's 409 `catalog_hit` (`enrichment.ts:226`) into **instant,
|
||||
free fulfillment**. RLS unchanged (B reads B's row). One paid extract serves every tenant
|
||||
that needs that parcel within 30 days, at marginal zero ANCPI cost.
|
||||
|
||||
## Phase D — Credential model (tenant-policy-driven)
|
||||
|
||||
Store the strategy per-tenant; don't pick one globally:
|
||||
|
||||
- **Internal Beletage group** → pooled company accounts (Infisical, encrypted in
|
||||
`epay_accounts`). Best batching + catalog sharing; per-credit attribution via audit.
|
||||
- **External paying tenants** (eterra-live model) → dedicated per-tenant accounts so
|
||||
credits/billing stay clean.
|
||||
- Record `account_id` + `creditsUsed` on every `CfExtract` for attribution regardless.
|
||||
- All three apps converge as thin callers of `POST /enrichment/cf` (Authentik multi-issuer
|
||||
+ tenant claim already in place, `gis-api/src/lib/auth.ts`). Reuse eterra-live
|
||||
`crypto.ts` (AES-256-GCM) + a 1-byte key-version prefix for rotation.
|
||||
|
||||
## Phase E — gis-api gaps for async consumption
|
||||
|
||||
1. **Completion webhook/SSE**, tenant-scoped + RLS-filtered (`GET /enrichment/cf/events`)
|
||||
→ kills polling and the dead-Brevo dependency.
|
||||
2. **Bulk-zip** `GET /enrichment/cf/zip?orderId=` streaming from MinIO (port the V3
|
||||
streaming-zip approach).
|
||||
3. `ExtractStatus` enum additions (see Phase A).
|
||||
4. List filters `creditsUsed=0` / `type='catalog'` so the UI can label shared extracts.
|
||||
|
||||
## Phase F — Reversible migration, per-tenant flip
|
||||
|
||||
- **Phase 0 (now):** `EPAY_ORDERING_VIA_GIS_AC=false`, hardened legacy queue is the sole
|
||||
fulfiller. `/api/ancpi/recover` stays as the manual safety net.
|
||||
- **Phase 1:** deploy worker + pool + catalog-write; seed `epay_accounts` with ONLY the
|
||||
Beletage account; flip the flag for `claims.tenant === 'architools'`.
|
||||
- **Phase 2:** run both paths in parallel a grace window; reconcile on orderId (no double
|
||||
charge).
|
||||
- **Phase 3:** onboard external tenants with dedicated accounts; delete `epay-queue` /
|
||||
`epay-client` / `epay-session-store` + `src/app/api/ancpi/*` from ArchiTools. The flag is
|
||||
the kill-switch throughout.
|
||||
|
||||
## Phase G — Shared `epay-client` package (do regardless of phase)
|
||||
|
||||
ArchiTools and eterra-live each have a near-identical `epay-client.ts` that has **already
|
||||
diverged dangerously**: ArchiTools got today's fixes; eterra-live got the method-internal
|
||||
ports (commit `eterra-live d30128b`) but lacks cart-hygiene + the per-page parser refactor.
|
||||
Extract `@beletage/epay-client` (natural home: `gis-sync-orchestrator`, which owns the
|
||||
account pool) so a fix lands once. Until then, any epay-client change MUST be mirrored to
|
||||
both repos in the same change.
|
||||
|
||||
## Known follow-ups not yet done
|
||||
|
||||
- eterra-live still lacks the cart-hygiene `numberOfItems` invariant (single-order flow
|
||||
makes it lower-risk, but a crashed prior order can still orphan a cart row). Needs a
|
||||
route-level touch + testing on that product before shipping.
|
||||
- `BREVO_API_KEY` returns 401 "Key not found" → ArchiTools email notifications are dead;
|
||||
the correct fix is the Phase E webhook, not patching Brevo. SMTP relay creds still work.
|
||||
- ArchiTools `auth-options.ts` has pre-existing `react-hooks/rules-of-hooks` lint errors on
|
||||
the `useGisAcFlag`/`useBasicPanelFlag` session calls (tolerated by `next build`).
|
||||
@@ -0,0 +1,64 @@
|
||||
// GET /api/ancpi/cleanup?dryRun=1 — preview what would be deleted
|
||||
// POST /api/ancpi/cleanup — run the cleanup now
|
||||
//
|
||||
// On-demand control over the 45-day ePay extract auto-cleanup (the scheduler
|
||||
// in epay-cleanup.ts runs it automatically on boot + every 24h). Useful to
|
||||
// preview (dryRun) before trusting the automatic run, or to trigger it now.
|
||||
//
|
||||
// Auth: a staff session (requireCfAccess), OR a cron Bearer token
|
||||
// (EPAY_CLEANUP_CRON_SECRET / NOTIFICATION_CRON_SECRET) so an external
|
||||
// scheduler can call it. ?days overrides the retention window.
|
||||
|
||||
import { NextResponse } from "next/server";
|
||||
import {
|
||||
cleanupExpiredEpayExtracts,
|
||||
EPAY_RETENTION_DAYS,
|
||||
} from "@/modules/parcel-sync/services/epay-cleanup";
|
||||
import { requireCfAccess } from "@/core/auth/cf-access";
|
||||
|
||||
export const runtime = "nodejs";
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
function cronAuthorized(req: Request): boolean {
|
||||
const secret =
|
||||
process.env.EPAY_CLEANUP_CRON_SECRET ?? process.env.NOTIFICATION_CRON_SECRET;
|
||||
if (!secret) return false;
|
||||
const auth = req.headers.get("authorization") ?? "";
|
||||
return auth === `Bearer ${secret}`;
|
||||
}
|
||||
|
||||
async function authorize(req: Request): Promise<boolean> {
|
||||
if (cronAuthorized(req)) return true;
|
||||
const access = await requireCfAccess();
|
||||
return access.ok;
|
||||
}
|
||||
|
||||
function parseDays(req: Request): number {
|
||||
const raw = new URL(req.url).searchParams.get("days");
|
||||
const n = raw ? parseInt(raw, 10) : NaN;
|
||||
return Number.isFinite(n) && n > 0 ? n : EPAY_RETENTION_DAYS;
|
||||
}
|
||||
|
||||
export async function GET(req: Request) {
|
||||
if (!(await authorize(req))) {
|
||||
return NextResponse.json({ error: "Neautorizat." }, { status: 401 });
|
||||
}
|
||||
// GET is always a dry-run (no side effects) — safe to preview from a browser.
|
||||
const result = await cleanupExpiredEpayExtracts({
|
||||
olderThanDays: parseDays(req),
|
||||
dryRun: true,
|
||||
});
|
||||
return NextResponse.json(result);
|
||||
}
|
||||
|
||||
export async function POST(req: Request) {
|
||||
if (!(await authorize(req))) {
|
||||
return NextResponse.json({ error: "Neautorizat." }, { status: 401 });
|
||||
}
|
||||
const dryRun = new URL(req.url).searchParams.get("dryRun") === "1";
|
||||
const result = await cleanupExpiredEpayExtracts({
|
||||
olderThanDays: parseDays(req),
|
||||
dryRun,
|
||||
});
|
||||
return NextResponse.json(result);
|
||||
}
|
||||
@@ -59,6 +59,7 @@ export async function GET(req: Request) {
|
||||
minioPath: true,
|
||||
documentDate: true,
|
||||
completedAt: true,
|
||||
status: true,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -71,7 +72,9 @@ export async function GET(req: Request) {
|
||||
for (let i = 0; i < ids.length; i++) {
|
||||
const id = ids[i]!;
|
||||
const extract = extractMap.get(id);
|
||||
if (!extract?.minioPath) continue;
|
||||
// Skip rows without a file, and "review" rows (PDF present but the
|
||||
// CF↔doc match is unverified — must not land in a "valid extracts" zip).
|
||||
if (!extract?.minioPath || extract.status === "review") continue;
|
||||
|
||||
const dateForName = extract.documentDate ?? extract.completedAt ?? new Date();
|
||||
const d = new Date(dateForName);
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { NextResponse } from "next/server";
|
||||
import { prisma } from "@/core/storage/prisma";
|
||||
import { requireCfAccess } from "@/core/auth/cf-access";
|
||||
import { enrichCfLocations } from "@/modules/parcel-sync/services/cf-enrich-location";
|
||||
|
||||
export const runtime = "nodejs";
|
||||
export const dynamic = "force-dynamic";
|
||||
@@ -57,6 +58,10 @@ export async function GET(req: Request) {
|
||||
prisma.cfExtract.count({ where }),
|
||||
]);
|
||||
|
||||
// Fill missing uatName/judetName from SIRUTA (old intern rows stored an
|
||||
// empty judetName) so the list shows localitate + judet for them too.
|
||||
await enrichCfLocations(orders);
|
||||
|
||||
// Build statusMap for multi-cadastral queries (or single if requested)
|
||||
if (cadastralNumbers.length > 0) {
|
||||
const now = new Date();
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
|
||||
import { NextResponse } from "next/server";
|
||||
import { prisma } from "@/core/storage/prisma";
|
||||
import { recoverBatch } from "@/modules/parcel-sync/services/epay-queue";
|
||||
import { recoverBatch, getQueueStatus } from "@/modules/parcel-sync/services/epay-queue";
|
||||
import { requireCfAccess } from "@/core/auth/cf-access";
|
||||
|
||||
export const runtime = "nodejs";
|
||||
@@ -85,6 +85,22 @@ export async function GET(req: Request) {
|
||||
);
|
||||
}
|
||||
|
||||
// Cross-guard against the live batch queue: while a batch is processing,
|
||||
// its rows sit at (orderId:null, status in cart/ordering/...) — exactly the
|
||||
// orphan window the WHERE below matches. Recovering then would re-stamp a
|
||||
// live batch's rows with THIS order's id (wrong PDF, status corruption).
|
||||
// A genuinely-crashed batch leaves __epayQueueProcessing=false (reset on
|
||||
// restart), so blocking here only defers against an actively-running queue.
|
||||
if (getQueueStatus().processing) {
|
||||
return NextResponse.json(
|
||||
{
|
||||
error:
|
||||
"O comandă ePay este în curs de procesare. Așteaptă finalizarea înainte de recuperare.",
|
||||
},
|
||||
{ status: 409 },
|
||||
);
|
||||
}
|
||||
|
||||
// Candidate rows: anything already tagged with this order that isn't
|
||||
// terminal, PLUS recent orphaned rows (orderId:null) in a recoverable
|
||||
// state — the operator asserts they belong to this order.
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
import { NextResponse } from "next/server";
|
||||
import { getAuthSession } from "@/core/auth/require-auth";
|
||||
import { gisApi, GisApiError } from "@/lib/gis-api-client";
|
||||
import { enrichCfLocations } from "@/modules/parcel-sync/services/cf-enrich-location";
|
||||
|
||||
export const runtime = "nodejs";
|
||||
export const dynamic = "force-dynamic";
|
||||
@@ -30,9 +31,14 @@ export async function GET(request: Request) {
|
||||
const offset = Number.isFinite(offsetRaw) && offsetRaw >= 0 ? offsetRaw : 0;
|
||||
|
||||
try {
|
||||
return NextResponse.json(
|
||||
await gisApi.enrichment.cf.list({ limit, offset, status }),
|
||||
);
|
||||
const data = await gisApi.enrichment.cf.list({ limit, offset, status });
|
||||
// gis-api returns uatName + siruta but judetName is null there — fill the
|
||||
// county (and any missing UAT name) from the local GisUat table so the UI
|
||||
// can show "localitate, jud. X" on intern rows too.
|
||||
if (Array.isArray(data?.rows)) {
|
||||
await enrichCfLocations(data.rows);
|
||||
}
|
||||
return NextResponse.json(data);
|
||||
} catch (err) {
|
||||
if (err instanceof GisApiError) {
|
||||
return NextResponse.json(
|
||||
|
||||
@@ -8,5 +8,9 @@ export async function register() {
|
||||
// ParcelSync auto-refresh scheduler DISABLED during GIS DB overhaul.
|
||||
// Re-enable by uncommenting the import below once the new schema is stable.
|
||||
// await import("@/modules/parcel-sync/services/auto-refresh-scheduler");
|
||||
|
||||
// ePay CF extract auto-cleanup (deletes rows + MinIO objects 45 days
|
||||
// after issuance). Self-contained scheduler; safe to run every deploy.
|
||||
await import("@/modules/parcel-sync/services/epay-cleanup");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -55,6 +55,10 @@ export interface CfExtractRow {
|
||||
userId: string;
|
||||
nrCadastral: string;
|
||||
nrCF?: string;
|
||||
type?: string;
|
||||
siruta?: string | null;
|
||||
uatName?: string | null;
|
||||
judetName?: string | null;
|
||||
status:
|
||||
| "pending"
|
||||
| "queued"
|
||||
|
||||
+1
-1
@@ -58,6 +58,6 @@ export const config = {
|
||||
* - /favicon.ico, /robots.txt, /sitemap.xml
|
||||
* - Files with extensions (images, fonts, etc.)
|
||||
*/
|
||||
"/((?!api/auth|api/version|api/basemap-|api/notifications/digest|api/eterra/auto-refresh|api/compress-pdf|api/address-book|api/projects|auth/signin|_next|favicon\\.ico|robots\\.txt|sitemap\\.xml|.*\\..*).*)",
|
||||
"/((?!api/auth|api/version|api/basemap-|api/notifications/digest|api/eterra/auto-refresh|api/ancpi/cleanup|api/compress-pdf|api/address-book|api/projects|auth/signin|_next|favicon\\.ico|robots\\.txt|sitemap\\.xml|.*\\..*).*)",
|
||||
],
|
||||
};
|
||||
|
||||
@@ -254,6 +254,23 @@ export function CfOrderModal({
|
||||
const row = (data.orders ?? []).find((o) => o.id === id);
|
||||
if (row) {
|
||||
const s = (row.status || "").toLowerCase();
|
||||
// "review" rows DO have minioPath + documentName but the PDF
|
||||
// may belong to another parcel (ambiguous CF↔doc match) — must
|
||||
// be checked BEFORE the minioPath/documentName completion
|
||||
// fallback, or it would short-circuit to "done" and hand the
|
||||
// operator an unverified extract.
|
||||
if (s === "review") {
|
||||
setError(
|
||||
"Comanda necesită verificare manuală (potrivire ambiguă document↔parcelă). Verifică în ParcelSync → ePay înainte de a folosi extrasul.",
|
||||
);
|
||||
setPhase("error");
|
||||
return;
|
||||
}
|
||||
if (s === "failed" || s === "error") {
|
||||
setError("Comanda a eșuat la ANCPI.");
|
||||
setPhase("error");
|
||||
return;
|
||||
}
|
||||
if (
|
||||
s === "completed" ||
|
||||
s === "done" ||
|
||||
@@ -264,11 +281,6 @@ export function CfOrderModal({
|
||||
setPhase("done");
|
||||
return;
|
||||
}
|
||||
if (s === "failed" || s === "error") {
|
||||
setError("Comanda a eșuat la ANCPI.");
|
||||
setPhase("error");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
|
||||
@@ -99,10 +99,10 @@ function adaptLegacyRow(row: LegacyCfExtract): CfExtractRecord {
|
||||
}
|
||||
|
||||
// Convert a gisApi CfExtractRow → the UI-side CfExtractRecord shape.
|
||||
// gis-api currently doesn't surface uatName/siruta/judetName on the list
|
||||
// endpoint, so we leave them blank; the row type defaults to "intern"
|
||||
// because gis_core's CfExtract is the cf-intern store (the cutover plan
|
||||
// hasn't yet moved ePay writes here).
|
||||
// gis-api returns siruta + uatName (judetName is null there, but the
|
||||
// /api/cf/orders proxy fills it from the local GisUat by SIRUTA — see
|
||||
// enrichCfLocations). The row type defaults to "intern" because gis's
|
||||
// CfExtract is primarily the cf-intern store.
|
||||
export function adaptCfRow(row: CfExtractRow & { type?: string }): CfExtractRecord {
|
||||
return {
|
||||
id: row.id,
|
||||
@@ -110,9 +110,9 @@ export function adaptCfRow(row: CfExtractRow & { type?: string }): CfExtractReco
|
||||
orderId: row.orderId ?? null,
|
||||
nrCadastral: row.nrCadastral,
|
||||
nrCF: row.nrCF ?? null,
|
||||
siruta: null,
|
||||
judetName: "",
|
||||
uatName: "",
|
||||
siruta: row.siruta ?? null,
|
||||
judetName: row.judetName ?? "",
|
||||
uatName: row.uatName ?? "",
|
||||
status: row.status,
|
||||
epayStatus: row.epayStatus ?? null,
|
||||
documentName: row.documentName ?? null,
|
||||
@@ -167,12 +167,18 @@ async function fetchGisAc(
|
||||
// single timeline shows ePay + intern history together. Sort newest-
|
||||
// first; dedupe by id (in case the same record ever lands in both
|
||||
// stores during the cutover migration).
|
||||
// Cancelled rows are dead (payment refused / cleaned-up bad orders) and just
|
||||
// clutter the list — hide them. failed/review stay (they're actionable).
|
||||
const isListable = (r: CfExtractRecord): boolean => r.status !== "cancelled";
|
||||
|
||||
export async function fetchCfOrdersList(
|
||||
useGisAc: boolean,
|
||||
params: { limit?: number; nrCadastral?: string; status?: string } = {},
|
||||
): Promise<{ orders: CfExtractRecord[]; total: number }> {
|
||||
if (!useGisAc) {
|
||||
return fetchLegacy(params);
|
||||
const r = await fetchLegacy(params);
|
||||
const orders = r.orders.filter(isListable);
|
||||
return { orders, total: orders.length };
|
||||
}
|
||||
|
||||
// Pull more rows from each side than the caller asked for so that the
|
||||
@@ -191,6 +197,7 @@ export async function fetchCfOrdersList(
|
||||
|
||||
const seen = new Set<string>();
|
||||
const dedup = merged.filter((r) => {
|
||||
if (!isListable(r)) return false;
|
||||
if (seen.has(r.id)) return false;
|
||||
seen.add(r.id);
|
||||
return true;
|
||||
@@ -198,14 +205,8 @@ export async function fetchCfOrdersList(
|
||||
|
||||
dedup.sort((a, b) => (a.createdAt < b.createdAt ? 1 : -1));
|
||||
|
||||
const total =
|
||||
(legacy.status === "fulfilled" ? legacy.value.total : 0) +
|
||||
(gisac.status === "fulfilled" ? gisac.value.total : 0);
|
||||
|
||||
return {
|
||||
orders: params.limit ? dedup.slice(0, params.limit) : dedup,
|
||||
total,
|
||||
};
|
||||
const orders = params.limit ? dedup.slice(0, params.limit) : dedup;
|
||||
return { orders, total: orders.length };
|
||||
}
|
||||
|
||||
// Existence check used by the per-parcel order button. We check both
|
||||
|
||||
@@ -141,17 +141,21 @@ export function EpayConnect({
|
||||
if (cancelled) return;
|
||||
setError("Eroare retea");
|
||||
shouldRetry = attempt < maxRetries;
|
||||
} finally {
|
||||
// ALWAYS clear the connecting spinner unless we're about to retry —
|
||||
// including the `cancelled` early-returns above. Otherwise a re-run
|
||||
// of this effect (e.g. when status.connected flips true) cancels the
|
||||
// in-flight attempt and leaves connecting stuck true → a perpetual
|
||||
// spinner on an already-connected (green) pill.
|
||||
if (!shouldRetry) setConnecting(false);
|
||||
}
|
||||
|
||||
if (cancelled) return;
|
||||
|
||||
if (shouldRetry) {
|
||||
// Keep connecting state true during retry wait
|
||||
autoConnectTimerRef.current = setTimeout(() => {
|
||||
void attemptConnect(attempt + 1);
|
||||
}, 3000);
|
||||
} else {
|
||||
setConnecting(false);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -164,7 +168,11 @@ export function EpayConnect({
|
||||
autoConnectTimerRef.current = null;
|
||||
}
|
||||
};
|
||||
}, [triggerConnect, status.connected, connecting, fetchStatus]);
|
||||
// `connecting` intentionally excluded: setConnecting(true) inside this
|
||||
// effect would otherwise re-trigger it and cancel its own in-flight
|
||||
// attempt. autoConnectAttempted (a ref) already prevents double-starts.
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [triggerConnect, status.connected, fetchStatus]);
|
||||
|
||||
const disconnect = async () => {
|
||||
try {
|
||||
@@ -202,10 +210,10 @@ export function EpayConnect({
|
||||
: "border-muted-foreground/20 bg-muted/50 text-muted-foreground",
|
||||
)}
|
||||
>
|
||||
{connecting ? (
|
||||
<Loader2 className="h-3 w-3 animate-spin" />
|
||||
) : status.connected ? (
|
||||
{status.connected ? (
|
||||
<span className="inline-flex h-2 w-2 rounded-full bg-emerald-500" />
|
||||
) : connecting ? (
|
||||
<Loader2 className="h-3 w-3 animate-spin" />
|
||||
) : null}
|
||||
|
||||
<span className="hidden sm:inline">ePay</span>
|
||||
|
||||
@@ -57,6 +57,10 @@ export function EpayOrderButton({
|
||||
|
||||
const [ordering, setOrdering] = useState(false);
|
||||
const [ordered, setOrdered] = useState(false);
|
||||
// After enqueue the order keeps processing in the background (~1–2 min on
|
||||
// the legacy queue): cart → submit → poll → download. Show that instead of
|
||||
// flipping straight to a misleading "valid" the instant it's queued.
|
||||
const [processing, setProcessing] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
const [epayStatus, setEpayStatus] = useState<EpaySessionStatus>({
|
||||
connected: false,
|
||||
@@ -111,7 +115,10 @@ export function EpayOrderButton({
|
||||
});
|
||||
if (mountedRef.current) {
|
||||
if (result.ok) {
|
||||
setOrdered(true);
|
||||
// Queued, not done — enter the processing state and let the poll
|
||||
// effect below flip to "valid" only once the extract is actually
|
||||
// ready (or surface a failure).
|
||||
setProcessing(true);
|
||||
} else {
|
||||
setError(result.error ?? "Eroare comanda");
|
||||
}
|
||||
@@ -119,6 +126,36 @@ export function EpayOrderButton({
|
||||
}
|
||||
}, [nrCadastral, siruta, judetName, uatName, useGisAc]);
|
||||
|
||||
// Poll while processing: flip to "valid" only when a completed extract
|
||||
// actually exists. Caps at ~3 min, then stops (the parent list refresh
|
||||
// will reflect the final state).
|
||||
useEffect(() => {
|
||||
if (!processing) return;
|
||||
let cancelled = false;
|
||||
let attempts = 0;
|
||||
const tick = async () => {
|
||||
attempts += 1;
|
||||
try {
|
||||
const has = await fetchCfHasCompletedForCadastral(useGisAc, nrCadastral);
|
||||
if (cancelled) return;
|
||||
if (has) {
|
||||
setOrdered(true);
|
||||
setProcessing(false);
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
/* keep polling */
|
||||
}
|
||||
if (!cancelled && attempts >= 36) setProcessing(false); // ~3 min
|
||||
};
|
||||
const id = setInterval(() => void tick(), 5000);
|
||||
void tick();
|
||||
return () => {
|
||||
cancelled = true;
|
||||
clearInterval(id);
|
||||
};
|
||||
}, [processing, useGisAc, nrCadastral]);
|
||||
|
||||
// On the (future) gis.ac path, the orchestrator dispatches ePay calls
|
||||
// through a shared account pool — no personally-connected ePay session
|
||||
// needed. The legacy queue (current route while the guard is on)
|
||||
@@ -143,6 +180,29 @@ export function EpayOrderButton({
|
||||
return tooltipText ?? "Comanda extras CF (1 credit)";
|
||||
};
|
||||
|
||||
if (processing) {
|
||||
return (
|
||||
<TooltipProvider>
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
size="sm"
|
||||
variant="ghost"
|
||||
className="h-7 gap-1 px-1.5 text-yellow-600 dark:text-yellow-400"
|
||||
disabled
|
||||
>
|
||||
<Loader2 className="h-3.5 w-3.5 animate-spin" />
|
||||
<span className="text-[10px]">Se procesează...</span>
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent>
|
||||
Comanda CF este în curs (coș → plată → descărcare, ~1–2 min)
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
);
|
||||
}
|
||||
|
||||
if (ordered) {
|
||||
return (
|
||||
<TooltipProvider>
|
||||
|
||||
@@ -74,7 +74,23 @@ function isActiveStatus(status: string): boolean {
|
||||
|
||||
type StatusStyle = { label: string; className: string; pulse?: boolean };
|
||||
|
||||
function statusBadge(status: string, expiresAt: string | null): StatusStyle {
|
||||
function statusBadge(
|
||||
status: string,
|
||||
expiresAt: string | null,
|
||||
type?: "epay" | "intern",
|
||||
): StatusStyle {
|
||||
// Intern (copycf) extracts have NO validity term — never label them "Valid"
|
||||
// or "Expirat" (which imply an ePay-style 30-day validity). The source
|
||||
// badge next to this already says "intern", so the STATUS just states the
|
||||
// document is available (no term).
|
||||
if (type === "intern" && status === "completed") {
|
||||
return {
|
||||
label: "Disponibil",
|
||||
className:
|
||||
"bg-muted text-foreground/70 border-muted-foreground/20",
|
||||
};
|
||||
}
|
||||
|
||||
if (status === "completed" && isExpired(expiresAt)) {
|
||||
return {
|
||||
label: "Expirat",
|
||||
@@ -328,17 +344,35 @@ export function EpayTab() {
|
||||
* order, no new charge. For rows that failed at the download/poll
|
||||
* stage (the order exists at ANCPI but we never stored the PDF). -- */
|
||||
const [retryingId, setRetryingId] = useState<string | null>(null);
|
||||
const [retryNotice, setRetryNotice] = useState<string | null>(null);
|
||||
const handleRetryDownload = async (order: CfExtractRecord) => {
|
||||
setRetryingId(order.id);
|
||||
setRetryNotice(null);
|
||||
try {
|
||||
const res = await fetch(
|
||||
`/api/ancpi/recover?extractId=${encodeURIComponent(order.id)}`,
|
||||
);
|
||||
// 409 → the row has no orderId yet (never reached ANCPI); nothing to
|
||||
// recover by row. Other errors surface on the next refresh.
|
||||
await res.json().catch(() => ({}));
|
||||
const data = (await res.json().catch(() => ({}))) as {
|
||||
error?: string;
|
||||
completed?: number;
|
||||
attempted?: number;
|
||||
};
|
||||
if (!res.ok) {
|
||||
// 409 (queue busy / no orderId yet), 404, 500 — tell the user.
|
||||
setRetryNotice(data.error ?? `Reîncercare eșuată (${res.status}).`);
|
||||
} else if ((data.completed ?? 0) > 0) {
|
||||
setRetryNotice(
|
||||
`Recuperat: ${data.completed}/${data.attempted ?? data.completed} extrase.`,
|
||||
);
|
||||
} else {
|
||||
setRetryNotice(
|
||||
"Nimic de recuperat — comanda nu există la ANCPI sau e deja finalizată.",
|
||||
);
|
||||
}
|
||||
void fetchOrders(true);
|
||||
void fetchEpayStatus();
|
||||
} catch {
|
||||
setRetryNotice("Eroare rețea la reîncercare.");
|
||||
} finally {
|
||||
setRetryingId(null);
|
||||
}
|
||||
@@ -623,7 +657,7 @@ export function EpayTab() {
|
||||
</thead>
|
||||
<tbody>
|
||||
{filteredOrders.map((order, idx) => {
|
||||
const badge = statusBadge(order.status, order.expiresAt);
|
||||
const badge = statusBadge(order.status, order.expiresAt, order.type);
|
||||
const expired =
|
||||
order.status === "completed" && isExpired(order.expiresAt);
|
||||
|
||||
@@ -911,6 +945,20 @@ export function EpayTab() {
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* -- Retry notice ------------------------------------------- */}
|
||||
{retryNotice && (
|
||||
<div className="flex items-center justify-between gap-2 rounded-md border bg-muted/40 px-3 py-2 text-xs">
|
||||
<span>{retryNotice}</span>
|
||||
<button
|
||||
type="button"
|
||||
className="text-muted-foreground hover:text-foreground"
|
||||
onClick={() => setRetryNotice(null)}
|
||||
>
|
||||
✕
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* -- Active orders indicator -------------------------------- */}
|
||||
{hasActive && (
|
||||
<div className="flex items-center gap-2 text-xs text-muted-foreground">
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
// Fill in uatName + judetName on CF extract rows from their SIRUTA.
|
||||
//
|
||||
// Intern (cf-intern) extracts — and ePay rows on the gis-api side — often
|
||||
// arrive without a judetName (it's null in gis_enrichment) and sometimes
|
||||
// without a uatName. Both are derivable from `siruta` via the local GisUat
|
||||
// table. This batches one query for the whole page instead of N lookups.
|
||||
|
||||
import { prisma } from "@/core/storage/prisma";
|
||||
|
||||
type LocatableRow = {
|
||||
siruta?: string | null;
|
||||
uatName?: string | null;
|
||||
judetName?: string | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Mutates rows in place: for any row with a SIRUTA whose uatName/judetName is
|
||||
* blank, fill it from GisUat. Best-effort — a missing SIRUTA or a DB error
|
||||
* leaves the row unchanged. Returns the same array for convenience.
|
||||
*/
|
||||
export async function enrichCfLocations<T extends LocatableRow>(
|
||||
rows: T[],
|
||||
): Promise<T[]> {
|
||||
const sirutas = Array.from(
|
||||
new Set(
|
||||
rows
|
||||
.map((r) => (r.siruta ? String(r.siruta).trim() : ""))
|
||||
.filter(Boolean),
|
||||
),
|
||||
);
|
||||
if (sirutas.length === 0) return rows;
|
||||
|
||||
try {
|
||||
const uats = await prisma.gisUat.findMany({
|
||||
where: { siruta: { in: sirutas } },
|
||||
select: { siruta: true, name: true, county: true },
|
||||
});
|
||||
const bySiruta = new Map(uats.map((u) => [u.siruta, u]));
|
||||
|
||||
for (const row of rows) {
|
||||
const s = row.siruta ? String(row.siruta).trim() : "";
|
||||
if (!s) continue;
|
||||
const uat = bySiruta.get(s);
|
||||
if (!uat) continue;
|
||||
if (!row.uatName && uat.name) row.uatName = uat.name;
|
||||
if (!row.judetName && uat.county) row.judetName = uat.county;
|
||||
}
|
||||
} catch (error) {
|
||||
console.warn("[cf-enrich-location] lookup failed:", error);
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
/**
|
||||
* Auto-cleanup of old ePay CF extracts.
|
||||
*
|
||||
* An ePay extract is valid 30 days after issuance (documentDate); after that
|
||||
* it's worthless (you'd re-order). At 45 days we delete the row + its MinIO
|
||||
* object to declutter the list and free storage. Only `type='epay'` rows are
|
||||
* touched — the free `cf-intern` (copycf) extracts are kept.
|
||||
*
|
||||
* Self-contained scheduler (same pattern as auto-refresh-scheduler): started
|
||||
* by importing this module from instrumentation.ts. Runs once shortly after
|
||||
* boot (so it happens at least once per deploy, since redeploys reset the
|
||||
* interval) and then every 24h. The cleanup is idempotent — a partial run
|
||||
* (e.g. interrupted by a restart) is simply finished by the next run.
|
||||
*/
|
||||
|
||||
import { PrismaClient } from "@prisma/client";
|
||||
import { deleteCfExtractObjects } from "./epay-storage";
|
||||
|
||||
const prisma = new PrismaClient();
|
||||
|
||||
/** Delete ePay extracts this many days after issuance. */
|
||||
export const EPAY_RETENTION_DAYS = 45;
|
||||
|
||||
const g = globalThis as {
|
||||
__epayCleanupTimer?: ReturnType<typeof setInterval>;
|
||||
__epayCleanupRunning?: boolean;
|
||||
};
|
||||
|
||||
export type CleanupResult = {
|
||||
candidates: number;
|
||||
rowsDeleted: number;
|
||||
objectsDeleted: number;
|
||||
cutoff: string;
|
||||
dryRun: boolean;
|
||||
};
|
||||
|
||||
/**
|
||||
* Find and (unless dryRun) delete ePay extracts older than `olderThanDays`
|
||||
* from issuance. Issuance = documentDate, falling back to createdAt for rows
|
||||
* that never got a document (old failed/cancelled). Deletes the MinIO object
|
||||
* first, then the DB rows.
|
||||
*/
|
||||
export async function cleanupExpiredEpayExtracts(opts?: {
|
||||
olderThanDays?: number;
|
||||
dryRun?: boolean;
|
||||
}): Promise<CleanupResult> {
|
||||
const olderThanDays = opts?.olderThanDays ?? EPAY_RETENTION_DAYS;
|
||||
const dryRun = opts?.dryRun ?? false;
|
||||
const cutoff = new Date(Date.now() - olderThanDays * 24 * 60 * 60 * 1000);
|
||||
|
||||
// COALESCE(documentDate, createdAt) < cutoff — raw query so the date logic
|
||||
// runs in Postgres and uses the createdAt index where possible.
|
||||
const rows = (await prisma.$queryRaw`
|
||||
SELECT id, "minioPath"
|
||||
FROM "CfExtract"
|
||||
WHERE type = 'epay'
|
||||
AND COALESCE("documentDate", "createdAt") < ${cutoff}
|
||||
`) as Array<{ id: string; minioPath: string | null }>;
|
||||
|
||||
const result: CleanupResult = {
|
||||
candidates: rows.length,
|
||||
rowsDeleted: 0,
|
||||
objectsDeleted: 0,
|
||||
cutoff: cutoff.toISOString(),
|
||||
dryRun,
|
||||
};
|
||||
|
||||
if (rows.length === 0 || dryRun) {
|
||||
console.log(
|
||||
`[epay-cleanup] ${dryRun ? "(dry-run) " : ""}${rows.length} ePay extract(s) older than ${olderThanDays}d (cutoff ${cutoff.toISOString().slice(0, 10)})`,
|
||||
);
|
||||
return result;
|
||||
}
|
||||
|
||||
// Delete MinIO objects first (best-effort) so a deleted DB row never leaves
|
||||
// an orphan file.
|
||||
const paths = rows.map((r) => r.minioPath).filter((p): p is string => !!p);
|
||||
result.objectsDeleted = await deleteCfExtractObjects(paths);
|
||||
|
||||
const del = await prisma.cfExtract.deleteMany({
|
||||
where: { id: { in: rows.map((r) => r.id) } },
|
||||
});
|
||||
result.rowsDeleted = del.count;
|
||||
|
||||
console.log(
|
||||
`[epay-cleanup] deleted ${result.rowsDeleted} row(s) + ${result.objectsDeleted} object(s) older than ${olderThanDays}d`,
|
||||
);
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Run the cleanup once, guarded against overlap. Never throws. */
|
||||
async function runCleanupSafely(): Promise<void> {
|
||||
if (g.__epayCleanupRunning) return;
|
||||
g.__epayCleanupRunning = true;
|
||||
try {
|
||||
await cleanupExpiredEpayExtracts();
|
||||
} catch (error) {
|
||||
console.error("[epay-cleanup] run failed:", error);
|
||||
} finally {
|
||||
g.__epayCleanupRunning = false;
|
||||
}
|
||||
}
|
||||
|
||||
const ONE_DAY_MS = 24 * 60 * 60 * 1000;
|
||||
const BOOT_DELAY_MS = 90_000; // let the app finish starting first
|
||||
|
||||
// Start the scheduler (idempotent — one per process).
|
||||
if (!g.__epayCleanupTimer) {
|
||||
setTimeout(() => void runCleanupSafely(), BOOT_DELAY_MS);
|
||||
g.__epayCleanupTimer = setInterval(() => void runCleanupSafely(), ONE_DAY_MS);
|
||||
console.log(
|
||||
`[epay-cleanup] scheduler armed (retention ${EPAY_RETENTION_DAYS}d, boot run in ${BOOT_DELAY_MS / 1000}s, then every 24h)`,
|
||||
);
|
||||
}
|
||||
@@ -51,6 +51,9 @@ const POLL_MAX_ATTEMPTS = 40;
|
||||
// ShowOrderDetails page size — large enough to fetch any realistic batch in
|
||||
// one request (see getOrderStatus / QW4).
|
||||
const ORDER_PAGE_SIZE = 50;
|
||||
// Document download retry (transient ANCPI 5xx / timeout / error-page).
|
||||
const DOWNLOAD_MAX_ATTEMPTS = 4;
|
||||
const DOWNLOAD_RETRY_DELAY_MS = 3_000; // linear backoff: 3s, 6s, 9s
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Session cache */
|
||||
@@ -233,9 +236,11 @@ export class EpayClient {
|
||||
|
||||
const data = response.data as EpayCartResponse;
|
||||
const items = Array.isArray(data?.items) ? data.items : [];
|
||||
// The freshly added row is the one we didn't know about; ePay returns
|
||||
// the full cart in `items`, newest typically last. Be defensive.
|
||||
const added = items[items.length - 1] ?? items[0];
|
||||
// ePay returns the full cart NEWEST-FIRST, so the just-added row is
|
||||
// items[0]. (Taking items[last] broke 2+ item batches: every add
|
||||
// reported the OLDEST row's id, so two rows collapsed onto one
|
||||
// basketRowId and metadata was saved to the wrong row — 2026-06-05.)
|
||||
const added = items[0];
|
||||
if (!added?.id) {
|
||||
throw new Error(`ePay addToCart failed: ${JSON.stringify(data).slice(0, 200)}`);
|
||||
}
|
||||
@@ -604,28 +609,34 @@ export class EpayClient {
|
||||
});
|
||||
const html = String(response.data ?? "");
|
||||
|
||||
// Find ALL orderIds on the page
|
||||
// ePay order numbers are sequential, so a genuinely NEW order is always
|
||||
// numerically GREATER than the latest order that existed before submit.
|
||||
// Requiring oid > previousOrderId is what stops us from adopting an
|
||||
// unrelated OLD order when our submit didn't actually create one — the
|
||||
// "!= previousOrderId" check alone let an older id through (2026-06-05:
|
||||
// a new batch grabbed yesterday's order 10009605 and attached its PDFs).
|
||||
const prevNum = previousOrderId ? Number(previousOrderId) : 0;
|
||||
const isGenuinelyNew = (oid: string): boolean =>
|
||||
!!oid &&
|
||||
oid !== previousOrderId &&
|
||||
!knownOrderIds?.has(oid) &&
|
||||
(!Number.isFinite(prevNum) || prevNum === 0 || Number(oid) > prevNum);
|
||||
|
||||
// Find ALL orderIds on the page; take the highest genuinely-new one.
|
||||
const allMatches = html.matchAll(/ShowOrderDetails\.action\?orderId=(\d+)/g);
|
||||
let best = "";
|
||||
for (const m of allMatches) {
|
||||
const oid = m[1] ?? "";
|
||||
if (!oid) continue;
|
||||
if (oid === previousOrderId) continue;
|
||||
if (knownOrderIds?.has(oid)) continue;
|
||||
console.log(`[epay] New orderId: ${oid}`);
|
||||
return oid;
|
||||
}
|
||||
|
||||
// If no new orderId found, the latest one might be it (first order) —
|
||||
// but NEVER adopt the previous/known order: after a submit that timed
|
||||
// out without creating anything, returning the stale id would attach
|
||||
// the wrong order and download its old documents.
|
||||
const latest = html.match(/ShowOrderDetails\.action\?orderId=(\d+)/);
|
||||
const latestId = latest?.[1];
|
||||
if (latestId && latestId !== previousOrderId && !knownOrderIds?.has(latestId)) {
|
||||
console.log(`[epay] Using latest orderId: ${latestId}`);
|
||||
return latestId;
|
||||
if (!isGenuinelyNew(oid)) continue;
|
||||
if (!best || Number(oid) > Number(best)) best = oid;
|
||||
}
|
||||
if (best) {
|
||||
console.log(`[epay] New orderId: ${best}`);
|
||||
return best;
|
||||
}
|
||||
|
||||
// No genuinely-new order on the dashboard → the submit created nothing.
|
||||
// Fail (recoverable) rather than adopting a stale/previous/known order.
|
||||
throw new Error("Could not determine orderId after checkout");
|
||||
}
|
||||
|
||||
@@ -771,10 +782,20 @@ export class EpayClient {
|
||||
onProgress?: (attempt: number, status: string) => void,
|
||||
): Promise<EpayOrderStatus> {
|
||||
for (let attempt = 1; attempt <= POLL_MAX_ATTEMPTS; attempt++) {
|
||||
const status = await this.getOrderStatus(orderId);
|
||||
if (onProgress) onProgress(attempt, status.status);
|
||||
if (["Finalizata", "Anulata", "Plata refuzata"].includes(status.status)) {
|
||||
return status;
|
||||
try {
|
||||
const status = await this.getOrderStatus(orderId);
|
||||
if (onProgress) onProgress(attempt, status.status);
|
||||
if (["Finalizata", "Anulata", "Plata refuzata"].includes(status.status)) {
|
||||
return status;
|
||||
}
|
||||
} catch (err) {
|
||||
// A transient ANCPI error (5xx, timeout) on ONE poll must not abort
|
||||
// the whole batch — the order is paid and still being processed.
|
||||
// Log and try again on the next cycle.
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn(
|
||||
`[epay] poll ${attempt}/${POLL_MAX_ATTEMPTS} for order ${orderId} errored (${msg}); continuing`,
|
||||
);
|
||||
}
|
||||
await sleep(POLL_INTERVAL_MS);
|
||||
}
|
||||
@@ -785,29 +806,57 @@ export class EpayClient {
|
||||
|
||||
async downloadDocument(idDocument: number, typeD = 4): Promise<Buffer> {
|
||||
const url = `${BASE_URL}/DownloadFile.action?typeD=${typeD}&id=${idDocument}&source=&browser=chrome`;
|
||||
// Angular sends Content-Type: application/pdf in the REQUEST
|
||||
const response = await this.client.post(url, null, {
|
||||
headers: { "Content-Type": "application/pdf" },
|
||||
timeout: DEFAULT_TIMEOUT_MS,
|
||||
responseType: "arraybuffer",
|
||||
});
|
||||
let lastErr = "unknown";
|
||||
|
||||
const data = response.data;
|
||||
if (!data || data.length < 100) {
|
||||
throw new Error(`ePay download empty (${data?.length ?? 0} bytes)`);
|
||||
// ANCPI's DownloadFile occasionally returns a transient 5xx / times out /
|
||||
// hands back an error page even when the order is finalized (2026-06-05:
|
||||
// 327649 got one 500, then succeeded on the very next attempt). The
|
||||
// download is idempotent, so retry transient failures with backoff before
|
||||
// giving up. A 4xx is treated as permanent (stop immediately).
|
||||
for (let attempt = 1; attempt <= DOWNLOAD_MAX_ATTEMPTS; attempt++) {
|
||||
try {
|
||||
const response = await this.client.post(url, null, {
|
||||
headers: { "Content-Type": "application/pdf" },
|
||||
timeout: DEFAULT_TIMEOUT_MS,
|
||||
responseType: "arraybuffer",
|
||||
validateStatus: () => true, // inspect status ourselves for retry
|
||||
});
|
||||
|
||||
if (response.status >= 400) {
|
||||
lastErr = `HTTP ${response.status}`;
|
||||
if (response.status < 500) break; // client error — won't fix on retry
|
||||
} else {
|
||||
const buf = Buffer.from(response.data ?? Buffer.alloc(0));
|
||||
if (buf.length < 100) {
|
||||
lastErr = `empty (${buf.length} bytes)`;
|
||||
} else if (buf.subarray(0, 5).toString("latin1") !== "%PDF-") {
|
||||
// Not a PDF — usually a transient ANCPI error page or an expired
|
||||
// session. Retry; a fresh attempt often returns the real PDF.
|
||||
const head = buf.subarray(0, 48).toString("latin1").replace(/\s+/g, " ");
|
||||
lastErr = `not a PDF (head="${head.slice(0, 40)}")`;
|
||||
} else {
|
||||
if (attempt > 1) {
|
||||
console.log(`[epay] download ${idDocument} recovered on attempt ${attempt}`);
|
||||
}
|
||||
console.log(`[epay] Downloaded document ${idDocument}: ${buf.length} bytes`);
|
||||
return buf;
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
// Network error / timeout — retryable.
|
||||
lastErr = err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
|
||||
if (attempt < DOWNLOAD_MAX_ATTEMPTS) {
|
||||
console.warn(
|
||||
`[epay] download ${idDocument} attempt ${attempt} failed (${lastErr}); retrying in ${DOWNLOAD_RETRY_DELAY_MS * attempt}ms`,
|
||||
);
|
||||
await sleep(DOWNLOAD_RETRY_DELAY_MS * attempt);
|
||||
}
|
||||
}
|
||||
const buf = Buffer.from(data);
|
||||
// R2: if the ePay session expired mid-batch, DownloadFile returns the
|
||||
// login/error HTML page (200 OK) instead of the PDF. Storing that as a
|
||||
// ".pdf" silently corrupts the extract. Assert the PDF magic bytes.
|
||||
if (buf.subarray(0, 5).toString("latin1") !== "%PDF-") {
|
||||
const head = buf.subarray(0, 64).toString("latin1");
|
||||
throw new Error(
|
||||
`ePay download not a PDF (idDocument=${idDocument}, ${buf.length} bytes, head="${head.replace(/\s+/g, " ").slice(0, 40)}") — session may have expired`,
|
||||
);
|
||||
}
|
||||
console.log(`[epay] Downloaded document ${idDocument}: ${buf.length} bytes`);
|
||||
return buf;
|
||||
throw new Error(
|
||||
`ePay download failed after ${DOWNLOAD_MAX_ATTEMPTS} attempts (idDocument=${idDocument}): ${lastErr}`,
|
||||
);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
import { prisma } from "@/core/storage/prisma";
|
||||
import { EpayClient } from "./epay-client";
|
||||
import { getEpayCredentials, updateEpayCredits } from "./epay-session-store";
|
||||
import { storeCfExtract } from "./epay-storage";
|
||||
import { storeCfExtract, getNextFileIndex } from "./epay-storage";
|
||||
import type { CfExtractCreateInput } from "./epay-types";
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
@@ -308,7 +308,12 @@ async function processBatch(
|
||||
// leftover row from a previously-crashed batch would be paid for and
|
||||
// attach the wrong PDF. We track our own basket ids for cleanup, and
|
||||
// bail the moment ANCPI reports more rows than we put in.
|
||||
let addedCount = 0;
|
||||
// cartCount tracks the rows actually in the cart (incremented on add,
|
||||
// decremented only on a CONFIRMED delete) so the invariant stays correct
|
||||
// even if a cleanup delete fails. cartDirty trips when a metadata-less
|
||||
// row could not be confirmed-deleted → we must not submit.
|
||||
let cartCount = 0;
|
||||
let cartDirty = false;
|
||||
for (let idx = 0; idx < items.length; idx++) {
|
||||
const item = items[idx]!;
|
||||
const { extractId, input } = item;
|
||||
@@ -316,14 +321,17 @@ async function processBatch(
|
||||
await updateStatus(extractId, "cart");
|
||||
const { basketRowId, numberOfItems, itemIds } =
|
||||
await client.addToCartDetailed(input.prodId ?? 14200);
|
||||
cartCount++;
|
||||
|
||||
// After N successful adds a clean cart reports exactly N items. More
|
||||
// than that = pre-existing junk (orphans from a crash). Never submit a
|
||||
// cart we didn't fully build: wipe everything ANCPI listed and abort —
|
||||
// the next retry starts clean. No charge happens (we never submit).
|
||||
if (numberOfItems > addedCount + 1) {
|
||||
// Right after the add, a clean cart reports exactly cartCount rows.
|
||||
// More than that = pre-existing junk (orphans from a crash). Never
|
||||
// submit a cart we didn't fully build: wipe everything ANCPI listed
|
||||
// and abort — the next retry starts clean. No charge (we never submit).
|
||||
// (numberOfItems falls back to items.length, so an unexpected response
|
||||
// shape degrades to "no excess detected" rather than a false abort.)
|
||||
if (numberOfItems > cartCount) {
|
||||
console.error(
|
||||
`[epay-queue] Dirty cart: expected ${addedCount + 1} rows, ANCPI reports ${numberOfItems}. Wiping + aborting batch.`,
|
||||
`[epay-queue] Dirty cart: expected ${cartCount} rows, ANCPI reports ${numberOfItems}. Wiping + aborting batch.`,
|
||||
);
|
||||
const toWipe = itemIds.length
|
||||
? itemIds
|
||||
@@ -338,7 +346,6 @@ async function processBatch(
|
||||
}
|
||||
|
||||
ourBasketIdsForCleanup.push(basketRowId);
|
||||
addedCount++;
|
||||
item.basketRowId = basketRowId;
|
||||
await updateStatus(extractId, "cart", { basketRowId });
|
||||
|
||||
@@ -379,15 +386,42 @@ async function processBatch(
|
||||
await updateStatus(extractId, "failed", {
|
||||
errorMessage: "Salvarea metadatelor în ePay a eșuat.",
|
||||
});
|
||||
// Remove this metadata-less row from the cart so it can't be
|
||||
// checked out and charged. Drop it from our tracking + batch.
|
||||
await client.deleteCartItem(basketRowId, idx);
|
||||
ourBasketIdsForCleanup.pop();
|
||||
addedCount--;
|
||||
item.basketRowId = undefined;
|
||||
item.basketRowId = undefined; // exclude from validItems regardless
|
||||
// Remove this metadata-less row from the cart so it can't be checked
|
||||
// out and charged. Only drop it from tracking + decrement cartCount
|
||||
// if ANCPI CONFIRMED the delete; otherwise the row is still in the
|
||||
// cart, must stay in cleanup tracking, and the cart is now "dirty".
|
||||
const deleted = await client.deleteCartItem(basketRowId, idx);
|
||||
if (deleted) {
|
||||
cartCount--;
|
||||
ourBasketIdsForCleanup.pop();
|
||||
} else {
|
||||
// Undeletable metadata-less row → submitting now would check it out
|
||||
// and charge for it. Refuse to submit this batch.
|
||||
cartDirty = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A metadata-less row we couldn't remove is still in the global cart;
|
||||
// submitOrder would check out the WHOLE cart and charge for it. Abort
|
||||
// (best-effort wipe) instead of submitting a cart we can't guarantee
|
||||
// clean. The catch below would NOT fire on the success path, so this
|
||||
// explicit guard is what stops the unintended charge.
|
||||
if (cartDirty) {
|
||||
console.error(
|
||||
"[epay-queue] Cart has an undeletable metadata-less row — aborting before submit.",
|
||||
);
|
||||
await client.deleteCartItems(ourBasketIdsForCleanup);
|
||||
for (const id of extractIds) {
|
||||
await updateStatus(id, "failed", {
|
||||
errorMessage:
|
||||
"Coș ePay nu a putut fi curățat (ștergere neconfirmată) — comanda a fost oprită pentru a evita o plată fără metadate. Reîncearcă.",
|
||||
});
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Filter to only items that had successful metadata saves
|
||||
const validItems = items.filter((i) => i.basketRowId !== undefined);
|
||||
if (validItems.length === 0) {
|
||||
@@ -499,8 +533,14 @@ async function finalizeOrder(
|
||||
item: QueueItem;
|
||||
doc: (typeof downloadableDocs)[number];
|
||||
matchedByIndex: boolean;
|
||||
index: number;
|
||||
};
|
||||
const plans: Plan[] = [];
|
||||
// Per-cadastral file index, pre-allocated SEQUENTIALLY so two parallel
|
||||
// tasks for the same cadastral never collide on the MinIO scan (which is
|
||||
// a read-modify-write). Seed each distinct cadastral from MinIO once,
|
||||
// then hand out 1,2,… within this batch.
|
||||
const nextIndexByCad = new Map<string, number>();
|
||||
for (let i = 0; i < validItems.length; i++) {
|
||||
const item = validItems[i]!;
|
||||
const nrCF = item.input.nrCF ?? item.input.nrCadastral;
|
||||
@@ -525,12 +565,19 @@ async function finalizeOrder(
|
||||
});
|
||||
continue;
|
||||
}
|
||||
plans.push({ item, doc, matchedByIndex });
|
||||
|
||||
const cad = item.input.nrCadastral;
|
||||
let next = nextIndexByCad.get(cad);
|
||||
if (next === undefined) next = await getNextFileIndex(cad);
|
||||
nextIndexByCad.set(cad, next + 1);
|
||||
|
||||
plans.push({ item, doc, matchedByIndex, index: next });
|
||||
}
|
||||
|
||||
// Step 6: download + store in parallel (bounded). Each task is fully
|
||||
// self-contained so a failure on one row doesn't abort the others.
|
||||
await runWithConcurrency(plans, DOWNLOAD_CONCURRENCY, async ({ item, doc, matchedByIndex }) => {
|
||||
// One plan's download + store. Returns true on success. On failure it
|
||||
// marks the row failed and returns false so the caller can retry it.
|
||||
const downloadAndStore = async (plan: Plan): Promise<boolean> => {
|
||||
const { item, doc, matchedByIndex, index: fileIndex } = plan;
|
||||
try {
|
||||
await updateStatus(item.extractId, "downloading", {
|
||||
idDocument: doc.idDocument,
|
||||
@@ -552,6 +599,7 @@ async function finalizeOrder(
|
||||
stare: finalStatus.status,
|
||||
produs: "EXI_ONLINE",
|
||||
},
|
||||
fileIndex,
|
||||
);
|
||||
|
||||
if (!doc.dataDocument) {
|
||||
@@ -581,15 +629,43 @@ async function finalizeOrder(
|
||||
console.log(
|
||||
`[epay-queue] ${matchedByIndex ? "Review" : "Completed"}: ${item.input.nrCadastral} → ${path}`,
|
||||
);
|
||||
return true;
|
||||
} catch (error) {
|
||||
const message =
|
||||
error instanceof Error ? error.message : "Eroare download/stocare";
|
||||
await updateStatus(item.extractId, "failed", {
|
||||
errorMessage: message,
|
||||
});
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
// Step 6: download + store in parallel (bounded). Each task is fully
|
||||
// self-contained so a failure on one row doesn't abort the others. The
|
||||
// file index is pre-allocated above, so parallel stores never overwrite.
|
||||
// downloadDocument already retries transient ANCPI errors per call; this
|
||||
// adds a SECOND layer — a final sweep that re-attempts any row still
|
||||
// failed (covers a longer ANCPI blip or a MinIO hiccup) with no new
|
||||
// charge, since the order is already paid.
|
||||
const failed: Plan[] = [];
|
||||
await runWithConcurrency(plans, DOWNLOAD_CONCURRENCY, async (plan) => {
|
||||
const ok = await downloadAndStore(plan);
|
||||
if (!ok) failed.push(plan);
|
||||
});
|
||||
|
||||
if (failed.length > 0) {
|
||||
console.warn(
|
||||
`[epay-queue] ${failed.length}/${plans.length} downloads failed for order ${orderId} — retry sweep in 5s...`,
|
||||
);
|
||||
await new Promise((r) => setTimeout(r, 5000));
|
||||
for (const plan of failed) {
|
||||
const ok = await downloadAndStore(plan);
|
||||
console.log(
|
||||
`[epay-queue] retry sweep ${plan.item.input.nrCadastral}: ${ok ? "recovered" : "still failed"}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Update credits after successful order
|
||||
const newCredits = await client.getCredits();
|
||||
updateEpayCredits(newCredits);
|
||||
|
||||
@@ -83,15 +83,25 @@ export function buildFileName(
|
||||
/**
|
||||
* Store a CF extract PDF in MinIO.
|
||||
* Returns the MinIO path and file index.
|
||||
*
|
||||
* Pass `explicitIndex` to skip the MinIO scan and use a caller-allocated
|
||||
* index. Required when storing concurrently (parallel downloads): the scan
|
||||
* is an unsynchronised read-modify-write, so two tasks for the same
|
||||
* cadastral would both compute index 1 and the second putObject would
|
||||
* silently overwrite the first. The caller pre-allocates distinct indices.
|
||||
*/
|
||||
export async function storeCfExtract(
|
||||
pdfBuffer: Buffer,
|
||||
nrCadastral: string,
|
||||
metadata: Record<string, string>,
|
||||
explicitIndex?: number,
|
||||
): Promise<{ path: string; fileName: string; index: number }> {
|
||||
await ensureAncpiBucket();
|
||||
|
||||
const index = await getNextFileIndex(nrCadastral);
|
||||
const index =
|
||||
explicitIndex !== undefined
|
||||
? explicitIndex
|
||||
: await getNextFileIndex(nrCadastral);
|
||||
const fileName = buildFileName(index, nrCadastral, new Date());
|
||||
// Store in subfolder per cadastral number
|
||||
const path = `parcele/${nrCadastral}/${fileName}`;
|
||||
@@ -124,6 +134,35 @@ export async function getCfExtractStream(
|
||||
return minioClient.getObject(BUCKET, minioPath);
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete stored CF extract objects from MinIO (best-effort, batched).
|
||||
* Returns how many were removed. Used by the 45-day auto-cleanup.
|
||||
*/
|
||||
export async function deleteCfExtractObjects(
|
||||
minioPaths: string[],
|
||||
): Promise<number> {
|
||||
const paths = minioPaths.filter(Boolean);
|
||||
if (paths.length === 0) return 0;
|
||||
try {
|
||||
await minioClient.removeObjects(BUCKET, paths);
|
||||
return paths.length;
|
||||
} catch (error) {
|
||||
// removeObjects can fail wholesale on a transport error — fall back to
|
||||
// per-object deletes so one bad key doesn't block the rest.
|
||||
console.warn("[epay-storage] batch delete failed, retrying per-object:", error);
|
||||
let removed = 0;
|
||||
for (const p of paths) {
|
||||
try {
|
||||
await minioClient.removeObject(BUCKET, p);
|
||||
removed++;
|
||||
} catch (err) {
|
||||
console.warn(`[epay-storage] could not delete ${p}:`, err);
|
||||
}
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* List all stored CF extracts for a cadastral number.
|
||||
*/
|
||||
|
||||
Reference in New Issue
Block a user