diff --git a/docs/MODULE-MAP.md b/docs/MODULE-MAP.md index df98264..2133e54 100644 --- a/docs/MODULE-MAP.md +++ b/docs/MODULE-MAP.md @@ -129,9 +129,11 @@ Quick reference: entry points, key files, API routes, and cross-module dependenc ### Geoportal - **Route**: `/geoportal` - **Main component**: `components/geoportal-module.tsx` -- **Key components**: `components/map-viewer.tsx` (MapLibre), `components/basemap-switcher.tsx`, `components/selection-toolbar.tsx`, `components/feature-info-panel.tsx` -- **API routes**: `/api/geoportal/*` (search, boundary-check, uat-bounds, setup-views) -- **Cross-deps**: **parcel-sync** (declared dependency — uses PostGIS data) +- **Key components**: `components/map-viewer.tsx` (MapLibre, PMTiles protocol), `components/basemap-switcher.tsx`, `components/selection-toolbar.tsx`, `components/feature-info-panel.tsx` +- **Tile infrastructure**: Martin v1.4.0 (live MVT) -> nginx tile-cache (7d TTL) -> Traefik; PMTiles (z0-z18, MinIO) for pre-generated overview tiles +- **Monitor page**: `/monitor` — nginx/Martin/PMTiles status, rebuild + warm-cache actions +- **API routes**: `/api/geoportal/*` (search, boundary-check, uat-bounds, setup-views, monitor) +- **Cross-deps**: **parcel-sync** (declared dependency — uses PostGIS data), **MinIO** (PMTiles storage), **N8N** (rebuild webhook) ### Visual CoPilot - **Route**: `/visual-copilot` diff --git a/geoportal/PROMPT-GEOPORTAL-IMPROVE.md b/geoportal/PROMPT-GEOPORTAL-IMPROVE.md new file mode 100644 index 0000000..aec589f --- /dev/null +++ b/geoportal/PROMPT-GEOPORTAL-IMPROVE.md @@ -0,0 +1,116 @@ +# Geoportal Continuous Improvement — Mega Prompt + +Use this prompt to start a new session focused on geoportal tile serving improvements. + +--- + +## Context Prompt (copy-paste to Claude) + +``` +Scopul acestei sesiuni este imbunatatirea continua a tile serving-ului pentru modulul Geoportal din ArchiTools. + +Citeste aceste fisiere INAINTE de orice: +- CLAUDE.md (project conventions) +- geoportal/TILE-SERVER-EVALUATION.md (current architecture + roadmap) +- src/modules/geoportal/components/map-viewer.tsx (MapLibre + PMTiles integration) +- martin.yaml (Martin tile server config) +- docker-compose.yml (infrastructure stack) +- scripts/rebuild-overview-tiles.sh (PMTiles generation pipeline) +- src/app/api/geoportal/monitor/route.ts (monitoring API) +- src/app/(modules)/monitor/page.tsx (monitoring dashboard) + +## Arhitectura curenta (2026-03-28): + +Pipeline: Browser → PMTiles (MinIO, z0-z18, ~1-2 GB) | Martin (PostGIS) doar pentru gis_terenuri_status + gis_cladiri_status +Cache: nginx tile-cache (7d TTL) in fata Martin | Browser cache 24h | PMTiles servit direct din MinIO + +Stack: +- PMTiles: overview.pmtiles pe MinIO (10.10.10.166:9002/tiles/overview.pmtiles) +- nginx tile-cache: port 3010, proxy_cache 2GB, 7d TTL +- Martin v1.4: port intern 3000, config baked in image, pool_size 8 +- tippecanoe Docker: one-shot rebuild, profiles: ["tools"] +- N8N webhook: auto-rebuild dupa weekend deep sync + +Rebuild PMTiles: ~45-60 min (565K+ features, z0-z18) +Server: VM satra (10.10.10.166), 6 CPU, 16 GB RAM, Docker, Portainer CE + +IMPORTANT: +- NEXT_PUBLIC_* vars TREBUIE declarate ca ARG+ENV in Dockerfile (altfel webpack nu le vede) +- Portainer CE nu monteaza fisiere din repo — bake configs in Docker images +- Dupa schimbari la Dockerfile/NEXT_PUBLIC_: docker compose build --no-cache architools + +Comenzi server (SSH bulibasa@10.10.10.166): +cd /tmp/ArchiTools && git pull && docker compose --profile tools build tippecanoe && docker compose --profile tools run --rm tippecanoe +docker compose build --no-cache architools && docker compose up -d architools +bash /tmp/ArchiTools/scripts/warm-tile-cache.sh http://10.10.10.166:3010 + +Monitor dashboard: https://tools.beletage.ro/monitor +N8N: http://n8n.beletage.ro (workflow "PMTiles Rebuild") + +npx next build TREBUIE sa treaca dupa fiecare schimbare. +``` + +--- + +## Checklist periodic (lunar): + +### 1. Check MLT Production Readiness +``` +Verifica daca Martin suporta generare MLT din PostGIS (nu doar servire din MBTiles). +Cauta: +- Martin releases: https://github.com/maplibre/martin/releases +- Martin MLT PR: https://github.com/maplibre/martin/pull/2512 +- PostGIS MLT: cauta "ST_AsMLT" in PostGIS development +- MapLibre GL JS MLT: https://maplibre.org/maplibre-tile-spec/implementation-status/ + +Daca Martin poate genera MLT din PostGIS live: +1. Testeaza pe un layer (gis_terenuri) cu encoding: "mlt" in map-viewer +2. Compara tile sizes MVT vs MLT +3. Daca merge, aplica pe toate layerele Martin + +Status curent (2026-03-28): NU e viabil. Martin doar serveste MLT pre-generat, nu transcodeaza din PostGIS. +``` + +### 2. mvt-rs Parallel Evaluation +``` +Evalueaza mvt-rs ca alternativa Martin pentru deployment multi-tenant. + +Prompt gata de folosit: + +"Deployeaza mvt-rs v0.16+ in parallel cu Martin pe ArchiTools. + +Context: +- PostgreSQL: 10.10.10.166:5432, db architools_db, user architools_user +- Martin actual: martin.yaml cu 9 surse PostGIS (EPSG:3844) +- Docker stack: Portainer CE, Traefik v3 +- Scopul: per-layer access control pentru clienti externi + +Steps: +1. Adauga mvt-rs in docker-compose.yml pe port 3011 +2. Configureaza aceleasi layere ca martin.yaml +3. Test: toate proprietatile apar in MVT? Performance vs Martin? +4. Admin UI: creeaza user test, asigneaza permisiuni per layer +5. Decision matrix: cand trecem de la Martin la mvt-rs + +NU modifica setup-ul Martin existent. Evaluare paralela doar. +mvt-rs repo: https://github.com/mvt-proj/mvt-rs +Citeste CLAUDE.md si geoportal/TILE-SERVER-EVALUATION.md inainte." +``` + +### 3. PMTiles Rebuild Optimization +``` +Daca rebuild dureaza >60 min sau fisierul >3 GB: +- Evalueaza tile-join pentru rebuild partial (doar layerul modificat) +- Evalueaza --no-tile-size-limit vs --drop-densest-as-needed trade-off +- Evalueaza split: un PMTiles per UAT sincronizat (rebuild doar orasul modificat) +- Evalueaza cron nightly vs rebuild per sync event +``` + +--- + +## Known Issues & Limitations +- tippecanoe `--drop-densest-as-needed` poate pierde features in zone dense la zoom mic +- PMTiles data e statica — parcele noi nu apar pana la rebuild +- MinIO CORS headers necesita Range + Content-Range exposed +- Martin `pool_size: 8` — nu creste fara upgrade PostgreSQL +- Portainer CE nu injecteaza env vars la build — toate in docker-compose.yml diff --git a/geoportal/TILE-SERVER-EVALUATION.md b/geoportal/TILE-SERVER-EVALUATION.md index b8118d7..d97a1b1 100644 --- a/geoportal/TILE-SERVER-EVALUATION.md +++ b/geoportal/TILE-SERVER-EVALUATION.md @@ -138,179 +138,113 @@ Changes applied: 4. **PostGIS view geometry type**: Martin logs `UNKNOWN GEOMETRY TYPE` for all views — this is normal for nested views (`SELECT * FROM parent_view`). Views don't register specific geometry types in `geometry_columns`. Does not affect tile generation or property inclusion. -### Phase 2A: nginx Tile Cache +### Phase 2A: nginx Tile Cache — DONE (2026-03-27) **Impact**: 10-100x faster on repeat requests, zero PostGIS load for cached tiles. -**Effort**: ~2 hours. -#### Implementation Prompt +Changes applied: +- `nginx/tile-cache.conf`: proxy_cache config with 2GB cache zone, 7-day TTL, stale serving +- `tile-cache.Dockerfile`: bakes nginx config into custom image (Portainer CE pattern) +- `docker-compose.yml`: `tile-cache` container, Martin no longer exposed on host +- Gzip passthrough (Martin already compresses), browser caching via Cache-Control headers +- CORS headers for cross-origin tile requests -``` -Add an nginx reverse proxy cache in front of Martin for tile serving in ArchiTools. +### Phase 2B: PMTiles — DONE (2026-03-27) -Context: -- Martin serves tiles at http://martin:3000 (container name: martin, internal port 3000) -- Martin is built from martin.Dockerfile (COPY martin.yaml into image) -- Traefik proxies external traffic to ArchiTools at tools.beletage.ro -- Current tile URL pattern: https://tools.beletage.ro/tiles/{source}/{z}/{x}/{y} -- NEXT_PUBLIC_MARTIN_URL=https://tools.beletage.ro/tiles -- Portainer CE deploys from Gitea repo — files must be in git, not just on host +**Impact**: Sub-10ms overview tiles, zero PostGIS load for z0-z18. -Requirements: -1. Create an nginx container `tile-cache` in docker-compose.yml -2. nginx config: proxy_cache for all paths with: - - Cache zone: 2GB max, keys in shared memory - - Cache valid: 200 responses for 1 hour - - Stale serving on error/timeout - - Cache-Control headers passed through - - CORS headers for tiles (Access-Control-Allow-Origin: *) - - Gzip/brotli passthrough (Martin already compresses) -3. Route Martin traffic through tile-cache: - - tile-cache listens on port 3010 (host) -> 80 (container) - - tile-cache proxies to http://martin:3000 - - Martin removes its host port mapping (only accessible via tile-cache) -4. Volume for persistent cache across container restarts -5. IMPORTANT: nginx config must be baked into a custom image (same pattern as martin.Dockerfile) - because Portainer CE cannot mount files from the repo. Create nginx/tile-cache.conf and - a tile-cache.Dockerfile. +Changes applied: +- `scripts/rebuild-overview-tiles.sh`: ogr2ogr export (3844->4326) + tippecanoe generation +- PMTiles archive: z0-z18, ~1-2 GB, includes all terenuri, cladiri, UATs, and administrativ layers +- `map-viewer.tsx`: pmtiles:// protocol registered on MapLibre, hybrid source switching +- MinIO bucket `tiles` with public read + CORS for Range Requests +- N8N webhook trigger for rebuild (via monitor page) +- Monitor page (`/monitor`): rebuild + warm-cache actions with live status polling -Do NOT change the frontend NEXT_PUBLIC_MARTIN_URL — keep the same external URL. -Build with `npx next build` to verify zero errors. -``` +### Phase 2C: MLT Format — DEFERRED -### Phase 2B: PMTiles for UAT Overview Layers +Martin v1.4 advertises MLT support, but it cannot generate MLT from PostGIS live queries. +MLT generation requires pre-built tile archives (tippecanoe does not output MLT either). +No actionable path until Martin or tippecanoe adds MLT output from PostGIS sources. -**Impact**: Sub-10ms overview tiles, zero PostGIS load for z0-z12. -**Effort**: ~4-6 hours. - -#### Implementation Prompt - -``` -Implement PMTiles pre-generation for UAT overview layers in ArchiTools Geoportal. - -Context: -- PostGIS at 10.10.10.166:5432, database architools_db, user architools_user -- Views: gis_uats_z0, gis_uats_z5, gis_uats_z8, gis_uats_z12, gis_administrativ -- All geometries EPSG:3844 (Stereo70), need reprojection to 4326 for tippecanoe -- MinIO at 10.10.10.166:9002 (API) / 9003 (console), bucket for tiles -- Frontend: MapLibre GL JS 5.21 in src/modules/geoportal/components/map-viewer.tsx -- Portainer CE deploys from Gitea — any config files must be baked into Docker images - -Requirements: - -1. Create `scripts/rebuild-overview-tiles.sh`: - - Export each view with ogr2ogr: -f FlatGeobuf -s_srs EPSG:3844 -t_srs EPSG:4326 - - Generate combined PMTiles with tippecanoe: - - --layer per view, --minimum-zoom=0, --maximum-zoom=14 - - --detect-shared-borders (critical for adjacent UAT polygons) - - --hilbert for compression - - Atomic upload to MinIO: upload as overview_new.pmtiles, then rename to overview.pmtiles - - Cleanup temp files - -2. Create Dockerfile for tippecanoe build container (or use ghcr.io/felt/tippecanoe) - -3. Add `tippecanoe` service to docker-compose.yml (one-shot, for manual/cron runs) - -4. Configure MinIO: - - Create bucket `tiles` with public read - - CORS: Allow GET/HEAD from tools.beletage.ro, expose Range/Content-Range headers - -5. Update map-viewer.tsx: - - npm install pmtiles - - Register pmtiles:// protocol on MapLibre - - Add PMTiles source for overview layers (z0-z14) - - Keep existing Martin sources for detail layers (z14+) - - Set zoom breakpoints: PMTiles below z14, Martin above z14 - -6. Add N8N webhook trigger or cron for nightly rebuild after weekend deep sync - -The UAT overview layers change rarely (only when new UATs are synced). -Parcel/building layers stay on Martin for live data freshness. - -Build with `npx next build` to verify zero errors. -Read CLAUDE.md for project conventions before starting. -``` - -### Phase 2C: MLT Format Testing - -**Impact**: 6x smaller tiles, 4x faster client decode. -**Effort**: ~1 hour to test. - -#### Implementation Prompt - -``` -Test MLT (MapLibre Tiles) format on one layer in ArchiTools Geoportal. - -Context: -- Martin v1.4.0 running with config baked in via martin.Dockerfile -- MapLibre GL JS 5.21 in src/modules/geoportal/components/map-viewer.tsx -- Test layer: gis_terenuri (largest layer, ~250K features) - -Requirements: - -1. Research how Martin v1.4 serves MLT format: - - Check if it's automatic via Accept header or needs config - - Check Martin docs for MLT serving configuration - -2. Update map-viewer.tsx to request MLT for one source (gis_terenuri): - - Add `encoding: "mlt"` to the vector source definition if MapLibre supports it - - Or configure via source URL parameter if Martin expects it - -3. Test and measure: - - Compare tile sizes: MVT vs MLT for same tile coordinates - - Compare decode time in browser DevTools Network tab - - Check that all properties (cadastral_ref, area_value, etc.) survive MLT encoding - - Check label rendering still works - -4. If MLT works correctly, apply to all Martin sources - If issues found, document them and revert to MVT - -This is experimental — keep MVT as fallback. Do not break existing functionality. -Build with `npx next build` to verify zero errors. -``` - -### Phase 2D: mvt-rs Evaluation (Future — Multi-Tenant) +### Phase 2D: mvt-rs Evaluation — FUTURE (Multi-Tenant) **Impact**: Built-in auth, admin UI, per-layer access control. **Effort**: 1-2 days for evaluation + migration. -#### Implementation Prompt +Reserved for when external client access to the geoportal is needed. +mvt-rs (v0.16.2+, Rust, Salvo framework) provides per-layer auth and admin UI. + +--- + +## Phase 3: Current Architecture (as of 2026-03-27) + +Full tile-serving pipeline in production: ``` -Evaluate mvt-rs as a replacement for Martin in ArchiTools Geoportal for multi-tenant deployment. +PostGIS (EPSG:3844) + | + +--> Martin v1.4.0 (live MVT from 9 PostGIS views) + | | + | +--> tile-cache (nginx reverse proxy, 2GB disk, 7d TTL) + | | + | +--> Traefik (tools.beletage.ro/tiles) + | + +--> ogr2ogr (3844->4326) + tippecanoe (z0-z18) + | + +--> PMTiles archive (~1-2 GB) + | + +--> MinIO bucket "tiles" (HTTP Range Requests) + | + +--> MapLibre (pmtiles:// protocol) +``` -Context: -- Current: Martin v1.4.0 serving 9 PostGIS sources (views in EPSG:3844) -- Martin config baked into image via martin.Dockerfile -- Goal: Expose geoportal to external clients with per-layer access control -- mvt-rs repo: https://github.com/mvt-proj/mvt-rs (v0.16.2+, Rust, Salvo framework) +**Hybrid strategy**: +- PMTiles serves pre-generated overview tiles (all zoom levels, all layers) +- Martin serves live detail tiles (real-time PostGIS data) +- nginx tile-cache sits in front of Martin to absorb repeat requests +- Rebuild triggered via N8N webhook from the `/monitor` page -Requirements: +--- -1. Deploy mvt-rs as a Docker container alongside Martin (don't replace yet) - - Use same DATABASE_URL - - Map to different port (e.g., 3011) +## Operational Commands -2. Configure mvt-rs with equivalent layers to martin.yaml: - - gis_uats_z0/z5/z8/z12, gis_administrativ, gis_terenuri, gis_cladiri - - gis_terenuri_status, gis_cladiri_status - - All with EPSG:3844, explicit properties +### Rebuild PMTiles -3. Test: - - Do all properties appear in MVT output? (especially cadastral_ref on gis_cladiri) - - Performance comparison: curl timing for 10 representative tiles vs Martin - - Admin UI: create test user, assign layer permissions - - Cache: configure disk cache, measure cold vs warm tile times +Trigger from the Monitor page (`/monitor` -> "Rebuild PMTiles" button), which sends a webhook to N8N. +N8N runs `scripts/rebuild-overview-tiles.sh` on the server. -4. Document findings: - - Property inclusion: pass/fail per layer - - Performance delta vs Martin - - Admin UI capabilities and limitations - - Missing features vs Martin (PMTiles, MLT, etc.) +Manual rebuild (SSH to 10.10.10.166): +```bash +cd /path/to/architools +bash scripts/rebuild-overview-tiles.sh +``` -5. Decision matrix: when to switch from Martin to mvt-rs +### Warm nginx Cache -Do NOT modify the production Martin setup. This is a parallel evaluation only. +Trigger from the Monitor page (`/monitor` -> "Warm Cache" button). +Pre-loads frequently accessed tiles into the nginx disk cache. + +### Purge nginx Tile Cache + +```bash +docker exec tile-cache rm -rf /var/cache/nginx/tiles/* +docker exec tile-cache nginx -s reload +``` + +### Restart Martin (after PostGIS view changes) + +```bash +docker restart martin +``` + +Martin caches source schema at startup — must restart after DDL changes to pick up new columns. + +### Check PMTiles Status + +```bash +# Check file size and last modified in MinIO +docker exec minio mc stat local/tiles/overview.pmtiles ``` --- diff --git a/src/app/(modules)/monitor/page.tsx b/src/app/(modules)/monitor/page.tsx index ea7f43f..487d9b6 100644 --- a/src/app/(modules)/monitor/page.tsx +++ b/src/app/(modules)/monitor/page.tsx @@ -80,15 +80,15 @@ export default function MonitorPage() { } } catch { /* continue polling */ } }, 15_000); - // Timeout after 30 min (z16 builds can take 15-25 min) + // Timeout after 90 min (z18 builds can take 45-60 min) setTimeout(() => { if (pollRef.current) { clearInterval(pollRef.current); pollRef.current = null; - addLog("error", "Timeout: rebuild nu s-a finalizat in 30 minute"); + addLog("error", "Timeout: rebuild nu s-a finalizat in 90 minute"); setActionLoading(""); } - }, 30 * 60_000); + }, 90 * 60_000); } catch { addLog("error", "Nu s-a putut trimite webhook-ul"); setActionLoading(""); @@ -225,7 +225,7 @@ export default function MonitorPage() {