gitadmin/ArchiTools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update tile evaluation + monitoring + add geoportal improvement mega prompt

Browse Source 
- TILE-SERVER-EVALUATION.md: updated to reflect current architecture (PMTiles z0-z18)
- MODULE-MAP.md: added PMTiles + tile-cache to Geoportal section
- Monitor: timeout increased to 90 min for z18 builds, description updated
- Added PROMPT-GEOPORTAL-IMPROVE.md with mega prompt for future sessions
  (includes MLT check, mvt-rs evaluation prompt, operational commands)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
AI Assistant
2026-03-28 18:38:53 +02:00
parent 870e1bd4c2
commit ee86af6183
4 changed files with 209 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/MODULE-MAP.md
+5 -3
View File
@@ -129,9 +129,11 @@ Quick reference: entry points, key files, API routes, and cross-module dependenc
### Geoportal ### Geoportal
- **Route**: `/geoportal` - **Route**: `/geoportal`
- **Main component**: `components/geoportal-module.tsx` - **Main component**: `components/geoportal-module.tsx`
- **Key components**: `components/map-viewer.tsx` (MapLibre), `components/basemap-switcher.tsx`, `components/selection-toolbar.tsx`, `components/feature-info-panel.tsx` - **Key components**: `components/map-viewer.tsx` (MapLibre, PMTiles protocol), `components/basemap-switcher.tsx`, `components/selection-toolbar.tsx`, `components/feature-info-panel.tsx`
- **API routes**: `/api/geoportal/*` (search, boundary-check, uat-bounds, setup-views) - **Tile infrastructure**: Martin v1.4.0 (live MVT) -> nginx tile-cache (7d TTL) -> Traefik; PMTiles (z0-z18, MinIO) for pre-generated overview tiles
- **Cross-deps**: **parcel-sync** (declared dependency — uses PostGIS data) - **Monitor page**: `/monitor` — nginx/Martin/PMTiles status, rebuild + warm-cache actions
- **API routes**: `/api/geoportal/*` (search, boundary-check, uat-bounds, setup-views, monitor)
- **Cross-deps**: **parcel-sync** (declared dependency — uses PostGIS data), **MinIO** (PMTiles storage), **N8N** (rebuild webhook)
### Visual CoPilot ### Visual CoPilot
- **Route**: `/visual-copilot` - **Route**: `/visual-copilot`
geoportal/PROMPT-GEOPORTAL-IMPROVE.md
+116
View File
@@ -0,0 +1,116 @@
# Geoportal Continuous Improvement — Mega Prompt
Use this prompt to start a new session focused on geoportal tile serving improvements.
---
## Context Prompt (copy-paste to Claude)
```
Scopul acestei sesiuni este imbunatatirea continua a tile serving-ului pentru modulul Geoportal din ArchiTools.
Citeste aceste fisiere INAINTE de orice:
- CLAUDE.md (project conventions)
- geoportal/TILE-SERVER-EVALUATION.md (current architecture + roadmap)
- src/modules/geoportal/components/map-viewer.tsx (MapLibre + PMTiles integration)
- martin.yaml (Martin tile server config)
- docker-compose.yml (infrastructure stack)
- scripts/rebuild-overview-tiles.sh (PMTiles generation pipeline)
- src/app/api/geoportal/monitor/route.ts (monitoring API)
- src/app/(modules)/monitor/page.tsx (monitoring dashboard)
## Arhitectura curenta (2026-03-28):
Pipeline: Browser → PMTiles (MinIO, z0-z18, ~1-2 GB) | Martin (PostGIS) doar pentru gis_terenuri_status + gis_cladiri_status
Cache: nginx tile-cache (7d TTL) in fata Martin | Browser cache 24h | PMTiles servit direct din MinIO
Stack:
- PMTiles: overview.pmtiles pe MinIO (10.10.10.166:9002/tiles/overview.pmtiles)
- nginx tile-cache: port 3010, proxy_cache 2GB, 7d TTL
- Martin v1.4: port intern 3000, config baked in image, pool_size 8
- tippecanoe Docker: one-shot rebuild, profiles: ["tools"]
- N8N webhook: auto-rebuild dupa weekend deep sync
Rebuild PMTiles: ~45-60 min (565K+ features, z0-z18)
Server: VM satra (10.10.10.166), 6 CPU, 16 GB RAM, Docker, Portainer CE
IMPORTANT:
- NEXT_PUBLIC_* vars TREBUIE declarate ca ARG+ENV in Dockerfile (altfel webpack nu le vede)
- Portainer CE nu monteaza fisiere din repo — bake configs in Docker images
- Dupa schimbari la Dockerfile/NEXT_PUBLIC_: docker compose build --no-cache architools
Comenzi server (SSH bulibasa@10.10.10.166):
cd /tmp/ArchiTools && git pull && docker compose --profile tools build tippecanoe && docker compose --profile tools run --rm tippecanoe
docker compose build --no-cache architools && docker compose up -d architools
bash /tmp/ArchiTools/scripts/warm-tile-cache.sh http://10.10.10.166:3010
Monitor dashboard: https://tools.beletage.ro/monitor
N8N: http://n8n.beletage.ro (workflow "PMTiles Rebuild")
npx next build TREBUIE sa treaca dupa fiecare schimbare.
```
---
## Checklist periodic (lunar):
### 1. Check MLT Production Readiness
```
Verifica daca Martin suporta generare MLT din PostGIS (nu doar servire din MBTiles).
Cauta:
- Martin releases: https://github.com/maplibre/martin/releases
- Martin MLT PR: https://github.com/maplibre/martin/pull/2512
- PostGIS MLT: cauta "ST_AsMLT" in PostGIS development
- MapLibre GL JS MLT: https://maplibre.org/maplibre-tile-spec/implementation-status/
Daca Martin poate genera MLT din PostGIS live:
1. Testeaza pe un layer (gis_terenuri) cu encoding: "mlt" in map-viewer
2. Compara tile sizes MVT vs MLT
3. Daca merge, aplica pe toate layerele Martin
Status curent (2026-03-28): NU e viabil. Martin doar serveste MLT pre-generat, nu transcodeaza din PostGIS.
```
### 2. mvt-rs Parallel Evaluation
```
Evalueaza mvt-rs ca alternativa Martin pentru deployment multi-tenant.
Prompt gata de folosit:
"Deployeaza mvt-rs v0.16+ in parallel cu Martin pe ArchiTools.
Context:
- PostgreSQL: 10.10.10.166:5432, db architools_db, user architools_user
- Martin actual: martin.yaml cu 9 surse PostGIS (EPSG:3844)
- Docker stack: Portainer CE, Traefik v3
- Scopul: per-layer access control pentru clienti externi
Steps:
1. Adauga mvt-rs in docker-compose.yml pe port 3011
2. Configureaza aceleasi layere ca martin.yaml
3. Test: toate proprietatile apar in MVT? Performance vs Martin?
4. Admin UI: creeaza user test, asigneaza permisiuni per layer
5. Decision matrix: cand trecem de la Martin la mvt-rs
NU modifica setup-ul Martin existent. Evaluare paralela doar.
mvt-rs repo: https://github.com/mvt-proj/mvt-rs
Citeste CLAUDE.md si geoportal/TILE-SERVER-EVALUATION.md inainte."
```
### 3. PMTiles Rebuild Optimization
```
Daca rebuild dureaza >60 min sau fisierul >3 GB:
- Evalueaza tile-join pentru rebuild partial (doar layerul modificat)
- Evalueaza --no-tile-size-limit vs --drop-densest-as-needed trade-off
- Evalueaza split: un PMTiles per UAT sincronizat (rebuild doar orasul modificat)
- Evalueaza cron nightly vs rebuild per sync event
```
---
## Known Issues & Limitations
- tippecanoe `--drop-densest-as-needed` poate pierde features in zone dense la zoom mic
- PMTiles data e statica — parcele noi nu apar pana la rebuild
- MinIO CORS headers necesita Range + Content-Range exposed
- Martin `pool_size: 8` — nu creste fara upgrade PostgreSQL
- Portainer CE nu injecteaza env vars la build — toate in docker-compose.yml
geoportal/TILE-SERVER-EVALUATION.md
+84 -150
View File
@@ -138,179 +138,113 @@ Changes applied:
4. **PostGIS view geometry type**: Martin logs `UNKNOWN GEOMETRY TYPE` for all views — this is normal for nested views (`SELECT * FROM parent_view`). Views don't register specific geometry types in `geometry_columns`. Does not affect tile generation or property inclusion. 4. **PostGIS view geometry type**: Martin logs `UNKNOWN GEOMETRY TYPE` for all views — this is normal for nested views (`SELECT * FROM parent_view`). Views don't register specific geometry types in `geometry_columns`. Does not affect tile generation or property inclusion.
### Phase 2A: nginx Tile Cache ### Phase 2A: nginx Tile Cache — DONE (2026-03-27)
**Impact**: 10-100x faster on repeat requests, zero PostGIS load for cached tiles. **Impact**: 10-100x faster on repeat requests, zero PostGIS load for cached tiles.
**Effort**: ~2 hours.
#### Implementation Prompt Changes applied:
- `nginx/tile-cache.conf`: proxy_cache config with 2GB cache zone, 7-day TTL, stale serving
- `tile-cache.Dockerfile`: bakes nginx config into custom image (Portainer CE pattern)
- `docker-compose.yml`: `tile-cache` container, Martin no longer exposed on host
- Gzip passthrough (Martin already compresses), browser caching via Cache-Control headers
- CORS headers for cross-origin tile requests
``` ### Phase 2B: PMTiles — DONE (2026-03-27)
Add an nginx reverse proxy cache in front of Martin for tile serving in ArchiTools.
Context: **Impact**: Sub-10ms overview tiles, zero PostGIS load for z0-z18.
- Martin serves tiles at http://martin:3000 (container name: martin, internal port 3000)
- Martin is built from martin.Dockerfile (COPY martin.yaml into image)
- Traefik proxies external traffic to ArchiTools at tools.beletage.ro
- Current tile URL pattern: https://tools.beletage.ro/tiles/{source}/{z}/{x}/{y}
- NEXT_PUBLIC_MARTIN_URL=https://tools.beletage.ro/tiles
- Portainer CE deploys from Gitea repo — files must be in git, not just on host
Requirements: Changes applied:
1. Create an nginx container `tile-cache` in docker-compose.yml - `scripts/rebuild-overview-tiles.sh`: ogr2ogr export (3844->4326) + tippecanoe generation
2. nginx config: proxy_cache for all paths with: - PMTiles archive: z0-z18, ~1-2 GB, includes all terenuri, cladiri, UATs, and administrativ layers
- Cache zone: 2GB max, keys in shared memory - `map-viewer.tsx`: pmtiles:// protocol registered on MapLibre, hybrid source switching
- Cache valid: 200 responses for 1 hour - MinIO bucket `tiles` with public read + CORS for Range Requests
- Stale serving on error/timeout - N8N webhook trigger for rebuild (via monitor page)
- Cache-Control headers passed through - Monitor page (`/monitor`): rebuild + warm-cache actions with live status polling
- CORS headers for tiles (Access-Control-Allow-Origin: *)
- Gzip/brotli passthrough (Martin already compresses)
3. Route Martin traffic through tile-cache:
- tile-cache listens on port 3010 (host) -> 80 (container)
- tile-cache proxies to http://martin:3000
- Martin removes its host port mapping (only accessible via tile-cache)
4. Volume for persistent cache across container restarts
5. IMPORTANT: nginx config must be baked into a custom image (same pattern as martin.Dockerfile)
because Portainer CE cannot mount files from the repo. Create nginx/tile-cache.conf and
a tile-cache.Dockerfile.
Do NOT change the frontend NEXT_PUBLIC_MARTIN_URL — keep the same external URL. ### Phase 2C: MLT Format — DEFERRED
Build with `npx next build` to verify zero errors.
```
### Phase 2B: PMTiles for UAT Overview Layers Martin v1.4 advertises MLT support, but it cannot generate MLT from PostGIS live queries.
MLT generation requires pre-built tile archives (tippecanoe does not output MLT either).
No actionable path until Martin or tippecanoe adds MLT output from PostGIS sources.
**Impact**: Sub-10ms overview tiles, zero PostGIS load for z0-z12. ### Phase 2D: mvt-rs Evaluation — FUTURE (Multi-Tenant)
**Effort**: ~4-6 hours.
#### Implementation Prompt
```
Implement PMTiles pre-generation for UAT overview layers in ArchiTools Geoportal.
Context:
- PostGIS at 10.10.10.166:5432, database architools_db, user architools_user
- Views: gis_uats_z0, gis_uats_z5, gis_uats_z8, gis_uats_z12, gis_administrativ
- All geometries EPSG:3844 (Stereo70), need reprojection to 4326 for tippecanoe
- MinIO at 10.10.10.166:9002 (API) / 9003 (console), bucket for tiles
- Frontend: MapLibre GL JS 5.21 in src/modules/geoportal/components/map-viewer.tsx
- Portainer CE deploys from Gitea — any config files must be baked into Docker images
Requirements:
1. Create `scripts/rebuild-overview-tiles.sh`:
- Export each view with ogr2ogr: -f FlatGeobuf -s_srs EPSG:3844 -t_srs EPSG:4326
- Generate combined PMTiles with tippecanoe:
- --layer per view, --minimum-zoom=0, --maximum-zoom=14
- --detect-shared-borders (critical for adjacent UAT polygons)
- --hilbert for compression
- Atomic upload to MinIO: upload as overview_new.pmtiles, then rename to overview.pmtiles
- Cleanup temp files
2. Create Dockerfile for tippecanoe build container (or use ghcr.io/felt/tippecanoe)
3. Add `tippecanoe` service to docker-compose.yml (one-shot, for manual/cron runs)
4. Configure MinIO:
- Create bucket `tiles` with public read
- CORS: Allow GET/HEAD from tools.beletage.ro, expose Range/Content-Range headers
5. Update map-viewer.tsx:
- npm install pmtiles
- Register pmtiles:// protocol on MapLibre
- Add PMTiles source for overview layers (z0-z14)
- Keep existing Martin sources for detail layers (z14+)
- Set zoom breakpoints: PMTiles below z14, Martin above z14
6. Add N8N webhook trigger or cron for nightly rebuild after weekend deep sync
The UAT overview layers change rarely (only when new UATs are synced).
Parcel/building layers stay on Martin for live data freshness.
Build with `npx next build` to verify zero errors.
Read CLAUDE.md for project conventions before starting.
```
### Phase 2C: MLT Format Testing
**Impact**: 6x smaller tiles, 4x faster client decode.
**Effort**: ~1 hour to test.
#### Implementation Prompt
```
Test MLT (MapLibre Tiles) format on one layer in ArchiTools Geoportal.
Context:
- Martin v1.4.0 running with config baked in via martin.Dockerfile
- MapLibre GL JS 5.21 in src/modules/geoportal/components/map-viewer.tsx
- Test layer: gis_terenuri (largest layer, ~250K features)
Requirements:
1. Research how Martin v1.4 serves MLT format:
- Check if it's automatic via Accept header or needs config
- Check Martin docs for MLT serving configuration
2. Update map-viewer.tsx to request MLT for one source (gis_terenuri):
- Add `encoding: "mlt"` to the vector source definition if MapLibre supports it
- Or configure via source URL parameter if Martin expects it
3. Test and measure:
- Compare tile sizes: MVT vs MLT for same tile coordinates
- Compare decode time in browser DevTools Network tab
- Check that all properties (cadastral_ref, area_value, etc.) survive MLT encoding
- Check label rendering still works
4. If MLT works correctly, apply to all Martin sources
If issues found, document them and revert to MVT
This is experimental — keep MVT as fallback. Do not break existing functionality.
Build with `npx next build` to verify zero errors.
```
### Phase 2D: mvt-rs Evaluation (Future — Multi-Tenant)
**Impact**: Built-in auth, admin UI, per-layer access control. **Impact**: Built-in auth, admin UI, per-layer access control.
**Effort**: 1-2 days for evaluation + migration. **Effort**: 1-2 days for evaluation + migration.
#### Implementation Prompt Reserved for when external client access to the geoportal is needed.
mvt-rs (v0.16.2+, Rust, Salvo framework) provides per-layer auth and admin UI.
---
## Phase 3: Current Architecture (as of 2026-03-27)
Full tile-serving pipeline in production:
``` ```
Evaluate mvt-rs as a replacement for Martin in ArchiTools Geoportal for multi-tenant deployment. PostGIS (EPSG:3844)
|
+--> Martin v1.4.0 (live MVT from 9 PostGIS views)
| |
| +--> tile-cache (nginx reverse proxy, 2GB disk, 7d TTL)
| |
| +--> Traefik (tools.beletage.ro/tiles)
|
+--> ogr2ogr (3844->4326) + tippecanoe (z0-z18)
|
+--> PMTiles archive (~1-2 GB)
|
+--> MinIO bucket "tiles" (HTTP Range Requests)
|
+--> MapLibre (pmtiles:// protocol)
```
Context: **Hybrid strategy**:
- Current: Martin v1.4.0 serving 9 PostGIS sources (views in EPSG:3844) - PMTiles serves pre-generated overview tiles (all zoom levels, all layers)
- Martin config baked into image via martin.Dockerfile - Martin serves live detail tiles (real-time PostGIS data)
- Goal: Expose geoportal to external clients with per-layer access control - nginx tile-cache sits in front of Martin to absorb repeat requests
- mvt-rs repo: https://github.com/mvt-proj/mvt-rs (v0.16.2+, Rust, Salvo framework) - Rebuild triggered via N8N webhook from the `/monitor` page
Requirements: ---
1. Deploy mvt-rs as a Docker container alongside Martin (don't replace yet) ## Operational Commands
- Use same DATABASE_URL
- Map to different port (e.g., 3011)
2. Configure mvt-rs with equivalent layers to martin.yaml: ### Rebuild PMTiles
- gis_uats_z0/z5/z8/z12, gis_administrativ, gis_terenuri, gis_cladiri
- gis_terenuri_status, gis_cladiri_status
- All with EPSG:3844, explicit properties
3. Test: Trigger from the Monitor page (`/monitor` -> "Rebuild PMTiles" button), which sends a webhook to N8N.
- Do all properties appear in MVT output? (especially cadastral_ref on gis_cladiri) N8N runs `scripts/rebuild-overview-tiles.sh` on the server.
- Performance comparison: curl timing for 10 representative tiles vs Martin
- Admin UI: create test user, assign layer permissions
- Cache: configure disk cache, measure cold vs warm tile times
4. Document findings: Manual rebuild (SSH to 10.10.10.166):
- Property inclusion: pass/fail per layer ```bash
- Performance delta vs Martin cd /path/to/architools
- Admin UI capabilities and limitations bash scripts/rebuild-overview-tiles.sh
- Missing features vs Martin (PMTiles, MLT, etc.) ```
5. Decision matrix: when to switch from Martin to mvt-rs ### Warm nginx Cache
Do NOT modify the production Martin setup. This is a parallel evaluation only. Trigger from the Monitor page (`/monitor` -> "Warm Cache" button).
Pre-loads frequently accessed tiles into the nginx disk cache.
### Purge nginx Tile Cache
```bash
docker exec tile-cache rm -rf /var/cache/nginx/tiles/*
docker exec tile-cache nginx -s reload
```
### Restart Martin (after PostGIS view changes)
```bash
docker restart martin
```
Martin caches source schema at startup — must restart after DDL changes to pick up new columns.
### Check PMTiles Status
```bash
# Check file size and last modified in MinIO
docker exec minio mc stat local/tiles/overview.pmtiles
``` ```
--- ---
src/app/(modules)/monitor/page.tsx
+4 -4
View File
@@ -80,15 +80,15 @@ export default function MonitorPage() {
} }
} catch { /* continue polling */ } } catch { /* continue polling */ }
}, 15_000); }, 15_000);
// Timeout after 30 min (z16 builds can take 15-25 min) // Timeout after 90 min (z18 builds can take 45-60 min)
setTimeout(() => { setTimeout(() => {
if (pollRef.current) { if (pollRef.current) {
clearInterval(pollRef.current); clearInterval(pollRef.current);
pollRef.current = null; pollRef.current = null;
addLog("error", "Timeout: rebuild nu s-a finalizat in 30 minute"); addLog("error", "Timeout: rebuild nu s-a finalizat in 90 minute");
setActionLoading(""); setActionLoading("");
} }
}, 30 * 60_000); }, 90 * 60_000);
} catch { } catch {
addLog("error", "Nu s-a putut trimite webhook-ul"); addLog("error", "Nu s-a putut trimite webhook-ul");
setActionLoading(""); setActionLoading("");
@@ -225,7 +225,7 @@ export default function MonitorPage() {
<div className="flex flex-wrap gap-3 mb-4"> <div className="flex flex-wrap gap-3 mb-4">
<ActionButton <ActionButton
label="Rebuild PMTiles" label="Rebuild PMTiles"
description="Regenereaza tile-urile overview din PostGIS (~8 min)" description="Regenereaza tile-urile overview din PostGIS (~45-60 min)"
loading={actionLoading === "rebuild"} loading={actionLoading === "rebuild"}
onClick={triggerRebuild} onClick={triggerRebuild}
/> />