From 67f3237761547e20f0ef8a3183701bf9935ac0fc Mon Sep 17 00:00:00 2001 From: AI Assistant Date: Fri, 27 Mar 2026 11:41:54 +0200 Subject: [PATCH] docs(geoportal): update evaluation + skills with deployment lessons learned - Portainer CE volume mount pitfall (silent empty directory creation) - Martin Docker tag format change at v1.0 (v prefix dropped) - UNKNOWN GEOMETRY TYPE log is normal for views - Bake-into-image pattern for config files in Portainer deployments - Updated all implementation prompts with Portainer-safe instructions Co-Authored-By: Claude Opus 4.6 (1M context) --- geoportal/TILE-SERVER-EVALUATION.md | 109 +++++++++++++++++++------ geoportal/skill-vector-tile-serving.md | 52 +++++++++--- 2 files changed, 125 insertions(+), 36 deletions(-) diff --git a/geoportal/TILE-SERVER-EVALUATION.md b/geoportal/TILE-SERVER-EVALUATION.md index 37693b3..b8118d7 100644 --- a/geoportal/TILE-SERVER-EVALUATION.md +++ b/geoportal/TILE-SERVER-EVALUATION.md @@ -4,7 +4,7 @@ ArchiTools Geoportal serves vector tiles (MVT) from PostgreSQL 16 + PostGIS 3 via Martin. Data: ~330K GIS features (parcels, buildings, admin boundaries) in EPSG:3844 (Stereo70), growing to 1M+. -Frontend: MapLibre GL JS 5.21, Next.js 16, Docker self-hosted. +Frontend: MapLibre GL JS 5.21, Next.js 16, Docker self-hosted via Portainer CE. --- @@ -12,7 +12,7 @@ Frontend: MapLibre GL JS 5.21, Next.js 16, Docker self-hosted. 1. Martin v0.15.0 was running in **auto-discovery mode** — the existing `martin.yaml` config was never mounted 2. Building labels (`cadastral_ref`) missing from MVT tiles despite the view exposing them -3. Performance concerns at scale (330K → 1M+ features) +3. Performance concerns at scale (330K -> 1M+ features) --- @@ -23,12 +23,12 @@ Frontend: MapLibre GL JS 5.21, Next.js 16, Docker self-hosted. | Aspect | Detail | |---|---| | Root cause | `martin.yaml` not mounted in docker-compose — Martin ran in auto-discovery mode | -| Fix | Mount config + upgrade v0.15 → v1.4.0 | +| Fix | Bake config into custom image via Dockerfile + upgrade v0.15 -> v1.4.0 | | Performance | Fastest tile server benchmarked (2-3x faster than #2 Tegola) | | EPSG:3844 | Native support via `default_srid: 3844` | | New in v1.4 | ZSTD compression, MLT format, materialized views, better logging | -**Status: IMPLEMENTED** — docker-compose.yml updated, building labels activated. +**Status: IMPLEMENTED AND VERIFIED IN PRODUCTION** (2026-03-27) ### 2. pg_tileserv (CrunchyData) @@ -39,7 +39,7 @@ Frontend: MapLibre GL JS 5.21, Next.js 16, Docker self-hosted. | Performance | 2-3x slower than Martin (Rechsteiner benchmark) | | EPSG:3844 | Supported (auto-reprojects via ST_Transform) | | Killer feature | Function-based sources (full SQL tile functions) | -| Dealbreaker | View extent estimation bug (#156) affects all our views, development stagnant | +| Dealbreaker | View extent estimation bug (#156) affects all our views, development stagnant (last release Feb 2025) | **Verdict: NO** — slower, buggy with views, stagnant development. @@ -67,7 +67,7 @@ Frontend: MapLibre GL JS 5.21, Next.js 16, Docker self-hosted. | Aspect | Detail | |---|---| -| 330K features | ~270 MB uncompressed, 800 MB–1.4 GB browser memory | +| 330K features | ~270 MB uncompressed, 800 MB-1.4 GB browser memory | | Browser impact | 10-30s main thread freeze, mobile crash | | Pan/zoom | Full re-fetch on every viewport change, flickering | | Viable range | Only at zoom 16+ with <500 features in viewport | @@ -116,12 +116,27 @@ Source: [github.com/FabianRechsteiner/vector-tiles-benchmark](https://github.com ## Implementation Roadmap -### Phase 1: Martin Fix (DONE) +### Phase 1: Martin Fix — DONE (2026-03-27) Changes applied: -- `docker-compose.yml`: Martin v0.15 → v1.4.0, config mounted, command changed to `--config` -- `martin.yaml`: Comment updated to reflect v1.4 -- `map-viewer.tsx`: Building labels layer activated (`cladiriLabel` at minzoom 16) +- `martin.Dockerfile`: custom image that COPY-s `martin.yaml` into `/config/` +- `docker-compose.yml`: Martin v0.15 -> v1.4.0, build from Dockerfile, `--config` flag +- `martin.yaml`: comment updated to reflect v1.4 +- `map-viewer.tsx`: building labels layer activated (`cladiriLabel` at minzoom 16) + +#### Deployment Lessons Learned + +1. **Docker image tag format changed at v1.0**: old tags use `v` prefix (`v0.15.0`), new tags do not (`1.4.0`). The tag `ghcr.io/maplibre/martin:v1.4.0` does NOT exist — correct is `ghcr.io/maplibre/martin:1.4.0`. + +2. **Portainer CE volume mount pitfall**: volume `./martin.yaml:/config/martin.yaml:ro` fails because Portainer deploys only the docker-compose.yml content, not the full git repo. Docker silently creates an empty directory instead of failing. Solution: bake config into a custom image with a 2-line Dockerfile: + ```dockerfile + FROM ghcr.io/maplibre/martin:1.4.0 + COPY martin.yaml /config/martin.yaml + ``` + +3. **Martin config format is stable**: YAML format unchanged from v0.15 to v1.4 — `postgres.tables`, `connection_string`, `auto_publish`, `properties` map all work identically. No migration needed. + +4. **PostGIS view geometry type**: Martin logs `UNKNOWN GEOMETRY TYPE` for all views — this is normal for nested views (`SELECT * FROM parent_view`). Views don't register specific geometry types in `geometry_columns`. Does not affect tile generation or property inclusion. ### Phase 2A: nginx Tile Cache @@ -134,14 +149,16 @@ Changes applied: Add an nginx reverse proxy cache in front of Martin for tile serving in ArchiTools. Context: -- Martin serves tiles at http://martin:3000 (container name: martin, port 3010 on host) +- Martin serves tiles at http://martin:3000 (container name: martin, internal port 3000) +- Martin is built from martin.Dockerfile (COPY martin.yaml into image) - Traefik proxies external traffic to ArchiTools at tools.beletage.ro - Current tile URL pattern: https://tools.beletage.ro/tiles/{source}/{z}/{x}/{y} - NEXT_PUBLIC_MARTIN_URL=https://tools.beletage.ro/tiles +- Portainer CE deploys from Gitea repo — files must be in git, not just on host Requirements: 1. Create an nginx container `tile-cache` in docker-compose.yml -2. nginx config: proxy_cache for /tiles/* with: +2. nginx config: proxy_cache for all paths with: - Cache zone: 2GB max, keys in shared memory - Cache valid: 200 responses for 1 hour - Stale serving on error/timeout @@ -149,14 +166,13 @@ Requirements: - CORS headers for tiles (Access-Control-Allow-Origin: *) - Gzip/brotli passthrough (Martin already compresses) 3. Route Martin traffic through tile-cache: - - tile-cache listens on port 3010 (replace Martin's host port) + - tile-cache listens on port 3010 (host) -> 80 (container) - tile-cache proxies to http://martin:3000 -4. Add cache purge endpoint or script for post-sync invalidation -5. Volume for persistent cache across container restarts - -Files to modify: -- docker-compose.yml (add tile-cache service, adjust martin ports) -- Create nginx/tile-cache.conf + - Martin removes its host port mapping (only accessible via tile-cache) +4. Volume for persistent cache across container restarts +5. IMPORTANT: nginx config must be baked into a custom image (same pattern as martin.Dockerfile) + because Portainer CE cannot mount files from the repo. Create nginx/tile-cache.conf and + a tile-cache.Dockerfile. Do NOT change the frontend NEXT_PUBLIC_MARTIN_URL — keep the same external URL. Build with `npx next build` to verify zero errors. @@ -173,11 +189,12 @@ Build with `npx next build` to verify zero errors. Implement PMTiles pre-generation for UAT overview layers in ArchiTools Geoportal. Context: -- PostGIS at 10.10.10.166:5432, database architools_db +- PostGIS at 10.10.10.166:5432, database architools_db, user architools_user - Views: gis_uats_z0, gis_uats_z5, gis_uats_z8, gis_uats_z12, gis_administrativ - All geometries EPSG:3844 (Stereo70), need reprojection to 4326 for tippecanoe -- MinIO at 10.10.10.166:9002, bucket for tiles +- MinIO at 10.10.10.166:9002 (API) / 9003 (console), bucket for tiles - Frontend: MapLibre GL JS 5.21 in src/modules/geoportal/components/map-viewer.tsx +- Portainer CE deploys from Gitea — any config files must be baked into Docker images Requirements: @@ -225,7 +242,7 @@ Read CLAUDE.md for project conventions before starting. Test MLT (MapLibre Tiles) format on one layer in ArchiTools Geoportal. Context: -- Martin v1.4.0 running with config at martin.yaml +- Martin v1.4.0 running with config baked in via martin.Dockerfile - MapLibre GL JS 5.21 in src/modules/geoportal/components/map-viewer.tsx - Test layer: gis_terenuri (largest layer, ~250K features) @@ -264,6 +281,7 @@ Evaluate mvt-rs as a replacement for Martin in ArchiTools Geoportal for multi-te Context: - Current: Martin v1.4.0 serving 9 PostGIS sources (views in EPSG:3844) +- Martin config baked into image via martin.Dockerfile - Goal: Expose geoportal to external clients with per-layer access control - mvt-rs repo: https://github.com/mvt-proj/mvt-rs (v0.16.2+, Rust, Salvo framework) @@ -299,7 +317,22 @@ Do NOT modify the production Martin setup. This is a parallel evaluation only. ## Key Technical Details -### Martin v1.4.0 Config (validated compatible) +### Martin v1.4.0 Deployment Architecture + +``` +Gitea repo (martin.yaml + martin.Dockerfile) + -> Portainer CE builds custom image: FROM martin:1.4.0, COPY martin.yaml + -> Container starts with --config /config/martin.yaml + -> Reads DATABASE_URL from environment + -> Serves 9 PostGIS view sources on port 3000 + -> Host maps 3010:3000 + -> Traefik proxies tools.beletage.ro/tiles -> host:3010 +``` + +**Critical**: Do NOT use volume mounts for config files in Portainer CE stacks. +Always bake configs into custom images via Dockerfile COPY. + +### Martin Config (validated compatible v0.15 through v1.4) The `martin.yaml` at project root defines 9 sources with explicit properties. Config format unchanged from v0.15 to v1.4 — no migration needed. @@ -309,13 +342,21 @@ Key config features used: - `default_srid: 3844` — all sources use Stereo70 - `properties:` map per source — explicit column name + PostgreSQL type - `minzoom/maxzoom` per source — controls tile generation range +- `bounds: [20.2, 43.5, 30.0, 48.3]` — approximate Romania extent + +### Docker Image Tag Convention + +Martin changed tag format at v1.0: +- Pre-1.0: `ghcr.io/maplibre/martin:v0.15.0` (with `v` prefix) +- Post-1.0: `ghcr.io/maplibre/martin:1.4.0` (no `v` prefix) +- Also available: `latest`, `nightly` ### PostGIS View Chain ``` -GisFeature table (Prisma) → gis_features view → gis_terenuri / gis_cladiri / gis_administrativ - → gis_terenuri_status / gis_cladiri_status (with JOINs) -GisUat table → gis_uats_z0/z5/z8/z12 (with ST_SimplifyPreserveTopology) +GisFeature table (Prisma) -> gis_features view -> gis_terenuri / gis_cladiri / gis_administrativ + -> gis_terenuri_status / gis_cladiri_status (with JOINs) +GisUat table -> gis_uats_z0/z5/z8/z12 (with ST_SimplifyPreserveTopology) ``` ### MapLibre Layer Architecture @@ -325,14 +366,30 @@ Sources (Martin): gis_uats_z0, z5, z8, z12, administrativ, terenuri, cladiri Layers per source: fill + line + label (where applicable) Selection: Separate highlight layers on terenuri source Drawing: GeoJSON source for freehand/rect polygon +Building labels: cladiriLabel layer, cadastral_ref at minzoom 16 ``` --- +## Deployment Pitfalls (Discovered During Implementation) + +1. **Portainer CE does not expose repo files to containers at runtime.** Volume mounts like `./file.conf:/etc/file.conf:ro` fail silently — Docker creates an empty directory. Always bake config files into custom images via Dockerfile COPY. + +2. **Martin Docker tag format change at v1.0.** `v1.4.0` does not exist, `1.4.0` does. Always check [ghcr.io/maplibre/martin](https://github.com/maplibre/martin/pkgs/container/martin) for actual tags. + +3. **Martin logs `UNKNOWN GEOMETRY TYPE` for PostGIS views.** This is normal — nested views don't register geometry types in `geometry_columns`. Does not affect functionality. + +4. **Martin auto-discovery mode is unreliable for property inclusion.** Always use explicit config with `auto_publish: false` and per-source `properties:` definitions. + +5. **Martin caches source schema at startup.** After PostGIS view DDL changes (e.g., adding columns to gis_features), Martin must be restarted to pick up new columns. + +--- + ## References - [Martin Documentation](https://maplibre.org/martin/) - [Martin Releases](https://github.com/maplibre/martin/releases) +- [Martin Container Registry](https://github.com/maplibre/martin/pkgs/container/martin) - [Vector Tiles Benchmark (Rechsteiner 2025)](https://github.com/FabianRechsteiner/vector-tiles-benchmark) - [PMTiles Specification](https://github.com/protomaps/PMTiles) - [tippecanoe (Felt)](https://github.com/felt/tippecanoe) diff --git a/geoportal/skill-vector-tile-serving.md b/geoportal/skill-vector-tile-serving.md index 9b1023c..7f4c0c6 100644 --- a/geoportal/skill-vector-tile-serving.md +++ b/geoportal/skill-vector-tile-serving.md @@ -65,11 +65,18 @@ postgres: area: float8 ``` +### Docker image tags + +Martin changed tag format at v1.0: +- Pre-1.0: `ghcr.io/maplibre/martin:v0.15.0` (with `v` prefix) +- Post-1.0: `ghcr.io/maplibre/martin:1.4.0` (no `v` prefix) + ### Docker deployment +**If your orchestrator has access to the full repo** (docker-compose CLI, Docker Swarm with repo checkout): ```yaml martin: - image: ghcr.io/maplibre/martin:v1.4.0 + image: ghcr.io/maplibre/martin:1.4.0 command: ["--config", "/config/martin.yaml"] environment: - DATABASE_URL=postgresql://user:pass@host:5432/db @@ -79,6 +86,28 @@ martin: - "3010:3000" ``` +**If using Portainer CE or any system that only sees docker-compose.yml** (not full repo): +Volume mounts for repo files fail silently — Docker creates an empty directory instead. +Bake config into a custom image: + +```dockerfile +# martin.Dockerfile +FROM ghcr.io/maplibre/martin:1.4.0 +COPY martin.yaml /config/martin.yaml +``` + +```yaml +martin: + build: + context: . + dockerfile: martin.Dockerfile + command: ["--config", "/config/martin.yaml"] + environment: + - DATABASE_URL=postgresql://user:pass@host:5432/db + ports: + - "3010:3000" +``` + ### Custom SRID handling Martin handles non-4326/3857 SRIDs natively. Set `default_srid` globally or `srid` per source. Martin reprojects to Web Mercator (3857) internally for tile envelope calculations. Your PostGIS spatial indexes on the source SRID are used correctly. @@ -170,12 +199,15 @@ map.addSource('overview', { ## Common Pitfalls 1. **Martin auto-discovery drops properties** — always use explicit config with `auto_publish: false` -2. **Nested views lose SRID metadata** — cast geometry: `geom::geometry(Geometry, 3844)` -3. **GisUat.geometry is huge** — always `select` to exclude in list queries -4. **Low-zoom tiles scan entire dataset** — use zoom-dependent simplified views -5. **No tile cache by default** — add nginx/Varnish in front of any tile server -6. **tippecanoe requires WGS84** — reproject from custom SRID before generating PMTiles -7. **PMTiles not incrementally updatable** — full rebuild required on data change -8. **Tegola doesn't support custom SRIDs** — only 3857/4326, requires ST_Transform everywhere -9. **pg_tileserv `ST_Estimated_Extent` fails on views** — use materialized views or function layers -10. **Martin caches source schema at startup** — restart after view DDL changes +2. **Martin Docker tag format changed at v1.0** — `v0.15.0` (with v) but `1.4.0` (without v). Check actual tags at ghcr.io. +3. **Portainer CE volume mounts fail silently** — Docker creates empty directory instead of file. Bake configs into images via Dockerfile COPY. +4. **Martin logs `UNKNOWN GEOMETRY TYPE` for views** — normal for nested views, does not affect tile generation +5. **Nested views lose SRID metadata** — cast geometry: `geom::geometry(Geometry, 3844)` +6. **GisUat.geometry is huge** — always `select` to exclude in list queries +7. **Low-zoom tiles scan entire dataset** — use zoom-dependent simplified views +8. **No tile cache by default** — add nginx/Varnish in front of any tile server +9. **tippecanoe requires WGS84** — reproject from custom SRID before generating PMTiles +10. **PMTiles not incrementally updatable** — full rebuild required on data change +11. **Tegola doesn't support custom SRIDs** — only 3857/4326, requires ST_Transform everywhere +12. **pg_tileserv `ST_Estimated_Extent` fails on views** — use materialized views or function layers +13. **Martin caches source schema at startup** — restart after view DDL changes