From db9bcd7192f67bfc0a28b68cd23be46aed73ba45 Mon Sep 17 00:00:00 2001 From: AI Assistant Date: Fri, 27 Feb 2026 22:27:07 +0200 Subject: [PATCH] docs: document N+1 performance bug findings and prevention rules --- CLAUDE.md | 176 +++++++++++++++++++++++++++---------------------- SESSION-LOG.md | 55 ++++++++++++++++ 2 files changed, 153 insertions(+), 78 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4802429..db7bb30 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,6 +18,7 @@ git push origin main # auto-deploys via Portainer webhook ## Project Overview **ArchiTools** is a modular internal web dashboard for an architecture/engineering office group of 3 companies: + - **Beletage** (architecture) - **Urban Switch** (urbanism) - **Studii de Teren** (geotechnics) @@ -25,19 +26,21 @@ git push origin main # auto-deploys via Portainer webhook It runs on an on-premise Ubuntu server at `10.10.10.166`, containerized with Docker, managed via Portainer, served by Nginx Proxy Manager. ### Stack -| Layer | Technology | -|---|---| -| Framework | Next.js 16.x, App Router, TypeScript (strict) | -| Styling | Tailwind CSS v4, shadcn/ui | -| Database | PostgreSQL (10.10.10.166:5432) via Prisma v6 ORM | -| Storage | `DatabaseStorageAdapter` (PostgreSQL) — localStorage fallback available | + +| Layer | Technology | +| ------------ | ---------------------------------------------------------------------------- | +| Framework | Next.js 16.x, App Router, TypeScript (strict) | +| Styling | Tailwind CSS v4, shadcn/ui | +| Database | PostgreSQL (10.10.10.166:5432) via Prisma v6 ORM | +| Storage | `DatabaseStorageAdapter` (PostgreSQL) — localStorage fallback available | | File Storage | MinIO (10.10.10.166:9002 API / 9003 UI) — client configured, adapter pending | -| Auth | NextAuth v4 + Authentik OIDC (auth.beletage.ro) | -| Deploy | Docker multi-stage, Portainer CE, Nginx Proxy Manager | -| Repo | Gitea at `https://git.beletage.ro/gitadmin/ArchiTools` | -| Language | Code in **English**, UI in **Romanian** | +| Auth | NextAuth v4 + Authentik OIDC (auth.beletage.ro) | +| Deploy | Docker multi-stage, Portainer CE, Nginx Proxy Manager | +| Repo | Gitea at `https://git.beletage.ro/gitadmin/ArchiTools` | +| Language | Code in **English**, UI in **Romanian** | ### Architecture Principles + - **Module platform, not monolith** — each module isolated with own types/services/hooks/components - **Feature flags** gate module loading (disabled = zero bundle cost) - **Storage abstraction**: `StorageService` interface with adapters (database default via Prisma, localStorage fallback) @@ -93,26 +96,27 @@ legacy/ # Original HTML tools for reference ## Implemented Modules (14/14 — zero placeholders) -| # | Module | Route | Key Features | -|---|---|---|---| -| 1 | **Dashboard** | `/` | KPI cards (6), activity feed (last 20), module grid, external tools | -| 2 | **Email Signature** | `/email-signature` | Multi-company branding, address toggle, live preview, zoom/copy/download | -| 3 | **Word XML Generator** | `/word-xml` | Category-based XML gen, simple/advanced mode, ZIP export | -| 4 | **Registratura** | `/registratura` | CRUD registry, stats, filters, **legal deadline tracking** | -| 5 | **Tag Manager** | `/tag-manager` | CRUD tags, category/scope/color, US/SDT seeds, mandatory categories | -| 6 | **IT Inventory** | `/it-inventory` | Equipment tracking, Address Book contact link, type/status/company filters | -| 7 | **Address Book** | `/address-book` | CRUD contacts, card grid, vCard export, Registratura reverse lookup, **dynamic types** | -| 8 | **Password Vault** | `/password-vault` | CRUD credentials, email field, clickable URLs, strength meter, company scope | -| 9 | **Mini Utilities** | `/mini-utilities` | Text case, char counter, percentage, area converter, U→R, artifact cleaner, MDLPA, PDF reducer, OCR | -| 10 | **Prompt Generator** | `/prompt-generator` | Template-driven prompt builder, 14 builtin templates | -| 11 | **Digital Signatures** | `/digital-signatures` | CRUD assets, drag-and-drop file upload, tag chips | -| 12 | **Word Templates** | `/word-templates` | Template library, 8 categories, version tracking, .docx placeholder auto-detection | -| 13 | **AI Chat** | `/ai-chat` | Session-based chat UI, demo mode (no API keys yet) | -| 14 | **Hot Desk** | `/hot-desk` | 4 desks, week-ahead calendar, room layout (window+door), reserve/cancel | +| # | Module | Route | Key Features | +| --- | ---------------------- | --------------------- | --------------------------------------------------------------------------------------------------- | +| 1 | **Dashboard** | `/` | KPI cards (6), activity feed (last 20), module grid, external tools | +| 2 | **Email Signature** | `/email-signature` | Multi-company branding, address toggle, live preview, zoom/copy/download | +| 3 | **Word XML Generator** | `/word-xml` | Category-based XML gen, simple/advanced mode, ZIP export | +| 4 | **Registratura** | `/registratura` | CRUD registry, stats, filters, **legal deadline tracking** | +| 5 | **Tag Manager** | `/tag-manager` | CRUD tags, category/scope/color, US/SDT seeds, mandatory categories | +| 6 | **IT Inventory** | `/it-inventory` | Equipment tracking, Address Book contact link, type/status/company filters | +| 7 | **Address Book** | `/address-book` | CRUD contacts, card grid, vCard export, Registratura reverse lookup, **dynamic types** | +| 8 | **Password Vault** | `/password-vault` | CRUD credentials, email field, clickable URLs, strength meter, company scope | +| 9 | **Mini Utilities** | `/mini-utilities` | Text case, char counter, percentage, area converter, U→R, artifact cleaner, MDLPA, PDF reducer, OCR | +| 10 | **Prompt Generator** | `/prompt-generator` | Template-driven prompt builder, 14 builtin templates | +| 11 | **Digital Signatures** | `/digital-signatures` | CRUD assets, drag-and-drop file upload, tag chips | +| 12 | **Word Templates** | `/word-templates` | Template library, 8 categories, version tracking, .docx placeholder auto-detection | +| 13 | **AI Chat** | `/ai-chat` | Session-based chat UI, demo mode (no API keys yet) | +| 14 | **Hot Desk** | `/hot-desk` | 4 desks, week-ahead calendar, room layout (window+door), reserve/cancel | ### Registratura — Legal Deadline Tracking (Termene Legale) The Registratura module includes a full legal deadline tracking engine for Romanian construction permitting: + - **16 deadline types** across 5 categories (Avize, Completări, Analiză, Autorizare, Publicitate) - **Working days vs calendar days** with Romanian public holiday support (including Orthodox Easter via Meeus algorithm) - **Backward deadlines** (e.g., AC extension: 45 working days BEFORE expiry) @@ -121,6 +125,7 @@ The Registratura module includes a full legal deadline tracking engine for Roman - **Tabbed UI**: "Registru" tab (existing registry) + "Termene legale" tab (deadline dashboard) Key files: + - `services/working-days.ts` — Romanian holidays, `addWorkingDays()`, `isWorkingDay()` - `services/deadline-catalog.ts` — 16 `DeadlineTypeDef` entries - `services/deadline-service.ts` — `createTrackedDeadline()`, `resolveDeadline()`, `aggregateDeadlines()` @@ -133,23 +138,23 @@ Key files: ### Server: `10.10.10.166` (Ubuntu) -| Service | Port | Purpose | -|---|---|---| -| **ArchiTools** | 3000 | This app (tools.beletage.ro) | -| **Gitea** | 3002 | Git hosting (git.beletage.ro) | -| **PostgreSQL** | 5432 | App database (Prisma ORM) | -| **Portainer** | 9000 | Docker management | -| **Nginx Proxy Manager** | 81 (admin) | Reverse proxy + SSL termination | -| **Uptime Kuma** | 3001 | Service monitoring | -| **MinIO** | 9002 (API) / 9003 (UI) | Object storage | -| **Authentik** | 9100 | SSO (auth.beletage.ro) — **active** | -| **N8N** | 5678 | Workflow automation (future) | -| **Stirling PDF** | 8087 | PDF tools | -| **IT-Tools** | 8085 | Developer utilities | -| **FileBrowser** | 8086 | File management | -| **Netdata** | 19999 | System monitoring | -| **Dozzle** | 9999 | Docker log viewer | -| **CrowdSec** | 8088 | Security | +| Service | Port | Purpose | +| ----------------------- | ---------------------- | ----------------------------------- | +| **ArchiTools** | 3000 | This app (tools.beletage.ro) | +| **Gitea** | 3002 | Git hosting (git.beletage.ro) | +| **PostgreSQL** | 5432 | App database (Prisma ORM) | +| **Portainer** | 9000 | Docker management | +| **Nginx Proxy Manager** | 81 (admin) | Reverse proxy + SSL termination | +| **Uptime Kuma** | 3001 | Service monitoring | +| **MinIO** | 9002 (API) / 9003 (UI) | Object storage | +| **Authentik** | 9100 | SSO (auth.beletage.ro) — **active** | +| **N8N** | 5678 | Workflow automation (future) | +| **Stirling PDF** | 8087 | PDF tools | +| **IT-Tools** | 8085 | Developer utilities | +| **FileBrowser** | 8086 | File management | +| **Netdata** | 19999 | System monitoring | +| **Dozzle** | 9999 | Docker log viewer | +| **CrowdSec** | 8088 | Security | ### Deployment Pipeline @@ -164,6 +169,7 @@ git push origin main ``` ### Docker + - `Dockerfile`: 3-stage build (deps → builder → runner), `node:20-alpine`, non-root user - `Dockerfile` includes `npx prisma generate` before build step - `docker-compose.yml`: single service, port 3000, **all env vars hardcoded** (Portainer CE can't inject env vars) @@ -175,12 +181,14 @@ git push origin main ## Development Rules ### TypeScript Strict Mode Gotchas + - `array.split()[0]` returns `string | undefined` — use `.slice(0, 10)` instead - `Record[key]` returns `T | undefined` — always guard with null check - Spread of possibly-undefined objects: `{ ...obj[key], field }` — check existence first - lucide-react Icons: cast through `unknown` → `React.ComponentType<{ className?: string }>` ### Conventions + - **Code**: English - **UI text**: Romanian - **Components**: functional, `'use client'` directive where needed @@ -189,8 +197,18 @@ git push origin main - **Dates**: ISO strings (`YYYY-MM-DD` for display, full ISO for timestamps) - **No emojis** in code or UI unless explicitly requested +### Storage Performance Rules + +- **NEVER** use `storage.list()` followed by `storage.get()` in a loop — this is an N+1 query bug +- `list()` fetches ALL items (keys+values) from DB but discards values, then each `get()` re-fetches individually +- **ALWAYS** use `storage.exportAll()` (namespaced) or `storage.export(namespace)` (service-level) to batch-load +- Filter items client-side after a single fetch: `for (const [key, value] of Object.entries(all)) { ... }` +- After mutations (add/update), either do optimistic local state update or a single `refresh()` — never both + ### Module Development Pattern + Every module follows: + ``` src/modules// ├── components/ # React components @@ -202,6 +220,7 @@ src/modules// ``` ### Before Pushing + 1. `npx next build` — must pass with zero errors 2. Test the feature manually on `localhost:3000` 3. Commit with descriptive message @@ -211,38 +230,39 @@ src/modules// ## Company IDs -| ID | Name | Prefix | -|---|---|---| -| `beletage` | Beletage | B | -| `urban-switch` | Urban Switch | US | -| `studii-de-teren` | Studii de Teren | SDT | -| `group` | Grup | G | +| ID | Name | Prefix | +| ----------------- | --------------- | ------ | +| `beletage` | Beletage | B | +| `urban-switch` | Urban Switch | US | +| `studii-de-teren` | Studii de Teren | SDT | +| `group` | Grup | G | --- ## Current Integrations -| Feature | Status | Notes | -|---|---|---| -| **Authentik SSO** | ✅ Active | NextAuth v4 + OIDC, group→role/company mapping | -| **PostgreSQL** | ✅ Active | Prisma ORM, `KeyValueStore` model, `/api/storage` route | -| **MinIO** | Client configured | 10.10.10.166:9002, bucket `tools`, adapter pending | -| **AI Chat API** | UI complete, demo mode | No API keys yet; supports Claude/GPT/Ollama | -| **N8N automations** | Webhook URL configured | For notifications, backups, workflows | +| Feature | Status | Notes | +| ------------------- | ---------------------- | ------------------------------------------------------- | +| **Authentik SSO** | ✅ Active | NextAuth v4 + OIDC, group→role/company mapping | +| **PostgreSQL** | ✅ Active | Prisma ORM, `KeyValueStore` model, `/api/storage` route | +| **MinIO** | Client configured | 10.10.10.166:9002, bucket `tools`, adapter pending | +| **AI Chat API** | UI complete, demo mode | No API keys yet; supports Claude/GPT/Ollama | +| **N8N automations** | Webhook URL configured | For notifications, backups, workflows | --- ## Model Recommendations -| Task Type | Claude | OpenAI | Google | Notes | -|---|---|---|---|---| -| **Bug fixes, config** | Haiku 4.5 | GPT-4o-mini | Gemini 2.5 Flash | Fast, cheap | -| **Features, tests, UI** | **Sonnet 4.6** | GPT-5.2 | Gemini 3 Flash | Best value — Opus-class quality at Sonnet price | -| **New modules, architecture** | Opus 4.6 | GPT-5.3-Codex | Gemini 3 Pro | Complex multi-file, business logic | +| Task Type | Claude | OpenAI | Google | Notes | +| ----------------------------- | -------------- | ------------- | ---------------- | ----------------------------------------------- | +| **Bug fixes, config** | Haiku 4.5 | GPT-4o-mini | Gemini 2.5 Flash | Fast, cheap | +| **Features, tests, UI** | **Sonnet 4.6** | GPT-5.2 | Gemini 3 Flash | Best value — Opus-class quality at Sonnet price | +| **New modules, architecture** | Opus 4.6 | GPT-5.3-Codex | Gemini 3 Pro | Complex multi-file, business logic | **Default: Sonnet 4.6** for most work. See `ROADMAP.md` for per-task recommendations. ### Session Handoff Tips + - Read this `CLAUDE.md` first — it has all context - Read `ROADMAP.md` for the complete task list with dependencies - Check `docs/` for deep dives on specific systems @@ -255,21 +275,21 @@ src/modules// ## Documentation Index -| Doc | Path | Content | -|---|---|---| +| Doc | Path | Content | +| ------------------- | ------------------------------------------ | -------------------------------------------- | | System Architecture | `docs/architecture/SYSTEM-ARCHITECTURE.md` | Overall architecture, module platform design | -| Module System | `docs/architecture/MODULE-SYSTEM.md` | Module registry, lifecycle, config format | -| Feature Flags | `docs/architecture/FEATURE-FLAGS.md` | Flag system, env overrides | -| Storage Layer | `docs/architecture/STORAGE-LAYER.md` | StorageService interface, adapters | -| Tagging System | `docs/architecture/TAGGING-SYSTEM.md` | Cross-module tags | -| Security & Roles | `docs/architecture/SECURITY-AND-ROLES.md` | Visibility, auth, roles | -| Module Dev Guide | `docs/guides/MODULE-DEVELOPMENT.md` | How to create a new module | -| HTML Integration | `docs/guides/HTML-TOOL-INTEGRATION.md` | Legacy tool migration | -| UI Design System | `docs/guides/UI-DESIGN-SYSTEM.md` | Design tokens, component patterns | -| Docker Deployment | `docs/guides/DOCKER-DEPLOYMENT.md` | Full Docker/Portainer/Nginx guide | -| Coding Standards | `docs/guides/CODING-STANDARDS.md` | TS strict, naming, patterns | -| Testing Strategy | `docs/guides/TESTING-STRATEGY.md` | Testing approach | -| Configuration | `docs/guides/CONFIGURATION.md` | Env vars, flags, companies | -| Data Model | `docs/DATA-MODEL.md` | All entity schemas | -| Repo Structure | `docs/REPO-STRUCTURE.md` | Directory layout | -| Prompt Generator | `docs/modules/PROMPT-GENERATOR.md` | Prompt module deep dive | +| Module System | `docs/architecture/MODULE-SYSTEM.md` | Module registry, lifecycle, config format | +| Feature Flags | `docs/architecture/FEATURE-FLAGS.md` | Flag system, env overrides | +| Storage Layer | `docs/architecture/STORAGE-LAYER.md` | StorageService interface, adapters | +| Tagging System | `docs/architecture/TAGGING-SYSTEM.md` | Cross-module tags | +| Security & Roles | `docs/architecture/SECURITY-AND-ROLES.md` | Visibility, auth, roles | +| Module Dev Guide | `docs/guides/MODULE-DEVELOPMENT.md` | How to create a new module | +| HTML Integration | `docs/guides/HTML-TOOL-INTEGRATION.md` | Legacy tool migration | +| UI Design System | `docs/guides/UI-DESIGN-SYSTEM.md` | Design tokens, component patterns | +| Docker Deployment | `docs/guides/DOCKER-DEPLOYMENT.md` | Full Docker/Portainer/Nginx guide | +| Coding Standards | `docs/guides/CODING-STANDARDS.md` | TS strict, naming, patterns | +| Testing Strategy | `docs/guides/TESTING-STRATEGY.md` | Testing approach | +| Configuration | `docs/guides/CONFIGURATION.md` | Env vars, flags, companies | +| Data Model | `docs/DATA-MODEL.md` | All entity schemas | +| Repo Structure | `docs/REPO-STRUCTURE.md` | Directory layout | +| Prompt Generator | `docs/modules/PROMPT-GENERATOR.md` | Prompt module deep dive | diff --git a/SESSION-LOG.md b/SESSION-LOG.md index cd40d18..bdf9af2 100644 --- a/SESSION-LOG.md +++ b/SESSION-LOG.md @@ -4,6 +4,61 @@ --- +## Session — 2026-02-27 late night #2 (GitHub Copilot - Claude Opus 4.6) + +### Context + +Performance investigation: Registratura loading extremely slowly with only 6 entries. Rack numbering inverted. + +### Root Cause Analysis — Findings + +**CRITICAL BUG: N+1 Query Pattern in ALL Storage Hooks** + +The `DatabaseStorageAdapter.list()` method fetches ALL items (keys + values) from PostgreSQL in one HTTP request, but **discards the values** and returns only the key names. Then every hook calls `storage.get(key)` for EACH key individually — making a separate HTTP request + DB query per item. + +With 6 registry entries + ~10 contacts + ~20 tags, the Registratura page fired **~40 sequential HTTP requests** on load (**1 list + N gets per hook × 3 hooks**). Each request goes through: browser → Next.js API route → Prisma → PostgreSQL → back. This is a textbook N+1 query problem. + +**Additional issues found:** + +- `addEntry()` called `getAllEntries()` for number generation, then `refresh()` called `getAllEntries()` again → double-fetch +- `closeEntry()` called `updateEntry()` (which refreshes), then manually called `refresh()` again → double-refresh +- Every single module hook had the same pattern (11 hooks total) +- Rack visualization rendered U1 at top instead of bottom + +### Fix Applied + +**Strategy:** Replace `list()` + N × `get()` with a single `exportAll()` call that fetches all items in one HTTP request and filters client-side. + +**Files fixed (13 total):** + +- `src/modules/registratura/services/registry-service.ts` — added `exportAll` to `RegistryStorage` interface, rewrote `getAllEntries` +- `src/modules/registratura/hooks/use-registry.ts` — `addEntry` uses optimistic local state update instead of double-refresh; `closeEntry` batches saves with `Promise.all` + single refresh +- `src/modules/address-book/hooks/use-contacts.ts` — `exportAll` batch load +- `src/modules/it-inventory/hooks/use-inventory.ts` — `exportAll` batch load +- `src/modules/password-vault/hooks/use-vault.ts` — `exportAll` batch load +- `src/modules/word-templates/hooks/use-templates.ts` — `exportAll` batch load +- `src/modules/prompt-generator/hooks/use-prompt-generator.ts` — `exportAll` batch load +- `src/modules/hot-desk/hooks/use-reservations.ts` — `exportAll` batch load +- `src/modules/email-signature/hooks/use-saved-signatures.ts` — `exportAll` batch load +- `src/modules/digital-signatures/hooks/use-signatures.ts` — `exportAll` batch load +- `src/modules/ai-chat/hooks/use-chat.ts` — `exportAll` batch load +- `src/core/tagging/tag-service.ts` — uses `storage.export()` instead of N+1 +- `src/modules/it-inventory/components/server-rack.tsx` — reversed slot rendering (U1 at bottom) + +**Performance impact:** ~90% reduction in HTTP requests on page load. Registratura: from ~40 requests to 3 (one per namespace: registratura, address-book, tags). + +### Prevention Rules (for future development) + +> **NEVER** use the `storage.list()` + loop `storage.get()` pattern. +> Always use `storage.exportAll()` to load all items in a namespace, then filter client-side. +> This is the #1 performance pitfall in the storage layer. + +### Commits + +- `c45a30e` perf: fix N+1 query pattern across all modules + rack numbering + +--- + ## Session — 2026-02-27 late night (GitHub Copilot - Claude Opus 4.6) ### Context