vreau-digital/services/seap-scraper/AFIR-HISTORICAL-PLAN.md

AFIR Historical Backfill — Plan & Status

Current state (2026-05-09)

source_year	rows	distinct beneficiars	sum UE (EUR)	fund
2023	474,720	320,230	1,411,870,796	FEADR
2024	563,310	316,304	1,373,722,134	FEADR
Total	1,037,930	—	~2.79 mld EUR	FEADR

Schema: fonduri.afir_plati (migration 017_fonduri_afir.sql). Importer: cron/import-afir-historical.sh + scripts/import-afir-historical.py.

Source survey

AFIR official portal — https://www.afir.ro/rapoarte/beneficiari-de-fonduri-europene/

Two complementary pages:

1. /date-deschise/ — only the most recent two years are linked.
   • Currently exposes 2023 + 2024 for FEADR (xlsx) and 2023 + 2024 for FEGA (rar).
2. /beneficiari-fega-si-feadr/ — ASP.NET portal at https://plati.afir.info/Plati/AfisareListaPlatii. Year selector currently exposes only 2023 and 2024. 3.7M total records in the live query interface but no programmatic XLSX dump older than 2023.

data.gov.ro CKAN — searched q=afir, q=fega, q=apia, q=feadr

Findings (relevant package IDs only):

Dataset	URL	Notes
Date privind proiectele PNDR (a2884dcf-…)	proiectepndr2020.csv (2014-2020), proiectepndr2013.csv (2007-2013)	Project-level, not payment-level. Useful for joining contracts/projects but does not replace plati. Worth ingesting separately.
Contracte AFIR (8845aa0d-…)	contracte-achizitii-publice-peste-5000-euro-2000.xlsx, centralizator-…2021_2022.xlsx	Procurement contracts >5K EUR run by AFIR itself; not beneficiary payments. Different schema.
Lista Fermierilor Campania APIA 2024 (39e5465d-…)	lista-fermieri-apia-2024.xlsx	One-off small dataset; APIA campaign list.
Parcele Agricole APIA LPIS 2025 etc.	shapefiles (.zip)	Geographic parcels, not payments. Useful later for map overlays.

Conclusion: data.gov.ro does not have listaplati_2020/2021/2022_* payment dumps. They exist nowhere public.

opendata.afir.info

A separate CKAN-style portal (http://opendata.afir.info/) lists ProiectePNDR2020 (53K views), ProiectePS2027, AchizitiiPrivate2020. The page itself doesn't expose direct download URLs without account login. Worth investigating in next session — it may contain the 2020-2022 payment data behind an export interface.

Importer architecture

Pipeline (FEADR XLSX)

AFIR XLSX  ──curl──▶  satra:/tmp/afir-historical-{YEAR}-{FUND}/
            │
            ▼
       openpyxl read_only  (skips 9 banner rows)
            │
            ▼
       pipe-delimited TSV  (RO decimals "12.345,67" → "12345.67")
            │
            ▼
       \\copy → fonduri.staging_afir
            │
            ▼
       DELETE FROM afir_plati WHERE source_year=YEAR  (idempotent)
            │
            ▼
       INSERT INTO afir_plati (source_year=YEAR, NULLIF + ::numeric casts)

Why pipe delimiter

Beneficiar names contain commas ("FULOP ZOLTAN, GERGELY"), Obiectiv contains both , and quote chars. Pipe is safer than comma + quoting and the loader already replaces any literal | in source text with / before serialization.

Idempotency

DELETE WHERE source_year = N runs only on full ingests (not when LIMIT is set for smoke tests). Re-running for the same year is safe and produces consistent counts.

Smoke test mode

./import-afir-historical.sh URL YEAR feadr 1000

The 4th arg (LIMIT) skips the DELETE step and truncates the TSV to N rows before COPY, so you can validate end-to-end without trampling production data.

Next-session work

1. FEGA ingest (HIGHEST IMPACT, 30-60 min)

Volume: 2,476,897 rows in 2023 alone, ~580 MB CSV inside 23 MB RAR. Source URLs:

• 2023: https://www.afir.ro/media/sxcnuvwc/listaplati_2023_fega_corectat.rar
• 2024: https://www.afir.ro/media/dqjddti2/lista-plati-beneificiari-fega-2024.rar

Schema differences vs FEADR XLSX (column-by-column):

FEADR XLSX (RO header)	FEGA CSV (concat header)	Notes
Numele beneficiarului	DenumireBeneficiar	same
Numele de familie	NumeFamilie	same
Denumirea societatii-mama si codul de inregistrare fiscala	Cui	FEGA CSV exposes a real CUI column (mostly empty for natural persons, populated for SRL/PFA — bonus enrichment vs FEADR XLSX)
Localitate	Localicate (typo in source)	same content
Codul masurii/tipului de interventie	Masura	same; FEGA codes look like MICA / scheme acronyms instead of M 06 etc
Obiectiv	ObiectivSpecific	longer descriptions
Data inceperii / Data incheierii	DataIncepere / DataSfarsit	usually empty
Cuantum {Operatiune,Total} {FEGA,FEADR}	same 4 columns	decimals already in . format (English-locale, no comma swap needed)
Cuantum aferent operatiunii	CuantumAferentOperatiune	same
Cuantum total cofinantare beneficiari	CuantumTotalCofinantareBeneficiar	same
Cuantum total UE Beneficiar	CuantumtotalUEBenefeciar (typo in source)	same

Implementation choices:

Option A — augment afir_plati with tip_fond discriminator. Add ALTER TABLE fonduri.afir_plati ADD COLUMN tip_fond text CHECK (tip_fond IN ('FEADR','FEGA')); Re-tag existing rows as 'FEADR'. Importer writes both. Uniform downstream query.

Option B — separate table fonduri.fega_plati. Different cardinality (5x rows), different measure code namespace; some queries naturally separate. But duplicates the index/MV maintenance burden.

Recommendation: Option A. The schema is identical, the differences are namespace-of-codes only. A single discriminator keeps things simple, fits the existing gin_trgm name index, and lets the recipe code do WHERE tip_fond='FEGA' cheaply (b-tree on tip_fond if needed).

FEGA importer changes vs current FEADR script:

1. Download → unrar x (already installed on satra now: apt install unrar was run).
2. New python normalizer import-afir-historical-fega.py — reads CSV not XLSX; column-name remapping; no RO-decimal swap.
3. Pass new FUND=fega flag → script writes tip_fond='FEGA' and uses CSV path.
4. Cui column passthrough — write directly into the existing cui column when non-empty, with cui_match_method='afir_self_reported' and cui_match_score=1.0. Skip fuzzy matcher for these.

Volume budget: 2.48M rows × 2 years = ~5M rows. Same staging table works (TRUNCATE between runs). Postgres COPY @ ~100K rows/s → ~25s/year for COPY, plus ~60s for INSERT. Total ~5 min per year.

2. Historical FEADR 2020/2021/2022 (BLOCKED on source)

Status: not publicly available.

Investigation outcome:

• AFIR /date-deschise/ page shows only 2023+2024.
• plati.afir.info portal shows only 2023+2024.
• data.gov.ro CKAN has no listaplati_<year> resources.

Options to unblock (in order of cost):

1. Email AFIR direct — comunicare@afir.info and request the historical payment lists 2020-2022 under Law 544/2001 (FOIA equivalent). They are legally obligated to provide. Expected: 2-4 week response.
2. Wayback Machine archive — check https://web.archive.org/web/2023*/afir.ro/rapoarte/beneficiari-de-fonduri-europene/date-deschise/ for snapshots that still link to old XLSX files. URLs may still resolve (AFIR media folder is content-addressed: /media/<hash>/file.xlsx).
3. opendata.afir.info account — the dataset titles AchizitiiPrivate2020, ProiectePNDR2020 suggest historical exports may live here, but the download interface needs login. Apply for an open-data access account.

Estimated row counts when obtained: ~450K-500K per year (extrapolating from 2023 = 475K and 2024 = 563K).

3. APIA-specific datasets (LOWER PRIORITY)

Lista Fermierilor Campania APIA 2024 (small file, ~50K rows expected). This is a subset of FEGA payments (only certain campaigns), so once FEGA 2024 is ingested, this dataset is partially redundant. Worth ingesting into a separate fonduri.apia_fermieri table only if it carries the geographic columns (parcel codes) the FEGA dump lacks.

Geographic LPIS shapefiles (Parcele Agricole APIA LPIS 2025, Categorii de Folosință) are map data, not payment data — defer to when we add map overlays to /achizitii/firma/[cui] profile pages.

Files modified/added in this session

• NEW services/seap-scraper/scripts/import-afir-historical.py — XLSX→TSV normalizer
• NEW services/seap-scraper/cron/import-afir-historical.sh — orchestrator
• NEW services/seap-scraper/AFIR-HISTORICAL-PLAN.md (this file)

fonduri.afir_plati schema unchanged — no migration. The DELETE+INSERT flow uses the existing table as-is. Adding tip_fond discriminator is a follow-up migration when FEGA ingest is implemented.