3 Harmonize Data

Published

July 20, 2026

3.1 Overview

Once raw microdata is available on datalibweb, it is harmonized to the 50by35 schema (see Schema & Variable Dictionary). Harmonization maps survey-specific variables, codes, and units onto the mandatory and optional variables defined by the schema, so that welfare and self-reliance indicators are comparable across surveys, countries, and years.

3.2 Sample harmonization scripts

Example harmonization scripts covering 15 country/survey combinations are available in the Stata/ folder of this repository. They are not intended to be used as-is, but illustrate the harmonization process and provide a starting point for writing survey-specific scripts. “Refugee definition” is abbreviated below — see each script’s header comment for the full sample restriction. welfare_self formulas omit the floor-at-zero (and, where applicable, cap-at-welfare) wrapper for brevity — see the script for the exact expression:

`code`	`year`	`survname`	Welfare	Refugee definition	`welfare_self`	Script
CHL	2022	CASEN	Income	Mother’s country VEN/COL/HTI (proxy)	`ytotcorh` − `yaimcorh` (HH assistance/subsidy income), then per-adult-equivalent-converted	do-file
CMR	2025	FDS	Consumption	`pops==1` — ⚠️ combined refugee/asylum-seeker code, not separable in source data	`nom_cons_minus_aid` (consumption net of aid, source-provided)	do-file
COL	2022	GEIH	Income	`refugee==1`, Venezuelan-born only	`welfare` − `inc_transf_inst` (institutional/gov’t transfers)	do-file
COL	2023	GEIH	Income	`refugee==1`, Venezuelan-born only	`welfare` − `inc_transf_inst` (institutional/gov’t transfers)	do-file
CRI	2023	ENAHO	Income	`lugnac==2` (born in Nicaragua)	`itf` − (`timas_hh` + `ts_hh`) (HH assistance/subsidy transfers), then per-capita/PPP-converted	do-file
CRI	2024	ENAHO	Income	`lugnac==2` (born in Nicaragua)	`itf` − (`timas_hh` + `ts_hh`) (HH assistance/subsidy transfers), then per-capita/PPP-converted	do-file
ETH	2022	SESRE	Consumption	`sample_type` 2–3 (out-/in-camp)	`total_exp_pre` (pre-assistance consumption, source-provided)	do-file
JOR	2021	VAF	Consumption	`Nationality` 1–2 (Syrian, urban/camp)	`pce_without_assistance` (consumption net of aid, source-provided)	do-file
JOR	2023	VAF	Consumption	`Nationality` 1–2 (Syrian, urban/camp)	`pce_without_assistance` (consumption net of aid, source-provided)	do-file
LBN	2023	VASYR	Consumption	`head_nationality==192` (Syria)	`CAwosocialasst` (consumption net of social assistance, source-provided)	do-file
NER	2018	EHCVM	Consumption	`pop_group` 2 (Mali), 3 (Nigeria); IDPs (1) excluded	`welfare` − `income_aid` (HH aid income)	do-file
PSE	2023	PECS	Consumption	`refugee==1` (UNRWA-registered)	`welfare` × (`cons_hh_n_noaid` / `cons_hh_n`) (nominal no-aid/total consumption ratio)	do-file
TCD	2018	EHCVM	Consumption	`pop_group` 1–2 (east/south refugee)	`welfare` − `income_aid` (HH aid income)	do-file
TCD	2022	EHCVM	Consumption	`pop_group` 1–2, 6 (+other-country refugee)	`welfare` − `income_aid` (HH aid income)	do-file
UGA	2018	RHS	Consumption	`refugee_status==1`	`cons_agg_less_aid` (consumption net of aid, source-provided)	do-file

Common conventions across all scripts:

Sample: refugees only — the 50by35 monitoring population excludes IDPs and asylum-seekers. Each script’s comment header documents the full survey-specific classification (population group codes, nationality, birth country, etc.). One exception: Cameroon FDS 2025 codes refugees and asylum-seekers under a single combined category (pops==1) with no variable in any of the raw files to separate them — flagged in that script and pending a fix from the source data provider.
welfare: the survey’s official welfare aggregate, converted to annual per capita LCU.
welfare_self: welfare net of humanitarian/social assistance (per capita, same periodicity and deflation as welfare), floored at zero and capped at welfare. Where the source provides a net-of-aid aggregate directly (e.g. Uganda’s cons_agg_less_aid), that is used; otherwise assistance is netted out of the underlying income/consumption components before the same per-capita/deflation/annualization transformation applied to welfare. Confirm the final definition with the 50by35 methodology team.
Sentinel codes (e.g. age == 9999) are converted to system missing.

After harmonizing, check the file against the schema before upload:

R
Stata

library(haven)
source("R/validate_50by35.R")

df <- read_dta("data/processed/<SurveyID>_WELF.dta")
validate_50by35(df)

use "data/processed/<SurveyID>_WELF.dta", clear
do "Stata/validate_50by35.do"

--- title: "Harmonize Data" date: today --- ## Overview Once raw microdata is available on datalibweb, it is harmonized to the **50by35 schema** (see [Schema & Variable Dictionary](05-schema.qmd)). Harmonization maps survey-specific variables, codes, and units onto the mandatory and optional variables defined by the schema, so that welfare and self-reliance indicators are comparable across surveys, countries, and years. ## Sample harmonization scripts Example harmonization scripts covering 15 country/survey combinations are available in the `Stata/` folder of this repository. They are not intended to be used as-is, but illustrate the harmonization process and provide a starting point for writing survey-specific scripts. "Refugee definition" is abbreviated below — see each script's header comment for the full sample restriction. `welfare_self` formulas omit the floor-at-zero (and, where applicable, cap-at-`welfare`) wrapper for brevity — see the script for the exact expression: | `code` | `year` | `survname` | Welfare | Refugee definition | `welfare_self` | Script | |---|---|---|---|---|---|---| | CHL | 2022 | CASEN | Income | Mother's country VEN/COL/HTI (proxy) | `ytotcorh` − `yaimcorh` (HH assistance/subsidy income), then per-adult-equivalent-converted | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/CHL_2022_CASEN_V01_M_V01_A_FDP_WELF.do) | | CMR | 2025 | FDS | Consumption | `pops==1` — ⚠️ combined refugee/asylum-seeker code, not separable in source data | `nom_cons_minus_aid` (consumption net of aid, source-provided) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/CMR_2025_FDS_V01_M_V01_A_FDP_WELF.do) | | COL | 2022 | GEIH | Income | `refugee==1`, Venezuelan-born only | `welfare` − `inc_transf_inst` (institutional/gov't transfers) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/COL_2022_GEIH_V01_M_V01_A_FDP_WELF.do) | | COL | 2023 | GEIH | Income | `refugee==1`, Venezuelan-born only | `welfare` − `inc_transf_inst` (institutional/gov't transfers) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/COL_2023_GEIH_V01_M_V01_A_FDP_WELF.do) | | CRI | 2023 | ENAHO | Income | `lugnac==2` (born in Nicaragua) | `itf` − (`timas_hh` + `ts_hh`) (HH assistance/subsidy transfers), then per-capita/PPP-converted | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/CRI_2023_ENAHO_V01_M_V01_A_FDP_WELF.do) | | CRI | 2024 | ENAHO | Income | `lugnac==2` (born in Nicaragua) | `itf` − (`timas_hh` + `ts_hh`) (HH assistance/subsidy transfers), then per-capita/PPP-converted | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/CRI_2024_ENAHO_V01_M_V01_A_FDP_WELF.do) | | ETH | 2022 | SESRE | Consumption | `sample_type` 2–3 (out-/in-camp) | `total_exp_pre` (pre-assistance consumption, source-provided) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/ETH_2022_SESRE_V01_M_V01_A_FDP_WELF.do) | | JOR | 2021 | VAF | Consumption | `Nationality` 1–2 (Syrian, urban/camp) | `pce_without_assistance` (consumption net of aid, source-provided) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/JOR_2021_VAF_V01_M_V01_A_FDP_WELF.do) | | JOR | 2023 | VAF | Consumption | `Nationality` 1–2 (Syrian, urban/camp) | `pce_without_assistance` (consumption net of aid, source-provided) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/JOR_2023_VAF_V01_M_V01_A_FDP_WELF.do) | | LBN | 2023 | VASYR | Consumption | `head_nationality==192` (Syria) | `CAwosocialasst` (consumption net of social assistance, source-provided) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/LBN_2023_VASYR_V01_M_V01_A_FDP_WELF.do) | | NER | 2018 | EHCVM | Consumption | `pop_group` 2 (Mali), 3 (Nigeria); IDPs (1) excluded | `welfare` − `income_aid` (HH aid income) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/NER_2018_EHCVM_V01_M_V01_A_FDP_WELF.do) | | PSE | 2023 | PECS | Consumption | `refugee==1` (UNRWA-registered) | `welfare` × (`cons_hh_n_noaid` / `cons_hh_n`) (nominal no-aid/total consumption ratio) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/PSE_2023_PECS_V01_M_V01_A_FDP_WELF.do) | | TCD | 2018 | EHCVM | Consumption | `pop_group` 1–2 (east/south refugee) | `welfare` − `income_aid` (HH aid income) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/TCD_2018_EHCVM_V01_M_V01_A_FDP_WELF.do) | | TCD | 2022 | EHCVM | Consumption | `pop_group` 1–2, 6 (+other-country refugee) | `welfare` − `income_aid` (HH aid income) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/TCD_2022_EHCVM_V01_M_V01_A_FDP_WELF.do) | | UGA | 2018 | RHS | Consumption | `refugee_status==1` | `cons_agg_less_aid` (consumption net of aid, source-provided) | [do-file](https://github.com/GPID-WB/50by35-data/blob/main/stata/UGA_2018_RHS_V01_M_V01_A_FDP_WELF.do) | Common conventions across all scripts: - **Sample**: refugees only — the 50by35 monitoring population excludes IDPs and asylum-seekers. Each script's comment header documents the full survey-specific classification (population group codes, nationality, birth country, etc.). One exception: Cameroon FDS 2025 codes refugees and asylum-seekers under a single combined category (`pops==1`) with no variable in any of the raw files to separate them — flagged in that script and pending a fix from the source data provider. - **`welfare`**: the survey's official welfare aggregate, converted to **annual per capita LCU**. - **`welfare_self`**: welfare net of humanitarian/social assistance (per capita, same periodicity and deflation as `welfare`), floored at zero and capped at `welfare`. Where the source provides a net-of-aid aggregate directly (e.g. Uganda's `cons_agg_less_aid`), that is used; otherwise assistance is netted out of the underlying income/consumption components before the same per-capita/deflation/annualization transformation applied to `welfare`. Confirm the final definition with the 50by35 methodology team. - **Sentinel codes** (e.g. `age == 9999`) are converted to system missing. After harmonizing, check the file against the schema before upload: ::: {.panel-tabset group="language"} ## R ```r library(haven) source("R/validate_50by35.R") df <- read_dta("data/processed/<SurveyID>_WELF.dta") validate_50by35(df) ``` ## Stata ```stata use "data/processed/<SurveyID>_WELF.dta", clear do "Stata/validate_50by35.do" ``` :::