4 Upload Harmonized Data

Published

July 20, 2026

4.1 Overview

Harmonized microdata and harmonization scripts are uploaded to datalibweb via PRIMUS. After review, approved files are published to datalibweb for 50by35 monitoring.

4.2 Uploading via PRIMUS

Harmonized files are uploaded as a PRIMUS transaction using the FDP-harmonized-data process (processid = 40) by users with uploader status.

4.2.1 Validation checks

All harmonized files must pass validation checks before they can be uploaded to datalibweb. These confirm that the harmonized file conforms to the schema, and that mandatory variables are present and non-missing. Any deviation from the schema will trigger an error and prevent the harmonized file from being uploaded to datalibweb. They check for missing values in mandatory variables, for negative or zero values in welfare and weight, and that welfare_self is not greater than welfare. Households where hhsize differs from the number of pid records are flagged with a warning (rosters can be incomplete, and refugee-only samples keep a subset of household members). The validation scripts are Stata/validate_50by35.do and R/validate_50by35.R in this repository.

4.2.2 XML file

A harmonized data upload starts with an indicator .xml file that opens the transaction and returns a transaction ID. The XML contains metadata about the harmonized file, the poverty headcount from welfare, and the 50by35 self-reliance share computed from welfare_self at the $3.00/day 2021 PPP line (placeholder — pending confirmation from the 50by35 methodology team).

The XML’s content can be customized, provided it follows the PRIMUS XML schema for harmonized data: a <Request> section (RequestKey, welfare, weight, By, N_By_Group, nParamSets, plus a key;value CDATA block of run metadata), a <Result> section (one <Welfare> per welfare variable, each containing a <ByGroup> with a <DATASUMMARY> and one <CALCULATION> per nParamSets — also key;value CDATA blocks), and a <LOG_DETAIL> free-text block that records calculation details but is not shown in the approval table. Both upload scripts in this repository write the same document, with two CALCULATION entries per group: the poverty headcount (from welfare) and the 50by35 self-reliance share (from welfare_self).

4.2.3 CPI and PPP conversion factors

The conversion factors used to calculate the indicator for the XML file come from the World Bank PIP auxiliary tables, pinned to the 2021 PPP framework (R: pipr::get_aux("cpi", ppp_version = 2021); Stata: pip tables, table(cpi) ppp_year(2021) clear):

PPP — the ICP 2021 purchasing-power-parity conversion factor (LCU per 2021 international $), one value per country.
CPI — PIP’s consumer price index, normalized so that CPI $= 1$ in 2021 and aligned to each survey’s fieldwork period, one value per country-year.

They enter the self-reliance indicator exactly as in the formula on the home page: welfare_self (annual per capita LCU at survey-year prices) is divided by 365 to get a daily value, by the survey-year CPI to convert to 2021 prices, and by the PPP factor to convert to 2021 PPP dollars, then compared with $z = \$3.00$ (placeholder value, pending confirmation from the 50by35 methodology team).

Surveys not in PIP

PIP’s CPI table only contains values for country-years with a PIP survey. For refugee surveys in years not covered by PIP, the upload scripts apply an automatic CPI fallback rather than stopping with an error:

Interpolation — if the survey year falls between two years that do have CPI values, the scripts linearly interpolate between them. For example, Uganda 2018 is bracketed by the 2016 and 2019 PIP survey years, giving an interpolated value of $0.875 + \tfrac{2}{3}(0.967 - 0.875) \approx 0.936$.
Nearest year — if the survey year is beyond the range of available PIP CPI values (e.g. a 2025 survey when PIP’s latest value is 2023), the most recent available value is used as-is (flat extrapolation).

The fallback method is recorded in the indicator XML’s Method field (e.g. Interpolated(CPI_2016=0.874967,CPI_2019=0.967133) or NearestYear(CPI_2023=1.140000)) so reviewers can see exactly how the CPI was derived.

If neither fallback is acceptable, set cpi_override in the upload script to the ratio $CPI_{t}/CPI_{2021}$ from the national CPI series (e.g. WDI series FP.CPI.TOTL). The override is recorded as Method;ManualOverride in the XML.

For convenience, the validation checks, XML creation and upload steps are combined in one script per client — Stata/03_upload_harmonized.do and R/03_upload_harmonized.R in this repository — condensed below.

After uploading, the uploader confirms the draft transaction. Only then does it become visible to approvers.

Include the harmonization scripts

Upload the code used to harmonize the raw data together with the harmonized dataset, so the harmonization is fully reproducible from source. Add them as additional files on the same transaction ID, using the folder path matching the file type:

Folder path	Extensions
`Programs`	`do`, `py`, `r`
`Data/Harmonized`	`dta`

Note - the examples below are summaries, use the full scripts in this repository.

R
Stata

library(primus)
library(pipr)
library(haven)

source("R/validate_50by35.R")
survey_id <- "COL_2023_GEIH_V01_M_V01_A_FDP"

# 1. Validate: any schema violation stops before upload
df <- read_dta(paste0("data/processed/", survey_id, "_WELF.dta"))
validate_50by35(df)

ccode <- df$code[1]
yr    <- df$year[1]

# 2. CPI / PPP conversion factors from PIP, pinned to the 2021 framework.
#    If no exact CPI match exists for this country-year, the script
#    automatically falls back to linear interpolation between the two
#    nearest PIP survey years (or the nearest single year at the edges).
#    See @sec-cpi-ppp and R/03_upload_harmonized.R for details.
cpi_table <- get_aux("cpi", ppp_version = 2021)
cpi_exact <- cpi_table[cpi_table$country_code == ccode &
                         as.integer(as.character(cpi_table$year)) == yr &
                         cpi_table$data_level == "national", ]
# exact match; for fallback logic see R/03_upload_harmonized.R
cpi_value <- cpi_exact$value

ppp_table <- get_aux("ppp", ppp_version = 2021)
icp_value <- ppp_table$value[ppp_table$country_code == ccode &
                               ppp_table$data_level == "national"][1]

# 3. Compute the self-reliance share and write the indicator XML —
#    see R/03_upload_harmonized.R for the XML writer, which follows the
#    PRIMUS XML schema (Request/Result/LOG_DETAIL sections, key;value
#    CDATA blocks, one CALCULATION per nParamSets).
#    3.00 is a placeholder PPP line, pending confirmation from the
#    50by35 methodology team.
sr <- weighted.mean(df$welfare_self / 365 / cpi_value / icp_value >= 3.00,
                    df$weight)

# 4. The XML opens the transaction; then attach the data and programs
up <- primus_upload("FDP-harmonized-data", survey_id, type = "harmonized",
                    infile = "indicator.xml", xml = "indicator.xml")
primus_upload("FDP-harmonized-data", survey_id, type = "harmonized",
              infile = paste0("data/processed/", survey_id, "_WELF.dta"),
              folder_name = "Data/Harmonized", transaction_id = up$transaction_id)
primus_upload("FDP-harmonized-data", survey_id, type = "harmonized",
              infile = "Stata/COL_2023_GEIH_V01_M_V01_A_FDP_WELF.do",
              folder_name = "Programs", transaction_id = up$transaction_id)

# 5. Confirm the draft transaction once all files are attached
primus_confirm(up$transaction_id, comments = "harmonized data ready for review")

local surveyid "COL_2023_GEIH_V01_M_V01_A_FDP"

* 1. Validate: any schema violation stops before upload
use "data/processed/`surveyid'_WELF.dta", clear
do "Stata/validate_50by35.do"

* 2. CPI / 2021 PPP conversion factors from PIP (see the full script for
*    the lookup; pip tables, table(cpi) / table(ppp))

* 3. Build the indicator XML — see Stata/03_upload_harmonized.do, which
*    follows the PRIMUS XML schema (Request/Result/LOG_DETAIL sections,
*    key;value CDATA blocks) and computes the self-reliance share at
*    $3.00/day 2021 PPP (placeholder, pending confirmation from the
*    50by35 methodology team)

* 4. The XML opens the transaction (xmlbl flag); the transaction ID is
*    returned in r(prmTransId). Then attach the data and programs.
primus upload, processid(40) surveyid(`surveyid') ///
    type(harmonized) infile("`surveyid'.xml") xmlbl new
local tranxid = r(prmTransId)

primus upload, proc(40) surveyid(`surveyid') type(harmonized) ///
    folderpath(Data/Harmonized) infile("data/processed/`surveyid'_WELF.dta") tranxid(`tranxid')

primus upload, proc(40) surveyid(`surveyid') type(harmonized) ///
    folderpath(Programs) infile("Stata/COL_2023_GEIH_V01_M_V01_A_FDP_WELF.do") tranxid(`tranxid')

* 5. Confirm the draft transaction (uploader step)
primus action, tranxid(`tranxid') proc(40) ///
    decision(confirm) comments(harmonized data ready for review)

4.3 Reviewing & approving via PRIMUS

Once the uploader confirms the transaction to the harmonized collection, it is queued for review by users with approver status for the FDP-harmonized-data process. For the harmonized data, first regional approvers review the transaction before it goes to global approvers. All approvers can approve or reject the transaction. If there is no consensus by global approvers, a user with “finalizer” status can make the final decision to approve or reject.

If the transaction is rejected, the uploader is notified. If approved at all levels, the files are published to datalibweb for 50by35 monitoring.

Note

A transaction cannot be approved or rejected more than once, and the action cannot be changed once it goes through.

R
Stata

library(primus)

# Check pending transactions for review
primus_query_transactions("FDP-harmonized-data", status = "pending")

primus_transaction_files(transaction_id)

# Approve (or reject) the transaction
primus_approve("040-000327173-LACHARM-COL-12945",
               comments = "harmonized data approved for publication")
# primus_reject("040-...", comments = "<what must be fixed>")  # comments required


* Check pending transactions for review
primus query, process(40) overallstatus(PENDING)

* list the attached files and review the indicator values
primus download, processid(40) tranxid(040-000327173-LACHARM-COL-12945) filelist
primus download, processid(40) tranxid(040-000327173-LACHARM-COL-12945) indicator

* download the indicator XML itself
primus download, xml processid(40) tranxid(040-000327173-LACHARM-COL-12945) ///
    outfile(indicator.xml)

* Approve a transaction
primus action, tranxid(040-000327173-LACHARM-COL-12945) processid(40) ///
    decision(approve) comments(harmonized data approved for publication)

--- title: "Upload Harmonized Data" date: today --- ## Overview Harmonized microdata and harmonization scripts are uploaded to **datalibweb** via **PRIMUS**. After review, approved files are published to datalibweb for 50by35 monitoring. ## Uploading via PRIMUS Harmonized files are uploaded as a PRIMUS **transaction** using the `FDP-harmonized-data` process (`processid = 40`) by users with uploader status. ### Validation checks All harmonized files must pass validation checks before they can be uploaded to datalibweb. These confirm that the harmonized file conforms to the schema, and that mandatory variables are present and non-missing. Any deviation from the schema will trigger an error and prevent the harmonized file from being uploaded to datalibweb. They check for missing values in mandatory variables, for negative or zero values in `welfare` and `weight`, and that `welfare_self` is not greater than `welfare`. Households where `hhsize` differs from the number of `pid` records are flagged with a warning (rosters can be incomplete, and refugee-only samples keep a subset of household members). The validation scripts are `Stata/validate_50by35.do` and `R/validate_50by35.R` in this repository. ### XML file A harmonized data upload starts with an **indicator `.xml`** file that opens the transaction and returns a transaction ID. The XML contains metadata about the harmonized file, the poverty headcount from `welfare`, and the **50by35 self-reliance share** computed from `welfare_self` at the $3.00/day 2021 PPP line (**placeholder** — pending confirmation from the 50by35 methodology team). The XML's content can be customized, provided it follows the **PRIMUS XML schema for harmonized data**: a `<Request>` section (`RequestKey`, `welfare`, `weight`, `By`, `N_By_Group`, `nParamSets`, plus a `key;value` CDATA block of run metadata), a `<Result>` section (one `<Welfare>` per welfare variable, each containing a `<ByGroup>` with a `<DATASUMMARY>` and one `<CALCULATION>` per `nParamSets` — also `key;value` CDATA blocks), and a `<LOG_DETAIL>` free-text block that records calculation details but is not shown in the approval table. Both upload scripts in this repository write the same document, with two `CALCULATION` entries per group: the poverty headcount (from `welfare`) and the 50by35 self-reliance share (from `welfare_self`). ### CPI and PPP conversion factors {#sec-cpi-ppp} The conversion factors used to calculate the indicator for the XML file come from the [World Bank PIP](https://pip.worldbank.org) auxiliary tables, pinned to the **2021 PPP framework** (R: `pipr::get_aux("cpi", ppp_version = 2021)`; Stata: `pip tables, table(cpi) ppp_year(2021) clear`): - **PPP** — the ICP 2021 purchasing-power-parity conversion factor (LCU per 2021 international $), one value per country. - **CPI** — PIP's consumer price index, normalized so that CPI $= 1$ in 2021 and aligned to each survey's fieldwork period, one value per country-year. They enter the self-reliance indicator exactly as in the [formula on the home page](../index.qmd): `welfare_self` (annual per capita LCU at survey-year prices) is divided by 365 to get a daily value, by the survey-year CPI to convert to 2021 prices, and by the PPP factor to convert to 2021 PPP dollars, then compared with $z = \$3.00$ (**placeholder** value, pending confirmation from the 50by35 methodology team). ::: {.callout-warning} ## Surveys not in PIP PIP's CPI table only contains values for country-years with a PIP survey. For refugee surveys in years not covered by PIP, the upload scripts apply an automatic **CPI fallback** rather than stopping with an error: 1. **Interpolation** — if the survey year falls between two years that do have CPI values, the scripts linearly interpolate between them. For example, Uganda 2018 is bracketed by the 2016 and 2019 PIP survey years, giving an interpolated value of $0.875 + \tfrac{2}{3}(0.967 - 0.875) \approx 0.936$. 2. **Nearest year** — if the survey year is beyond the range of available PIP CPI values (e.g. a 2025 survey when PIP's latest value is 2023), the most recent available value is used as-is (flat extrapolation). The fallback method is recorded in the indicator XML's `Method` field (e.g. `Interpolated(CPI_2016=0.874967,CPI_2019=0.967133)` or `NearestYear(CPI_2023=1.140000)`) so reviewers can see exactly how the CPI was derived. If neither fallback is acceptable, set `cpi_override` in the upload script to the ratio $CPI_{t}/CPI_{2021}$ from the national CPI series (e.g. WDI series `FP.CPI.TOTL`). The override is recorded as `Method;ManualOverride` in the XML. ::: For convenience, the validation checks, XML creation and upload steps are combined in one script per client — `Stata/03_upload_harmonized.do` and `R/03_upload_harmonized.R` in this repository — condensed below. After uploading, the uploader confirms the draft transaction. Only then does it become visible to approvers. ::: {.callout-tip} ## Include the harmonization scripts Upload the code used to harmonize the raw data together with the harmonized dataset, so the harmonization is fully reproducible from source. Add them as additional files on the same transaction ID, using the folder path matching the file type: | Folder path | Extensions | |---|---| | `Programs` | `do`, `py`, `r` | | `Data/Harmonized` | `dta` | ::: Note - the examples below are summaries, use the full scripts in this repository. ::: {.panel-tabset group="language"} ## R ```r library(primus) library(pipr) library(haven) source("R/validate_50by35.R") survey_id <- "COL_2023_GEIH_V01_M_V01_A_FDP" # 1. Validate: any schema violation stops before upload df <- read_dta(paste0("data/processed/", survey_id, "_WELF.dta")) validate_50by35(df) ccode <- df$code[1] yr <- df$year[1] # 2. CPI / PPP conversion factors from PIP, pinned to the 2021 framework. # If no exact CPI match exists for this country-year, the script # automatically falls back to linear interpolation between the two # nearest PIP survey years (or the nearest single year at the edges). # See @sec-cpi-ppp and R/03_upload_harmonized.R for details. cpi_table <- get_aux("cpi", ppp_version = 2021) cpi_exact <- cpi_table[cpi_table$country_code == ccode & as.integer(as.character(cpi_table$year)) == yr & cpi_table$data_level == "national", ] # exact match; for fallback logic see R/03_upload_harmonized.R cpi_value <- cpi_exact$value ppp_table <- get_aux("ppp", ppp_version = 2021) icp_value <- ppp_table$value[ppp_table$country_code == ccode & ppp_table$data_level == "national"][1] # 3. Compute the self-reliance share and write the indicator XML — # see R/03_upload_harmonized.R for the XML writer, which follows the # PRIMUS XML schema (Request/Result/LOG_DETAIL sections, key;value # CDATA blocks, one CALCULATION per nParamSets). # 3.00 is a placeholder PPP line, pending confirmation from the # 50by35 methodology team. sr <- weighted.mean(df$welfare_self / 365 / cpi_value / icp_value >= 3.00, df$weight) # 4. The XML opens the transaction; then attach the data and programs up <- primus_upload("FDP-harmonized-data", survey_id, type = "harmonized", infile = "indicator.xml", xml = "indicator.xml") primus_upload("FDP-harmonized-data", survey_id, type = "harmonized", infile = paste0("data/processed/", survey_id, "_WELF.dta"), folder_name = "Data/Harmonized", transaction_id = up$transaction_id) primus_upload("FDP-harmonized-data", survey_id, type = "harmonized", infile = "Stata/COL_2023_GEIH_V01_M_V01_A_FDP_WELF.do", folder_name = "Programs", transaction_id = up$transaction_id) # 5. Confirm the draft transaction once all files are attached primus_confirm(up$transaction_id, comments = "harmonized data ready for review") ``` ## Stata ```stata local surveyid "COL_2023_GEIH_V01_M_V01_A_FDP" * 1. Validate: any schema violation stops before upload use "data/processed/`surveyid'_WELF.dta", clear do "Stata/validate_50by35.do" * 2. CPI / 2021 PPP conversion factors from PIP (see the full script for * the lookup; pip tables, table(cpi) / table(ppp)) * 3. Build the indicator XML — see Stata/03_upload_harmonized.do, which * follows the PRIMUS XML schema (Request/Result/LOG_DETAIL sections, * key;value CDATA blocks) and computes the self-reliance share at * $3.00/day 2021 PPP (placeholder, pending confirmation from the * 50by35 methodology team) * 4. The XML opens the transaction (xmlbl flag); the transaction ID is * returned in r(prmTransId). Then attach the data and programs. primus upload, processid(40) surveyid(`surveyid') /// type(harmonized) infile("`surveyid'.xml") xmlbl new local tranxid = r(prmTransId) primus upload, proc(40) surveyid(`surveyid') type(harmonized) /// folderpath(Data/Harmonized) infile("data/processed/`surveyid'_WELF.dta") tranxid(`tranxid') primus upload, proc(40) surveyid(`surveyid') type(harmonized) /// folderpath(Programs) infile("Stata/COL_2023_GEIH_V01_M_V01_A_FDP_WELF.do") tranxid(`tranxid') * 5. Confirm the draft transaction (uploader step) primus action, tranxid(`tranxid') proc(40) /// decision(confirm) comments(harmonized data ready for review) ``` ::: ## Reviewing & approving via PRIMUS Once the uploader confirms the transaction to the harmonized collection, it is queued for review by users with approver status for the `FDP-harmonized-data` process. For the harmonized data, first regional approvers review the transaction before it goes to global approvers. All approvers can approve or reject the transaction. If there is no consensus by global approvers, a user with "finalizer" status can make the final decision to approve or reject. If the transaction is rejected, the uploader is notified. If approved at all levels, the files are published to datalibweb for 50by35 monitoring. ::: {.callout-warning} ## Note A transaction cannot be approved or rejected more than once, and the action cannot be changed once it goes through. ::: ::: {.panel-tabset group="language"} ## R ```r library(primus) # Check pending transactions for review primus_query_transactions("FDP-harmonized-data", status = "pending") primus_transaction_files(transaction_id) # Approve (or reject) the transaction primus_approve("040-000327173-LACHARM-COL-12945", comments = "harmonized data approved for publication") # primus_reject("040-...", comments = "<what must be fixed>") # comments required ``` ## Stata ```stata * Check pending transactions for review primus query, process(40) overallstatus(PENDING) * list the attached files and review the indicator values primus download, processid(40) tranxid(040-000327173-LACHARM-COL-12945) filelist primus download, processid(40) tranxid(040-000327173-LACHARM-COL-12945) indicator * download the indicator XML itself primus download, xml processid(40) tranxid(040-000327173-LACHARM-COL-12945) /// outfile(indicator.xml) * Approve a transaction primus action, tranxid(040-000327173-LACHARM-COL-12945) processid(40) /// decision(approve) comments(harmonized data approved for publication) ``` :::