2 Upload Raw Data

Published

July 20, 2026

2.1 Overview

Before a survey can be harmonized, its raw microdata should exist on datalibweb. If a raw dataset is not already available there (e.g. on the GMD or region server), it must first be uploaded through PRIMUS.

Check first

Query datalibweb to confirm whether the raw survey is already present before starting a new upload transaction.

2.2 Uploading via PRIMUS

Raw files are uploaded as a PRIMUS transaction using the FDPRaw-data process (processid = 39) by users with uploader status. A raw data upload starts with a new file that returns a transaction ID; additional files are then attached to that same transaction ID. Specify the destination folderpath and type(raw).

Include supporting documents

Upload the survey documentation and questionnaires together with the raw data, so the source instruments travel with the microdata on datalibweb. Add them as additional files on the same transaction ID, using the folder path matching the file type:

Folder path	Extensions
`Doc/Questionnaires`	`doc`, `docx`, `pdf`, `rar`, `xls`, `xlsx`, `zip`
`Data/Stata`	`dta`

Users can interact with PRIMUS via the web interface or programmatically using the primus Stata package (Windows) or the primus R package (cross-platform). Both call the same PRIMUS API; use whichever matches your harmonization workflow.

2.2.1 Authentication

Both clients use the same datalibweb token (valid 30 days, renewed on the datalibweb website).

R
Stata

primus_set_token("<your datalibweb token>")

primus register, token(<your datalibweb token>)

R
Stata

library(primus)

# Discover the processes you can act on
primus_list_processes()

# What folders / extensions does the process accept?
primus_folders(process_name = "FDPRaw-data")

# Upload a raw data file (creates a new transaction)
up <- primus_upload(
  process_name = "FDPRaw-data",
  survey_id    = "COL_2023_GEIH_V01_M",
  type         = "raw",
  infile       = "household_data_2023.dta",
  folder_name  = "Data/Stata"
)

# Add additional files to the same transaction
primus_upload(
  process_name   = "FDPRaw-data",
  survey_id      = "COL_2023_GEIH_V01_M",
  type           = "raw",
  infile         = "individual_data_2023.dta",
  folder_name    = "Data/Stata",
  transaction_id = up$transaction_id
)

# Include the survey questionnaire/documentation on the same transaction
primus_upload(
  process_name   = "FDPRaw-data",
  survey_id      = "COL_2023_GEIH_V01_M",
  type           = "raw",
  infile         = "questionnaire.pdf",
  folder_name    = "Doc/Questionnaires",
  transaction_id = up$transaction_id
)

# Alternatively, upload a zip that mirrors the process folder structure
# (omitting transaction_id starts a new transaction)
primus_upload(
  process_name = "FDPRaw-data",
  survey_id    = "COL_2023_GEIH_V01_M",
  type         = "raw",
  infile       = "COL_2023_GEIH_V01_M.zip",
  zip          = TRUE
)

# Confirm the draft transaction once all files are attached
primus_confirm(up$transaction_id, comments = "raw data ready for review")

* Check the processes you can act on (lists all processes and their metadata)
primus download, meta

* What folders / extensions does the process accept?
primus download, processid(39) folders

* Upload a raw data file as a NEW transaction; the transaction ID is
* returned in r(prmTransId)
primus upload, processid(39) surveyid(COL_2023_GEIH_V01_M) ///
    type(raw) folderpath(Data/Stata) infile(c:\Temp\household_data_2023.dta) new
local tranxid = r(prmTransId)

* Add other files to the SAME transaction ID
primus upload, proc(39) surveyid(COL_2023_GEIH_V01_M) ///
    type(raw) folderpath(Data/Stata) infile(c:\Temp\individual_data_2023.dta) ///
    tranxid(`tranxid')

* Include the survey questionnaire/documentation on the same transaction
primus upload, proc(39) surveyid(COL_2023_GEIH_V01_M) ///
    type(raw) folderpath(Doc/Questionnaires) infile(c:\Temp\questionnaire.pdf) ///
    tranxid(`tranxid')

* Alternatively, upload a zip that mirrors the process folder structure
primus upload, processid(39) surveyid(COL_2023_GEIH_V01_M) ///
    type(raw) infile(c:\Temp\COL_2023_GEIH_V01_M.zip) zip new

* Confirm the draft transaction (uploader step)
primus action, tranxid(`tranxid') proc(39) ///
    decision(confirm) comments(raw data ready for review)

2.3 Reviewing & approving via PRIMUS

Once the uploader confirms the transaction, it is queued for review by users with approver status for the FDPRaw-data process. Approvers can approve or reject the transaction. If approved, the files are published to datalibweb (and become available for harmonization). If rejected, the uploader is notified and can revise the files and resubmit.

The raw files become the input to the harmonization step.

R
Stata

library(primus)

# Check pending transactions for review
primus_query_transactions("FDPRaw-data", status = "pending")

# inspect and download the files for review
primus_transaction_files("039-000327173-LACRAW-COL-12944")

# Approve (or reject) the transaction
primus_approve("039-000327173-LACRAW-COL-12944",
               comments = "raw data approved for harmonization")
# primus_reject("039-...", comments = "<what must be fixed>")  # comments required


* Check pending transactions for review
primus query, process(39) overallstatus(PENDING)

* list the files attached to a transaction
* (the Stata client lists files but does not download the data itself —
* use the PRIMUS web interface or the R client to inspect file contents)
primus download, processid(39) tranxid(039-000327173-LACRAW-COL-12944) filelist

* Approve a transaction
primus action, tranxid(039-000327173-LACRAW-COL-12944) processid(39) ///
    decision(approve) comments(raw data approved for harmonization)

--- title: "Upload Raw Data" date: today --- ## Overview Before a survey can be harmonized, its raw microdata should exist on **datalibweb**. If a raw dataset is not already available there (e.g. on the GMD or region server), it must first be uploaded through **PRIMUS**. ::: {.callout-tip} ## Check first Query datalibweb to confirm whether the raw survey is already present before starting a new upload transaction. ::: ## Uploading via PRIMUS Raw files are uploaded as a PRIMUS **transaction** using the `FDPRaw-data` process (`processid = 39`) by users with uploader status. A raw data upload starts with a `new` file that returns a transaction ID; additional files are then attached to that same transaction ID. Specify the destination `folderpath` and `type(raw)`. ::: {.callout-tip} ## Include supporting documents Upload the survey **documentation and questionnaires** together with the raw data, so the source instruments travel with the microdata on datalibweb. Add them as additional files on the same transaction ID, using the folder path matching the file type: | Folder path | Extensions | |---|---| | `Doc/Questionnaires` | `doc`, `docx`, `pdf`, `rar`, `xls`, `xlsx`, `zip` | | `Data/Stata` | `dta` | ::: Users can interact with PRIMUS via the [web interface](https://primusuiprod.ase.worldbank.org) or programmatically using the [`primus` Stata package](https://github.com/worldbank/primus-stata) (Windows) or the [`primus` R package](https://github.com/worldbank/primus-r) (cross-platform). Both call the same PRIMUS API; use whichever matches your harmonization workflow. ### Authentication Both clients use the same **datalibweb token** (valid 30 days, renewed on the [datalibweb website](https://datalibweb2.worldbank.org)). ::: {.panel-tabset group="language"} ## R ```r primus_set_token("<your datalibweb token>") ``` ## Stata ```stata primus register, token(<your datalibweb token>) ``` ::: ::: {.panel-tabset group="language"} ## R ```r library(primus) # Discover the processes you can act on primus_list_processes() # What folders / extensions does the process accept? primus_folders(process_name = "FDPRaw-data") # Upload a raw data file (creates a new transaction) up <- primus_upload( process_name = "FDPRaw-data", survey_id = "COL_2023_GEIH_V01_M", type = "raw", infile = "household_data_2023.dta", folder_name = "Data/Stata" ) # Add additional files to the same transaction primus_upload( process_name = "FDPRaw-data", survey_id = "COL_2023_GEIH_V01_M", type = "raw", infile = "individual_data_2023.dta", folder_name = "Data/Stata", transaction_id = up$transaction_id ) # Include the survey questionnaire/documentation on the same transaction primus_upload( process_name = "FDPRaw-data", survey_id = "COL_2023_GEIH_V01_M", type = "raw", infile = "questionnaire.pdf", folder_name = "Doc/Questionnaires", transaction_id = up$transaction_id ) # Alternatively, upload a zip that mirrors the process folder structure # (omitting transaction_id starts a new transaction) primus_upload( process_name = "FDPRaw-data", survey_id = "COL_2023_GEIH_V01_M", type = "raw", infile = "COL_2023_GEIH_V01_M.zip", zip = TRUE ) # Confirm the draft transaction once all files are attached primus_confirm(up$transaction_id, comments = "raw data ready for review") ``` ## Stata ```stata * Check the processes you can act on (lists all processes and their metadata) primus download, meta * What folders / extensions does the process accept? primus download, processid(39) folders * Upload a raw data file as a NEW transaction; the transaction ID is * returned in r(prmTransId) primus upload, processid(39) surveyid(COL_2023_GEIH_V01_M) /// type(raw) folderpath(Data/Stata) infile(c:\Temp\household_data_2023.dta) new local tranxid = r(prmTransId) * Add other files to the SAME transaction ID primus upload, proc(39) surveyid(COL_2023_GEIH_V01_M) /// type(raw) folderpath(Data/Stata) infile(c:\Temp\individual_data_2023.dta) /// tranxid(`tranxid') * Include the survey questionnaire/documentation on the same transaction primus upload, proc(39) surveyid(COL_2023_GEIH_V01_M) /// type(raw) folderpath(Doc/Questionnaires) infile(c:\Temp\questionnaire.pdf) /// tranxid(`tranxid') * Alternatively, upload a zip that mirrors the process folder structure primus upload, processid(39) surveyid(COL_2023_GEIH_V01_M) /// type(raw) infile(c:\Temp\COL_2023_GEIH_V01_M.zip) zip new * Confirm the draft transaction (uploader step) primus action, tranxid(`tranxid') proc(39) /// decision(confirm) comments(raw data ready for review) ``` ::: ## Reviewing & approving via PRIMUS Once the uploader confirms the transaction, it is queued for review by users with approver status for the `FDPRaw-data` process. Approvers can approve or reject the transaction. If approved, the files are published to datalibweb (and become available for harmonization). If rejected, the uploader is notified and can revise the files and resubmit. The raw files become the input to the [harmonization step](02-harmonization.qmd). ::: {.panel-tabset group="language"} ## R ```r library(primus) # Check pending transactions for review primus_query_transactions("FDPRaw-data", status = "pending") # inspect and download the files for review primus_transaction_files("039-000327173-LACRAW-COL-12944") # Approve (or reject) the transaction primus_approve("039-000327173-LACRAW-COL-12944", comments = "raw data approved for harmonization") # primus_reject("039-...", comments = "<what must be fixed>") # comments required ``` ## Stata ```stata * Check pending transactions for review primus query, process(39) overallstatus(PENDING) * list the files attached to a transaction * (the Stata client lists files but does not download the data itself — * use the PRIMUS web interface or the R client to inspect file contents) primus download, processid(39) tranxid(039-000327173-LACRAW-COL-12944) filelist * Approve a transaction primus action, tranxid(039-000327173-LACRAW-COL-12944) processid(39) /// decision(approve) comments(raw data approved for harmonization) ``` :::