8.1 Interacting with PRIMUS

The interaction with PRIMUS is done through different systems, so it is best to begin by clarifying terms.

net install github, from("https://haghish.github.io/github/")

github install worldbank/primus

8.2 Understanding PRIMUS

Each time a database is uploaded into PRIMUS, it is assigned a transaction ID. During the uploading process (or right after it has finished), the three parties–DECDG, DECRG, or the Poverty GP–evaluate the quality of the new or corrected data and approve them or reject them in the system. Depending on the decision of all the parties, each transaction will take one of three possible status, pending, approved, or rejected.

The transaction ID is pending when at least one of the three parties (DECDG, DECRG, or the Poverty GP) has not approved it in the system. You can click on the check box PENDING in the PRIMUS website to see which surveys have such a status, or you can use the primus command list this,

qui primus query, overallstatus(PENDING)
list transaction_id country year date_modified in 1/`=min(10, _N)'
     +----------------------------------------------+
     |              transaction_id   country   year |
     |----------------------------------------------|
  1. | TRN-000327173-EAP-IDN-QR48Q       IDN   2017 |
  2. | TRN-000327173-ECA-DEU-YJYVZ       DEU   1995 |
  3. | TRN-000327173-ECA-DEU-2P4DR       DEU   2002 |
  4. | TRN-000327173-ECA-DEU-LJN8R       DEU   2003 |
  5. | TRN-000327173-ECA-DEU-ZSN9J       DEU   2005 |
     |----------------------------------------------|
  6. | TRN-000327173-ECA-DEU-UBS7M       DEU   2008 |
  7. | TRN-000327173-ECA-DEU-41TOU       DEU   2009 |
  8. | TRN-000327173-EAP-AUS-KKZ2E       AUS   2004 |
     +----------------------------------------------+

Notice that the overall status of a transaction is independent from survey ID. Thus, it is possible to find several transactions for the same country and year. Indonesia 2017, for instance, has three transactions, two of them rejected and one of them pending.

qui primus query, country(IDN) year(2017)
list transaction_id overall_status date_modified in 1/`=min(10, _N)'
     +--------------------------------------------------+
     |              transaction_id        date_modified |
     |--------------------------------------------------|
  1. | TRN-000104674-EAP-IDN-8R9IF   23may2018 15:28:47 |
  2. | TRN-000327173-EAP-IDN-TYA1A   23may2018 23:57:27 |
  3. | TRN-000327173-EAP-IDN-QR48Q   24may2018 00:27:33 |
     +--------------------------------------------------+

A transaction is rejected when at least one of the three parties rejected the database. Finally, a transaction is approved only when all three parties have approved it into the system.

8.3 Checking PRIMUS estimates

The real first step to check the quality of the recently uploaded data into PRIMUS is to download the basic estimates of each data and compare them with our own. There is no need to calculate and compare all the estimates available in PRIMUS but the mean in PPP, the poverty headcount, and the Gini index.

The primus command allows us to download the estimates of each transaction, but it has to be done one by one. Fortunately, the pcn command downloads all the estimates of pending transactions for us and properly stores them in the folder p:\01.PovcalNet\03.QA\02.PRIMUS\pending\ 🎉 🎉 . You only need to type,

pcn primus pending, down(estimates)

In addition, pcn checks the date for which you’re downloading the estimates and keeps only those transactions that have been uploaded for the next spring or annual-meetings release. For instance, assume that today, 2020-11-20, you want to see the estimates of pending transactions in PRIMUS. Since annual meetings take place around September, pcn assumes you are interested in the estimates for the Spring-meetings release, around March next year. Thus, it will filter the results from primus, keeping only those transactions that were uploaded from November 2020. Now it is likely that the PRIMUS system has not been opened for uploading new data in November, as it usually opens around December and July. Thus, it is likely that you will find and error saying There is no pending data in PRIMUS for the combination of country/years selected.

You can load the recently-downloaded estimates by typing,

pcn primus pending, load(estimates)

Now, you have to check whether the new estimates make sense. Once way to that is to follow this do-file, p:\01.PovcalNet\03.QA\02.PRIMUS\pending\2020_SM\estimates\checks\comparisons_wrk_data.do.

8.4 Confirming and approving data in PRIMUS

Once you have checked that the estimates of pending transactions make sense, you need to approve them. As explained in section 8.2, the approval on PRIMUS requires the consent of three parties. The PovcalNet team had the responsibility to approve on behalf or two of them, DECDG and DECRG. This process can easily done with the code below, which can be found in this file, p:\01.PovcalNet\03.QA\02.PRIMUS\pending\2020_SM\approve\primus_approve.do.

/*==================================================
0: Program set up
==================================================*/
version 14
drop _all

*---------- Modify this
local excl = "BRA SOM SSD" // countries to exclude 
local excl = "" // countries to exclude 

/*==================================================
Load data
==================================================*/

primus query, overalls(pending)
//------------Cut off date
local filtdate = "2019-12-01" // filter date (december last year)
local filtdate = "2020-02-18" // filter date (surveys uploaded by Minh)
keep if  date_modified >= clock("`filtdate'", "YMD")

//------------Select username
if (lower("`c(username)'") == "wb424681") {
  local dep = "povcalnet"
}
else if (lower("`c(username)'") == "wb384996") {
  local dep = "decdg"
}
else {
  noi disp in red "you don't have rights to run this code"
  break
}

tab `dep'
keep if `dep' == "PENDING"

if ("`excl'" != "") {
  local excl: subinstr local excl " " "|", all
  drop if regexm("`country'", "`excl'") 
}

/*=================================================
Approve (Do NOT modify)
==================================================*/

local n = _N
preserve 
qui foreach i of numlist 1/`n' {
  restore, preserve
  local country = country[`i']
  local year    = year[`i']
  local id      = transaction_id[`i']
  
  noi disp in y "primus action, tranxid(`id') decision(approved)"
  cap noi primus action, tranxid(`id') decision(approved)
  if (_rc) noi disp "problem with `id'"
}

Basically, this is what you need to do with this file.

1. Modify local excl in case you do not want to approve one or several countries.
2. Modify local filtdate in which you select the date from which you want to approve transactions.
3. Make sure at least two people approve. One on behalf of “povcalnet” (which is the alias used for DECRG) and another on behalf of “decdg.”
4. PRIMUS has a double-confirmation process, so you need to “confirm” and then “approve” the transaction. For that, you only need to change the option decision() from approved to confirmed.

For some unknown reason, the PRIMUS system did not accept the approval of some transactions. If this happens again, you need to talk to Minh Cong Nguyen, so he can do the approval manually.