Updating the “SurveyMean” sheet

The first step is to extract some important metadata information from the .rds files generated in the previous step. This is done with the following code,

lf <- 
  update_master_info(df       = pcn, 
                    countries = countries,
                    years     = years)
st <- lf$s
df <- lf$df

table(st$status)
filter(st, status != "OK")

The object lf is a list with two objects, st and df. Object st is merely the status of each survey in the update_master_info process, whereas df is the actual metadata of the new data. Now, the following code loads the data in the most recent version of the Master file,

lmf     <- load_masterfile()
mf      <- lmf$data$SurveyMean

reg_ctry <- lmf$data$CountryList %>% 
  select(
    Region      = WBRegionCode, 
    CountryCode = CountryCode, 
    countryName = CountryName
  )

The function load_masterfile returns a list that is then bound to the name lmf (this function takes a while to run especially if you’re working remotely). The main object of lmf is another list, data, that contains a data frame per sheet. So, in the code above, you’re creating object mf with the “SurveyMean” and red_ctry, with the “CountryList” sheet. The reason why we load the whole master file is that, when we create a new version we want that version to include all the sheets, even those that were not modified. Function writexl::write_xls, which is the function that saves the new version of the master file, uses a list with all the sheets to save the file.

Now, we need to organize the new information into the “SurveyMean” format. This is done with function survey_mean_info. However, and this one of the tricky parts, we need to make sure that the five objectives above are met when we include the information of the new data. The survey_mean_info function handles all the generic cases, but there are some cases that need to be removed manually. This is why we have the object condition that goes directly as one of the arguments of survey_mean_info. If it is necessary to manage any special cases, this object should be set to an empty string, condition <- "". Let’s see some examples of real cases in which we had to use the object condition.

condition <- '!(countrycode == "IND" & year == 2012)'

In this case, we needed to remove the observation for India 2012, because even though the CPI_Time variable in the master file is 2012, the SurveyTime variable is 2011.5.

condition <- '!(countrycode == "CHN" & grepl("A$", module))'

Here we needed to remove all the observations of China for which the module finished in a letter A.

condition <- '!(countrycode == "CHN" & year >= 1990 & welfaretype3 == "y")'

Here, there was a problem in the metadata and we needed to remove all the observations for China after 1990 for which the welfare type was coded as income, when in reality it was consumption.

condition <- paste0('!(countrycode  %in% ', 
                    deparse(countries),
                    ' & year  %in%  ', 
                    deparse(years),')'
                    )

This final example is a general form in which we remove old observations for all the countries and years for which there is new data. This is very useful if we want to start a country from scratch.

Now, we just need to identify the data that remains unchanged in the “SurveyMean” sheet using the function unchanged_data() and then append together the new data, dfn, and the unchanged data. Finally, we update the Master file using the function update_master_file, which receives four arguments, lmf, the current version of the master file with all its sheets; vintage, the vintage control sheet which is loaded separately and it is useful only for institutional-memory purposes; new_mf, which is the new “SurveyMean” sheet with unchanged and new data; and mdir, which is the directory of the master file.

You should be fine if you execute all these steps one by one, while checking the intermediate outputs.