diff --git a/docs/articles/vignette_preparing_data.html b/docs/articles/vignette_preparing_data.html index f79c9dc3..1bf570c3 100644 --- a/docs/articles/vignette_preparing_data.html +++ b/docs/articles/vignette_preparing_data.html @@ -113,7 +113,7 @@

Preparing data for gcamfaostat

-

2024-03-16

+

2024-04-04

Source: vignettes/vignette_preparing_data.Rmd @@ -139,11 +139,16 @@

IntroductionPrebuilt -Data of the package. * The package can be run with using those data, -but make sure Process_Raw_FAO_Data <- FALSE in -constants.R. * The prebuilt data were generated using -FAOSTAT data archived in Zenodo address specified in the -FF_download_RemoteArchive function (Zhao 2023).

+Data of the package.

+

1. Get the latest FAOSTAT Metadata

@@ -225,10 +230,10 @@

2. Generate the me -PD +FBSH -Prices: Deflators +Food Balances: Food Balances (-2013, old methodology and population) TRUE @@ -237,18 +242,18 @@

2. Generate the me TRUE -2023-11-07 +2023-03-10 -1MB +69MB -FBS +OA -Food Balances: Food Balances (2010-) +Population and Employment: Annual population TRUE @@ -257,18 +262,18 @@

2. Generate the me TRUE -2023-09-04 +2022-10-24 -55MB +2MB -FBSH +FO -Food Balances: Food Balances (-2013, old methodology and population) +Forestry: Forestry Production and Trade TRUE @@ -277,18 +282,18 @@

2. Generate the me TRUE -2023-03-10 +2023-12-21 -69MB +15MB -FO +QCL -Forestry: Forestry Production and Trade +Production: Crops and livestock products TRUE @@ -297,18 +302,18 @@

2. Generate the me TRUE -2023-12-21 +2023-12-18 -15MB +32MB -RFN +PD -Land, Inputs and Sustainability: Fertilizers by Nutrient +Prices: Deflators TRUE @@ -317,18 +322,18 @@

2. Generate the me TRUE -2023-07-05 +2023-11-07 -2MB +1MB -RL +TCL -Land, Inputs and Sustainability: Land Use +Trade: Crops and livestock products TRUE @@ -337,18 +342,18 @@

2. Generate the me TRUE -2024-02-15 +2023-12-13 -2MB +244MB -OA +FBS -Population and Employment: Annual population +Food Balances: Food Balances (2010-) TRUE @@ -357,18 +362,18 @@

2. Generate the me TRUE -2022-10-24 +2023-09-04 -2MB +55MB -PP +RFN -Prices: Producer Prices +Land, Inputs and Sustainability: Fertilizers by Nutrient TRUE @@ -377,18 +382,18 @@

2. Generate the me TRUE -2023-12-19 +2023-07-05 -10MB +2MB -QCL +RL -Production: Crops and livestock products +Land, Inputs and Sustainability: Land Use TRUE @@ -397,18 +402,18 @@

2. Generate the me TRUE -2023-12-18 +2024-02-15 -32MB +2MB -SCL +PP -Food Balances: Supply Utilization Accounts (2010-) +Prices: Producer Prices TRUE @@ -417,18 +422,18 @@

2. Generate the me TRUE -2023-09-04 +2023-12-19 -72MB +10MB -TCL +SCL -Trade: Crops and livestock products +Food Balances: Supply Utilization Accounts (2010-) TRUE @@ -437,10 +442,10 @@

2. Generate the me TRUE -2023-12-13 +2023-09-04 -244MB +72MB @@ -491,18 +496,18 @@

3. Download FAOSTAT raw datagcamfaostat_metadata(OnlyReturnDatasetCodeRequired = T).

Example:

-# Dataset PP, producer prices, is downloaded from RemoteArchiveURL to DATA_FOLDER
-# RemoteArchiveURL = "https://zenodo.org/record/8260225/files/"
-# DATA_FOLDER = "inst/extdata/aglu/FAO/FAOSTAT"
-FF_download_RemoteArchive( 
-  DATASETCODE = "PP",   
-  OverWrite = TRUE     # overwrite existing PP dataset 
-  )
-
-# Dataset OA, population, is downloaded from FAOSTAT to DATA_FOLDER
-FF_download_FAOSTAT(DATASETCODE = "OA", OverWrite = TRUE)
-
-# Note that single DATASETCODE is allowed in both functions.
+# Dataset PP, producer prices, is downloaded from RemoteArchiveURL to DATA_FOLDER +# RemoteArchiveURL = "https://zenodo.org/record/8260225/files/" +# DATA_FOLDER = "inst/extdata/aglu/FAO/FAOSTAT" +FF_download_RemoteArchive( + DATASETCODE = "PP", + OverWrite = TRUE # overwrite existing PP dataset + ) + +# Dataset OA, population, is downloaded from FAOSTAT to DATA_FOLDER +FF_download_FAOSTAT(DATASETCODE = "OA", OverWrite = TRUE) + +# Note that single DATASETCODE is allowed in both functions.

4. Check local raw data info @@ -518,11 +523,11 @@

4. Check local raw data info

Example:

-# Provide detailed metadata of "PP" and "OA" in DATA_FOLDER ("inst/extdata/aglu/FAO/FAOSTAT")
-FF_rawdata_info(DATASETCODE = c("PP", "OA"), DOWNLOAD_NONEXIST = FALSE)
-
-# If "PP" or "OA" does not exist, download from remote archive
-FF_rawdata_info(DATASETCODE = c("PP", "OA"), DOWNLOAD_NONEXIST = TRUE, FAOSTAT_or_Archive = "Archive")
+# Provide detailed metadata of "PP" and "OA" in DATA_FOLDER ("inst/extdata/aglu/FAO/FAOSTAT") +FF_rawdata_info(DATASETCODE = c("PP", "OA"), DOWNLOAD_NONEXIST = FALSE) + +# If "PP" or "OA" does not exist, download from remote archive +FF_rawdata_info(DATASETCODE = c("PP", "OA"), DOWNLOAD_NONEXIST = TRUE, FAOSTAT_or_Archive = "Archive")

5. Load raw data into package @@ -536,8 +541,8 @@

5. Load raw data into package
-# Read raw data of "PP" and "OA" from DATA_FOLDER ("inst/extdata/aglu/FAO/FAOSTAT") to .GlobalEnv
-FAOSTAT_load_raw_data(DATASETCODE = c("PP", "OA"), .Envir = .GlobalEnv)

+# Read raw data of "PP" and "OA" from DATA_FOLDER ("inst/extdata/aglu/FAO/FAOSTAT") to .GlobalEnv +FAOSTAT_load_raw_data(DATASETCODE = c("PP", "OA"), .Envir = .GlobalEnv)

6. Update the Prebulit data diff --git a/docs/articles/vignette_processing_flow.html b/docs/articles/vignette_processing_flow.html index ed687a68..46516db4 100644 --- a/docs/articles/vignette_processing_flow.html +++ b/docs/articles/vignette_processing_flow.html @@ -112,7 +112,7 @@

gcamfaostat data processing

-

2024-03-16

+

2024-04-04

Source: vignettes/vignette_processing_flow.Rmd @@ -210,87 +210,95 @@

Key functionsinfo -* The function returns information of an object, including name, -metadata information, precursors and dependents.

-

dstrace -* The function is able to trace data flows by providing precursors and -dependents of an object recursively.

-

load_from_cache -* If a drake cache is available, e.g., when driver_drake() -had been run, this function, if given a list of object names, loads the -objects from the cache into a list of data frames.
-* The function get_data_list -can be used to assign each object in the list to a data frame.

+

info

+
    +
  • The function returns information of an object, including name, +metadata information, precursors and dependents.
    • +
+

dstrace

+
    +
  • The function is able to trace data flows by providing precursors and +dependents of an object recursively.
    • +
+

load_from_cache

+
    +
  • If a drake cache is available, e.g., when +driver_drake() had been run, this function, if given a list +of object names, loads the objects from the cache into a list of data +frames.
    +
    • +
  • The function get_data_list +can be used to assign each object in the list to a data frame.
    • +

Examples 

-# Load gcamfaostat
-# devtools::load_all() 
-library(gcamfaostat)
-# package data, GCAM_DATA_MAP, will be loaded
-
-# Get information for an intermediate data frame QCL_CROP_PRIMARY
-info("QCL_CROP_PRIMARY")
-#> QCL_CROP_PRIMARY - produced by module_xfaostat_L102_ProductionArea 
-#>  FAO primary crop area and production (various)
-#>  Detailed FAO QCL data processing for crop area and production
-#>  Dependent: xfaostat_L199_GCAMDATA_FAOSTAT_CSV
-#>  Precursor: aglu/FAO/FAO_an_items_PRODSTAT
-#>  Precursor: QCL_wide
-#>  Precursor: FBS_wide
-#>  Precursor: FBSH_CB_wide
-
-# Trace all downstream data flows for SCL_wide
-dstrace("SCL_wide", direction = "downstream", recurse = T) 
-#> 1 - SCL_wide - produced by module_xfaostat_L101_RawDataPreProc3_SCL_FBS 
-#>  FAO supply utilization account dataset, 2010+, wide (tonne)
-#>  Preprocessed FAOSTAT SCL; ** PRE-BUILT **
-#>  Dependent: Bal_new_all (#2 below)
-#>  Dependent: SUA_food_macronutrient_rate (#3 below)
-#> 2 - Bal_new_all - produced by module_xfaostat_L105_DataConnectionToSUA 
-#>  Bal_new_all (Ktonne)
-#>  Preprocessed FAO SUA 2010 - 2019
-#>  Dependent: xfaostat_L199_GCAMDATA_FAOSTAT_CSV (#4 below)
-#> 4 - xfaostat_L199_GCAMDATA_FAOSTAT_CSV - produced by module_xfaostat_L199_CSVExportAgSUA 
-#>  Export CSV to DIR_OUTPUT_CSV (NA)
-#>  Export CSV
-#>  No dependents
-#> 3 - SUA_food_macronutrient_rate - produced by module_xfaostat_L106_FoodMacroNutrient 
-#>  FAO food calories and macronutrient rate (rates)
-#>  Detailed FAO food calories and macrotunitent info for 414 SUA items +
-#>  12 fish items
-#>  Dependent: xfaostat_L199_GCAMDATA_FAOSTAT_CSV (#4 above)
-
-# Get input of a module
-inputs_of("module_xfaostat_L105_DataConnectionToSUA")
-#> [1] "aglu/FAO/FAO_items"                "aglu/FAO/Mapping_FBSH_SCL_OilCake"
-#> [3] "QCL_PROD"                          "QCL_AN_LIVEANIMAL_MEATEQ"         
-#> [5] "TCL_wide"                          "TM_bilateral_wide"                
-#> [7] "FBSH_CB_wide"                      "FBS_wide"                         
-#> [9] "SCL_wide"
-
-# Get output of a module
-outputs_of("module_xfaostat_L106_FoodMacroNutrient")
-#> [1] "SUA_food_macronutrient_rate"
+# Load gcamfaostat +# devtools::load_all() +library(gcamfaostat) +# package data, GCAM_DATA_MAP, will be loaded + +# Get information for an intermediate data frame QCL_CROP_PRIMARY +info("QCL_CROP_PRIMARY") +#> QCL_CROP_PRIMARY - produced by module_xfaostat_L102_ProductionArea +#> FAO primary crop area and production (various) +#> Detailed FAO QCL data processing for crop area and production +#> Dependent: xfaostat_L199_GCAMDATA_FAOSTAT_CSV +#> Precursor: aglu/FAO/FAO_an_items_PRODSTAT +#> Precursor: QCL_wide +#> Precursor: FBS_wide +#> Precursor: FBSH_CB_wide + +# Trace all downstream data flows for SCL_wide +dstrace("SCL_wide", direction = "downstream", recurse = T) +#> 1 - SCL_wide - produced by module_xfaostat_L101_RawDataPreProc3_SCL_FBS +#> FAO supply utilization account dataset, 2010+, wide (tonne) +#> Preprocessed FAOSTAT SCL; ** PRE-BUILT ** +#> Dependent: Bal_new_all (#2 below) +#> Dependent: SUA_food_macronutrient_rate (#3 below) +#> 2 - Bal_new_all - produced by module_xfaostat_L105_DataConnectionToSUA +#> Bal_new_all (Ktonne) +#> Preprocessed FAO SUA 2010 - 2019 +#> Dependent: xfaostat_L199_GCAMDATA_FAOSTAT_CSV (#4 below) +#> 4 - xfaostat_L199_GCAMDATA_FAOSTAT_CSV - produced by module_xfaostat_L199_CSVExportAgSUA +#> Export CSV to DIR_OUTPUT_CSV (NA) +#> Export CSV +#> No dependents +#> 3 - SUA_food_macronutrient_rate - produced by module_xfaostat_L106_FoodMacroNutrient +#> FAO food calories and macronutrient rate (rates) +#> Detailed FAO food calories and macrotunitent info for 414 SUA items + +#> 12 fish items +#> Dependent: xfaostat_L199_GCAMDATA_FAOSTAT_CSV (#4 above) + +# Get input of a module +inputs_of("module_xfaostat_L105_DataConnectionToSUA") +#> [1] "aglu/FAO/FAO_items" "aglu/FAO/Mapping_FBSH_SCL_OilCake" +#> [3] "QCL_PROD" "QCL_AN_LIVEANIMAL_MEATEQ" +#> [5] "TCL_wide" "TM_bilateral_wide" +#> [7] "FBSH_CB_wide" "FBS_wide" +#> [9] "SCL_wide" + +# Get output of a module +outputs_of("module_xfaostat_L106_FoodMacroNutrient") +#> [1] "SUA_food_macronutrient_rate"
-# Not evaluated for convenience 
-# Load gcamfaostat
-devtools::load_all() 
-
-# Load FAO_items from cache
-load_from_cache("aglu/FAO/FAO_items") %>% first() -> FAO_items
-# Print
-head(FAO_items)
-
-# Define a list of input data
-MODULE_INPUTS <- c("Bal_new_all")
-# Load a list of MODULE_INPUTS from cache and assign to their name
-get_data_list(all_data = load_from_cache(MODULE_INPUTS), data_list = MODULE_INPUTS)
-# Print
-head(Bal_new_all)
+# Not evaluated for convenience +# Load gcamfaostat +devtools::load_all() + +# Load FAO_items from cache +load_from_cache("aglu/FAO/FAO_items") %>% first() -> FAO_items +# Print +head(FAO_items) + +# Define a list of input data +MODULE_INPUTS <- c("Bal_new_all") +# Load a list of MODULE_INPUTS from cache and assign to their name +get_data_list(all_data = load_from_cache(MODULE_INPUTS), data_list = MODULE_INPUTS) +# Print +head(Bal_new_all)

diff --git a/vignettes/vignette_preparing_data.Rmd b/vignettes/vignette_preparing_data.Rmd index 7e1d0dc0..c1e40cab 100644 --- a/vignettes/vignette_preparing_data.Rmd +++ b/vignettes/vignette_preparing_data.Rmd @@ -22,7 +22,8 @@ knitr::opts_chunk$set( The `gcamfaostat` tool is designed to streamline the processing and synthesis of raw data sourced from [FAOSTAT](https://www.fao.org/faostat/en/#data). The initial phase of this process involves data procurement, with a critical awareness of FAOSTAT's ongoing data updates. These updates encompass a spectrum of improvements, including, e.g., the addition of historical data for new countries and items, data completion for previously missing entries, and refinements such as changes in country nomenclature. This dynamic data landscape underscores the need for a robust and adaptable approach in the `gcamfaostat` workflow. Here we describe a few key functions in `gcamdatafaostat` created to procure the raw data and facilitate the processing. When `gcamfaostat` is downloaded, preprocessed FAOSTAT data, i.e., output of the `xfaostat_L101_RawDataPreProc*` modules, are stored in the [Prebuilt Data](https://github.com/JGCRI/gcamfaostat/blob/main/data/PREBUILT_DATA.rda) of the package. -* The package can be run with using those data, but make sure `Process_Raw_FAO_Data <- FALSE` in `constants.R`. + +* The package can be run with using those data, but make sure `Process_Raw_FAO_Data <- FALSE` in `constants.R`. * The prebuilt data were generated using FAOSTAT data archived in Zenodo address specified in the `FF_download_RemoteArchive` function [@xin_zhao_2023_8260225]. diff --git a/vignettes/vignette_processing_flow.Rmd b/vignettes/vignette_processing_flow.Rmd index 362d1957..41915fbf 100644 --- a/vignettes/vignette_processing_flow.Rmd +++ b/vignettes/vignette_processing_flow.Rmd @@ -58,13 +58,16 @@ As an illustrative example, the first tier comprises 168 commodities, generated As gcamfaostat is built upon the foundation of `gcamdata` and leverages the powerful drake framework, inheriting functions designed for tracking data flows. In the following, we will delve into several key features that exemplify the strengths of this integration. [`info`](https://jgcri.github.io/gcamfaostat/reference/info.html) + * The function returns information of an object, including name, metadata information, precursors and dependents. [`dstrace`](https://jgcri.github.io/gcamfaostat/reference/dstrace.html) + * The function is able to trace data flows by providing precursors and dependents of an object recursively. [`load_from_cache`](https://jgcri.github.io/gcamfaostat/reference/load_from_cache.html) -* If a drake cache is available, e.g., when `driver_drake()` had been run, this function, if given a list of object names, loads the objects from the cache into a list of data frames. + +* If a drake cache is available, e.g., when `driver_drake()` had been run, this function, if given a list of object names, loads the objects from the cache into a list of data frames. * The function [`get_data_list`](https://jgcri.github.io/gcamfaostat/reference/get_data_list.html) can be used to assign each object in the list to a data frame.