README.Rmd

---
output: github_document
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
library(dplyr)
library(bitfield)
library(CoordinateCleaner)
library(stringr)
library(magrittr)
library(knitr)
```

# bitfield <a href='https://github.com/ehrmanns/bitfield/'><img src='man/figures/logo.svg' align="right" height="200" /></a>

<!-- badges: start -->
<!-- [![CRAN\_Status\_Badge](http://www.r-pkg.org/badges/version/)](https://cran.r-project.org/package=) -->
<!-- [![DOI](https://zenodo.org/badge/DOI/)](https://doi.org/) -->

[![R-CMD-check](https://github.com/ehrmanns/bitfield/workflows/R-CMD-check/badge.svg)](https://github.com/ehrmanns/bitfield/actions)
[![codecov](https://codecov.io/gh/ehrmanns/bitfield/branch/master/graph/badge.svg?token=hjppymcGr3)](https://codecov.io/gh/ehrmanns/bitfield)
[![Lifecycle: experimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental)

<!-- [![](http://cranlogs.r-pkg.org/badges/grand-total/)](https://cran.r-project.org/package=) -->
<!-- badges: end -->

## Overview

This package is designed to capture the computational footprint of any model workflow or output. It achieves this by encoding computational decisions into sequences of bits (i.e., [bitfields](https://en.wikipedia.org/wiki/Bit_field)) that are transformed to integer values. This allows storing a range of information into a single column of a table or a raster layer, which can be useful when documenting 

  - the metadata of any dataset by collecting information throughout the dataset creation process, 
  - intermediate data that accrue along a workflow, or 
  - a set of output metrics or parameters.
  - ...

Think of a bit as a switch representing off and on states. A combination of a pair of bits can store four states, and n bits can accommodate 2^n states. These states could be the outcomes of (simple) tests that document binary responses, cases or numeric values. The data produced in that way could be described as meta-analytic or meta-algorithmic data, because they can be re-used to extend an analysis pipeline or algorithm by downstream applications.

## Installation

Install the official version from CRAN:

```{r, eval=FALSE}
# install.packages("bitfield")
```

Install the latest development version from github:

```{r, eval=FALSE}
devtools::install_github("bitfloat/bitfield")
```


## Examples

```{r}
library(bitfield)
library(dplyr, warn.conflicts = FALSE)
library(terra, warn.conflicts = FALSE)
```

Let's first load an example dataset

```{r}
bf_tbl$x                                       # invalid (259) and improbable (0) coordinate value

bf_tbl$y                                       # Inf and NaN value

bf_tbl$commodity                               # NA value or mislabelled term ("honey")

bf_tbl$yield                                   # correct range?!

bf_tbl$year                                    # flags (*r)

# and there is a set of valid commodity terms
validComm <- c("soybean", "maize")
```

The first step is in creating what is called `registry` in `bitfield`. This registry captures all the information required to build the bitfield

```{r}
reg <- bf_registry(name = "yield_QA",
                   description = "this example bitfield documents quality assessment in a table of yield data.")
```

Then, individual bit flags need to be grown by specifying the respective operators. These operators create flags for the most common applications, such as `na` (to test for missing values), `case` (to test what case/class the observations are part of),`nChar` (to count the number of characters of a variable), or `numeric` to encode a numeric (floating point) variable as bit sequence.
  
```{r}
# tests for longitude availability
reg <- 
  bf_test(operator = "na",                       # the operator with which to build the test
          data = bf_tbl,                         # specify where to determine flags
          x = x,                                 # ... and which variable to test
          pos = 1,                               # specify at which position to store the flag
          registry = reg)                        # provide the registry to update

# test which case an observation is part of
reg <- 
  bf_test(operator = "case", data = bf_tbl,
          yield >= 11, yield < 11 & yield > 9, yield < 9 & commodity == "maize",
          registry = reg)

# test the length (number of characters) of values
reg <- 
  bf_test(operator = "nChar", data = bf_tbl, x = y, registry = reg)
  
# store a simplified (e.g. rounded) numeric value
reg <-
  bf_test(operator = "numeric", data = bf_tbl, x = yield, format = "half", 
          registry = reg)
```

These are functions that represent the possible encoding types boolean (`bool`), enumerated cases (`enum`), (signed) integers (`int`), and numeric floating-point (`num`). The encoding type determines various storage parameters of the resulting flags. This is, however, not yet the bitfield. The registry is merely the instruction manual, so to speak, to create the bitfield and encode it as integer, with the function `bf_encode()`.

```{r}
reg
(field <- bf_encode(registry = reg))
```

The bitfield can be decoded based on the registry with the function `bf_decode()` at a later point in time or another workflow, where the metadata contained in the bitfield can be used or extended in a downstream application.

```{r}
flags <- bf_decode(x = field, registry = reg, sep = "-")

# -> prints legend by default, which is also available in bf_env$legend

bf_tbl |>
  bind_cols(flags) |>
  kable()
```

The column `bf_binary`, in combination with the legend, can be read one step at a time. For example, considering the first bit, we see that no observation has an `NA` value and considering the second bit, we see that observations 4 and 6 have a `yield` smaller than 9 and a `commodity` value "maize" (case 3 with binary value `10`).

However, more computation friendly, we can also separate the bitfield into distinct columns per flag and we can load the decoded values from the package environment `bf_env`. 

```{r}
bf_decode(x = field, registry = reg, verbose = FALSE)

# access values manually
ls(bf_env)
bf_env[["nChar_y"]]
```

Beware that numeric values that have been encoded in this way likely have a lower precision than the input values (which may not be a problem when only rounded values are of interest). This can be adjusted by setting the respective parameters in the operator that encodes numeric values (a vignette explaining this in detail will follow).

```{r}
old <- options(pillar.sigfig = 7)
tibble::tibble(original = bf_tbl$yield, 
               bitfield = bf_env$numeric_yield)
options(old)
```

## Bitfields for raster data

An interesting use case is in encoding metadata for modelled gridded data. This is possible simply by calling the `.rast()` function to wrap the gridded object, applying the functions of this package, and overwriting the integer values of the bitfield into a copy of the original gridded object.

```{r}
# example data
bf_rst <- rast(nrows = 3, ncols = 3, 
               vals = as.integer(c(1, 2, 3, NA, 5, 6, 7, 8, 9)))

# build the registry
reg <- bf_registry(name = "raster_meta",
                   description = "this example bitfield documents metadata for a raster object.")

reg <- bf_test(operator = "na", 
               data = .rast(bf_rst), x = lyr.1)
reg <- bf_test(operator = "range", data = .rast(bf_rst), x = lyr.1, min = 4, 
               max = 8, name = "range_lyr.1", na.val = FALSE, registry = reg)

# encode as bitfield (and make raster out of it)
field <- bf_encode(registry = reg)
rst_field <- rast(bf_rst, vals = field, names = names(field))

# decode (gridded) bitfield somewhere downstream
flags <- bf_decode(x = values(rst_field, dataframe = TRUE), registry = reg)

bind_cols(flags, field)

plot(c(bf_rst, rst_field))
```