[general] including targets as dependencies without loading them into memory (for use with DuckDB/parquet)

[adamaltmejd]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

What is the best way to include targets as dependencies without loading them into memory?

More specifically I have a pipeline that deals with some rather large data sets and I'm running out of memory. I was thinking of using format="parquet" and using DuckDB to process some parts of the pipeline. I would then include a target with a dbExecute() call that loads the parquet files with read_parquet, outputs a parquet file and returns its path (to be stored in targets with format="file"). But how do I ensure that the code runs when the upstream dependencies change without actually loading the targets?

library(targets)
library(tarchetypes)

tar_option_set(
  packages = c("arrow", "duckdb", "DBI")
)

list(
    tar_target(mean, sample.int(10)),
    tar_target(sd, sample.int(10)),
    tar_target(dt, data.table(id = 1:1000, norm = rnorm(1000, mean = mean, sd = sd), exp = rexp(1000, rate = 1/mean)),
               pattern = map(mean = mean, sd = sd), iteration = "list",
               format = "parquet"),
    tar_target(out, {
      con <- dbConnect(duckdb())
      res <- dbExecute(con, "
        COPY
          (SELECT * FROM read_parquet('_targets/objects/dt_*', union_by_name = true))
          TO 'result.parquet'
          (FORMAT parquet);")
        "result.parquet"
    }, format = "file")
)

[wlandau]

You could try tar_target(out, ..., retrieval = "none").

[adamaltmejd]

Jesus was it that easy, here I was trying to write my own custom targets and stuff and all I needed was to read the docs more carefully. Thanks!