[help] Rprofile conflicted rules not recognized when using crew

[icrichmond]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hello,

I have a large {targets} pipeline that I am now trying to implement distributed computing with {crew} on (for the first time). My workflow uses {conflicted}, which is called in a project level .Rprofile as suggested in Chapter 7 of the {targets} book.

After adding a {crew} controller to my target options, individual targets now error as if {conflicted} has not been loaded/implemented (though it appears to be loaded at the start of the pipeline, based on output messaging). I attempted the solutions proposed in discussion 1437 , with no success.

I have attempted to produce a minimal reproducible example below:

My _targets.R file looks like this:

# === Targets -------------------------------------------------------------


# Source ------------------------------------------------------------------
library(targets)
tar_source('R')



# Options -----------------------------------------------------------------
# Targets
tar_option_set(format = 'qs',
               controller = crew_controller_local(workers = 10),
               seed = 666)


# Renv --------------------------------------------------------------------
activate()
snapshot()
restore()


# Targets: all ------------------------------------------------------------

c(
  
  tar_target(
    data, 
    tibble(x = seq(1:5), y = seq(5:1))
  ),
  
  tar_target(
    data_renamed,
    data %>% 
      rename(x_rename = x)
  )
  
)

My packages are loaded in R/packages.R. I've included all the packages I use in my real repository. My packages.R script looks like:

suppressPackageStartupMessages({

  library(targets)
  library(tarchetypes)
  library(crew)
  library(qs2)
  library(quarto)
  library(renv)
  
  library(conflicted)
  
  library(downloader)
  library(readr)
  library(bit64)
 
  library(sf)
  library(geojsonio)
  library(stars)
  library(osmdata)
  library(rnaturalearth)
  library(ggrepel)
  
  library(plyr)
  library(dplyr)
  library(tidyr)
  library(stringr)
  library(tibble)
  
  library(data.table)
  
  library(units)
  library(anytime)
  
  library(FactoMineR)
  library(factoextra)
  library(ggplot2)
  library(MetBrewer)
  library(visNetwork)
  
  
  library(vegan)
  
  library(mice)
  library(cluster)
  library(ranger)
  library(tidymodels)
  library(themis)
  library(vip)
  library(future)
  library(DALEXtra)
  
  library(xfun)
  

})

My .Rprofile looks like this:

# renv
source("renv/activate.R")


# timeout
options(timeout=2000)

# packages
#source('packages.R')

# conflicted
conflicted::conflict_prefer_all('dplyr', quiet = T)
conflicted::conflict_prefer_all('base', 'bit64', quiet = T)
conflicted::conflicts_prefer(base::`%in%`)
library(conflicted)

The error message associated with this example is here:

When I remove the controller argument from tar_option_set(), this error goes away and the code runs with no issues. Interestingly, if I remove the {themis} package from the packages script, the code will work for the minimum reproducible example, though this fix does not apply to my larger workflow.

I have tried sourcing the packages in the .Rprofile instead of _targets.R, I have tried sourcing the packages in tar_option_set(), I have tried putting the conflicted calls in _targets.R, none of these seem to have any luck. This is definitely a me problem, but I would really appreciate any insight that folks could give me on this!

Also, last but not least -- thank you so much for creating the wonderful world of {targets}. It is an absolutely incredible piece of work, including the reference material which helps me so much!

---

Session Info :

R version 4.5.0 (2025-04-11 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 10 x64 (build 19045)

Matrix products: default
LAPACK version 3.12.1

locale:
[1] LC_COLLATE=English_Canada.utf8 LC_CTYPE=English_Canada.utf8 LC_MONETARY=English_Canada.utf8
[4] LC_NUMERIC=C LC_TIME=English_Canada.utf8

time zone: America/Toronto
tzcode source: internal

attached base packages:
[1] stats graphics grDevices datasets utils methods base

other attached packages:
[1] plyr_1.8.9 conflicted_1.2.0 renv_1.1.4 qs2_0.1.5 tarchetypes_0.13.1
[6] crew_1.2.1 targets_1.11.3 dplyr_1.1.4 tibble_3.3.0

loaded via a namespace (and not attached):
[1] base64url_1.4 compiler_4.5.0 promises_1.3.3 tidyselect_1.2.1 Rcpp_1.1.0
[6] parallel_4.5.0 callr_3.7.6 later_1.4.2 fastmap_1.2.0 yaml_2.3.10
[11] R6_2.6.1 generics_0.1.4 igraph_2.1.4 nanonext_1.6.2 knitr_1.50
[16] backports_1.5.0 mirai_2.4.0 stringfish_0.17.0 pillar_1.11.0 rlang_1.1.6
[21] cachem_1.1.0 xfun_0.52 fs_1.6.6 RcppParallel_5.1.10 memoise_2.0.1
[26] cli_3.6.5 withr_3.0.2 magrittr_2.0.3 ps_1.9.1 processx_3.8.6
[31] rstudioapi_0.17.1 secretbase_1.0.5 lifecycle_1.0.4 prettyunits_1.2.0 vctrs_0.6.5
[36] evaluate_1.0.4 glue_1.8.0 data.table_1.17.8 codetools_0.2-20 tools_4.5.0
[41] pkgconfig_2.0.3

[wlandau]

The crew worker does run .Rprofile (from checking the worker log messages, see crew_options_local()). However, targets then loads the packages in tar_option_get("packages") in the order listed. The default value is (.packages()), which lists plyr after dplyr. But you can explicitly list packages in a suitable order using tar_option_set(). For example, this pipeline fails:

library(crew)
library(targets)

tar_option_set(
  controller = crew_controller_local(
    options_local = crew_options_local(log_directory = "~/Desktop")
  ),
  packages = c("dplyr", "plyr")
)

list(
  tar_target(
    data, 
    tibble(x = seq(1:5), y = seq(5:1))
  ),
  tar_target(
    data_renamed,
    data %>% 
      rename(x_rename = x)
  )
)

whereas this one succeeds:

library(crew)
library(targets)

tar_option_set(
  controller = crew_controller_local(
    options_local = crew_options_local(log_directory = "~/Desktop")
  ),
  packages = c("plyr", "dplyr")
)

list(
  tar_target(
    data, 
    tibble(x = seq(1:5), y = seq(5:1))
  ),
  tar_target(
    data_renamed,
    data %>% 
      rename(x_rename = x)
  )
)

[icrichmond]

Got it, this makes sense! Thank you so much for your response and for all the amazing work you do!!

Ran the larger workflow with this solution and it worked.