[Feature] TensorDictPrimer with single default_value callable

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] DensifyReward postproc #2823
• [Feature] DataLoadingPrimer.repeat #2822
• [Feature] DataLoadingPrimer handling of dataloader with batch-size > 0 #2821
• [Lint] pyupgrade #2819
• [Feature] LLMEnv and DataLoadingPrimer #2818
• [BugFix] Fix batch_locked check in check_env_specs + error message callable #2817
• [Feature] NonTensor batched arg #2816
• [BugFix] Fix env.full_done_spec~s~ #2815
• -> [Feature] TensorDictPrimer with single default_value callable #2732

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2732

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit d0ad84e with merge base 8c9dc05 ():

NEW FAILURE - The following job has failed:

• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 913adc82c2a7957ef01545ebf765c8bfa4cc7521c644cd1ac2519be48d133955 /exec failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestConditionalSkip::test_trans_parallel_env_check[True-False]
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh) (trunk failure)
  test/test_transforms.py::TestTimer::test_transform_env

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Example script:

from torchrl.envs import TensorDictPrimer, PendulumEnv, StepCounter, TrajCounter
import torch
from tensordict import assert_close
# let's build a primer that outputs the same Pendulum initial state 4 times in a row
env = PendulumEnv()

def val_iterator(N = 4, env=env):
    i = 0
    while True:
        i += 1
        r = env.reset()
        for _ in range(N):
            torch.manual_seed(i)
            yield r

iterator = val_iterator()

def val_generator(iterator=iterator):
    return next(iterator)

print('observation_spec', env.observation_spec)
primer = TensorDictPrimer(primers=env.observation_spec, default_value=val_generator, single_default_value=True)
env = env.append_transform(primer)
env = env.append_transform(StepCounter(max_steps=50))
env = env.append_transform(TrajCounter())
r = env.rollout(1000, break_when_any_done=False)
print(r[0]["th"], r[0]["thdot"])
print(r[50]["th"], r[50]["thdot"])

# To compute the GAE with empirical adv, we fist need to collapse the trajs
r["traj_count"] = r["traj_count"]//4
r["next", "traj_count"] = r["next", "traj_count"]//4

assert_close(r[:50], r[50:100])
assert_close(r[:50], r[100:150])

# Reshape r
print(r.shape)
r = r.reshape(-1, 4, 50)
assert r[0]["traj_count"].unique().numel() == 1

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 149. Improved: $\large\color{#35bf28}9$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_simple	0.6091s	0.5242s	1.9078 Ops/s	1.9953 Ops/s	$\color{#d91a1a}-4.39\%$
test_transformed	1.1301s	1.0452s	0.9568 Ops/s	0.9681 Ops/s	$\color{#d91a1a}-1.17\%$
test_serial	1.6408s	1.5436s	0.6479 Ops/s	0.6512 Ops/s	$\color{#d91a1a}-0.51\%$
test_parallel	1.3916s	1.3032s	0.7673 Ops/s	0.7602 Ops/s	$\color{#35bf28}+0.94\%$
test_step_mdp_speed[True-True-True-True-True]	0.1772ms	30.3643μs	32.9334 KOps/s	32.6400 KOps/s	$\color{#35bf28}+0.90\%$
test_step_mdp_speed[True-True-True-True-False]	42.0200μs	18.2386μs	54.8289 KOps/s	55.3440 KOps/s	$\color{#d91a1a}-0.93\%$
test_step_mdp_speed[True-True-True-False-True]	52.8100μs	17.5393μs	57.0150 KOps/s	56.6458 KOps/s	$\color{#35bf28}+0.65\%$
test_step_mdp_speed[True-True-True-False-False]	39.6050μs	10.3882μs	96.2630 KOps/s	98.0335 KOps/s	$\color{#d91a1a}-1.81\%$
test_step_mdp_speed[True-True-False-True-True]	75.9940μs	33.2766μs	30.0511 KOps/s	30.4378 KOps/s	$\color{#d91a1a}-1.27\%$
test_step_mdp_speed[True-True-False-True-False]	44.9450μs	20.2635μs	49.3497 KOps/s	50.0526 KOps/s	$\color{#d91a1a}-1.40\%$
test_step_mdp_speed[True-True-False-False-True]	81.4810μs	19.6370μs	50.9243 KOps/s	51.2091 KOps/s	$\color{#d91a1a}-0.56\%$
test_step_mdp_speed[True-True-False-False-False]	45.0950μs	12.3038μs	81.2757 KOps/s	82.2219 KOps/s	$\color{#d91a1a}-1.15\%$
test_step_mdp_speed[True-False-True-True-True]	94.5390μs	34.6725μs	28.8413 KOps/s	28.8837 KOps/s	$\color{#d91a1a}-0.15\%$
test_step_mdp_speed[True-False-True-True-False]	59.1620μs	22.1739μs	45.0982 KOps/s	45.8259 KOps/s	$\color{#d91a1a}-1.59\%$
test_step_mdp_speed[True-False-True-False-True]	50.8370μs	19.4629μs	51.3797 KOps/s	51.3829 KOps/s	$-0.01\%$
test_step_mdp_speed[True-False-True-False-False]	46.5280μs	12.2654μs	81.5301 KOps/s	82.2614 KOps/s	$\color{#d91a1a}-0.89\%$
test_step_mdp_speed[True-False-False-True-True]	74.3320μs	36.4387μs	27.4434 KOps/s	27.5165 KOps/s	$\color{#d91a1a}-0.27\%$
test_step_mdp_speed[True-False-False-True-False]	77.6010μs	23.7465μs	42.1115 KOps/s	42.2972 KOps/s	$\color{#d91a1a}-0.44\%$
test_step_mdp_speed[True-False-False-False-True]	42.0700μs	21.2267μs	47.1105 KOps/s	47.1683 KOps/s	$\color{#d91a1a}-0.12\%$
test_step_mdp_speed[True-False-False-False-False]	37.7710μs	13.9835μs	71.5127 KOps/s	72.2021 KOps/s	$\color{#d91a1a}-0.95\%$
test_step_mdp_speed[False-True-True-True-True]	70.6340μs	34.7140μs	28.8068 KOps/s	28.1863 KOps/s	$\color{#35bf28}+2.20\%$
test_step_mdp_speed[False-True-True-True-False]	59.5640μs	22.1637μs	45.1189 KOps/s	45.6589 KOps/s	$\color{#d91a1a}-1.18\%$
test_step_mdp_speed[False-True-True-False-True]	58.1200μs	22.2846μs	44.8740 KOps/s	44.8082 KOps/s	$\color{#35bf28}+0.15\%$
test_step_mdp_speed[False-True-True-False-False]	36.5490μs	13.5895μs	73.5860 KOps/s	73.6049 KOps/s	$\color{#d91a1a}-0.03\%$
test_step_mdp_speed[False-True-False-True-True]	0.1007ms	36.4335μs	27.4472 KOps/s	27.0703 KOps/s	$\color{#35bf28}+1.39\%$
test_step_mdp_speed[False-True-False-True-False]	72.6770μs	23.6606μs	42.2643 KOps/s	42.0693 KOps/s	$\color{#35bf28}+0.46\%$
test_step_mdp_speed[False-True-False-False-True]	2.5610ms	23.8901μs	41.8583 KOps/s	42.0576 KOps/s	$\color{#d91a1a}-0.47\%$
test_step_mdp_speed[False-True-False-False-False]	45.7970μs	15.3431μs	65.1760 KOps/s	65.5372 KOps/s	$\color{#d91a1a}-0.55\%$
test_step_mdp_speed[False-False-True-True-True]	72.7180μs	38.0451μs	26.2846 KOps/s	26.1322 KOps/s	$\color{#35bf28}+0.58\%$
test_step_mdp_speed[False-False-True-True-False]	68.6000μs	25.8228μs	38.7255 KOps/s	38.9689 KOps/s	$\color{#d91a1a}-0.62\%$
test_step_mdp_speed[False-False-True-False-True]	60.8050μs	24.0185μs	41.6345 KOps/s	41.7499 KOps/s	$\color{#d91a1a}-0.28\%$
test_step_mdp_speed[False-False-True-False-False]	62.0210μs	15.1981μs	65.7975 KOps/s	65.3246 KOps/s	$\color{#35bf28}+0.72\%$
test_step_mdp_speed[False-False-False-True-True]	80.0930μs	39.5115μs	25.3091 KOps/s	24.9022 KOps/s	$\color{#35bf28}+1.63\%$
test_step_mdp_speed[False-False-False-True-False]	52.9910μs	27.1673μs	36.8089 KOps/s	36.9817 KOps/s	$\color{#d91a1a}-0.47\%$
test_step_mdp_speed[False-False-False-False-True]	59.9840μs	25.6127μs	39.0431 KOps/s	39.2083 KOps/s	$\color{#d91a1a}-0.42\%$
test_step_mdp_speed[False-False-False-False-False]	52.1590μs	16.8212μs	59.4488 KOps/s	58.6264 KOps/s	$\color{#35bf28}+1.40\%$
test_values[generalized_advantage_estimate-True-True]	10.9890ms	9.7375ms	102.6963 Ops/s	97.8178 Ops/s	$\color{#35bf28}+4.99\%$
test_values[vec_generalized_advantage_estimate-True-True]	25.4592ms	24.1131ms	41.4713 Ops/s	41.2173 Ops/s	$\color{#35bf28}+0.62\%$
test_values[td0_return_estimate-False-False]	0.2522ms	0.1767ms	5.6607 KOps/s	5.6167 KOps/s	$\color{#35bf28}+0.78\%$
test_values[td1_return_estimate-False-False]	25.7207ms	24.0957ms	41.5012 Ops/s	40.6521 Ops/s	$\color{#35bf28}+2.09\%$
test_values[vec_td1_return_estimate-False-False]	27.9720ms	24.2519ms	41.2339 Ops/s	40.7073 Ops/s	$\color{#35bf28}+1.29\%$
test_values[td_lambda_return_estimate-True-False]	37.5117ms	35.1980ms	28.4107 Ops/s	28.4375 Ops/s	$\color{#d91a1a}-0.09\%$
test_values[vec_td_lambda_return_estimate-True-False]	26.5685ms	24.2116ms	41.3026 Ops/s	41.2574 Ops/s	$\color{#35bf28}+0.11\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	11.7673ms	8.5152ms	117.4375 Ops/s	115.1205 Ops/s	$\color{#35bf28}+2.01\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	2.2076ms	1.9390ms	515.7303 Ops/s	533.1293 Ops/s	$\color{#d91a1a}-3.26\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4959ms	0.3715ms	2.6915 KOps/s	2.6708 KOps/s	$\color{#35bf28}+0.78\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	45.1792ms	43.2924ms	23.0987 Ops/s	23.8673 Ops/s	$\color{#d91a1a}-3.22\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	5.2308ms	3.4832ms	287.0885 Ops/s	291.2284 Ops/s	$\color{#d91a1a}-1.42\%$
test_dqn_speed[False-None]	2.3976ms	1.4322ms	698.2309 Ops/s	705.9476 Ops/s	$\color{#d91a1a}-1.09\%$
test_dqn_speed[False-backward]	2.0444ms	1.9193ms	521.0098 Ops/s	522.9396 Ops/s	$\color{#d91a1a}-0.37\%$
test_dqn_speed[True-None]	0.7423ms	0.4820ms	2.0749 KOps/s	1.9729 KOps/s	$\textbf{\color{#35bf28}+5.17\%}$
test_dqn_speed[True-backward]	0.9563ms	0.9059ms	1.1038 KOps/s	909.2286 Ops/s	$\textbf{\color{#35bf28}+21.40\%}$
test_dqn_speed[reduce-overhead-None]	0.7112ms	0.4936ms	2.0260 KOps/s	2.0394 KOps/s	$\color{#d91a1a}-0.66\%$
test_dqn_speed[reduce-overhead-backward]	0.9890ms	0.9136ms	1.0946 KOps/s	1.0722 KOps/s	$\color{#35bf28}+2.09\%$
test_ddpg_speed[False-None]	3.9407ms	2.9735ms	336.3020 Ops/s	334.4878 Ops/s	$\color{#35bf28}+0.54\%$
test_ddpg_speed[False-backward]	4.4258ms	4.1277ms	242.2684 Ops/s	242.4775 Ops/s	$\color{#d91a1a}-0.09\%$
test_ddpg_speed[True-None]	1.8401ms	1.2378ms	807.9150 Ops/s	803.5753 Ops/s	$\color{#35bf28}+0.54\%$
test_ddpg_speed[True-backward]	2.1910ms	2.1342ms	468.5661 Ops/s	419.1653 Ops/s	$\textbf{\color{#35bf28}+11.79\%}$
test_ddpg_speed[reduce-overhead-None]	1.7117ms	1.2369ms	808.4464 Ops/s	811.3037 Ops/s	$\color{#d91a1a}-0.35\%$
test_ddpg_speed[reduce-overhead-backward]	2.2908ms	2.1358ms	468.2074 Ops/s	463.2291 Ops/s	$\color{#35bf28}+1.07\%$
test_sac_speed[False-None]	8.6191ms	8.1725ms	122.3614 Ops/s	121.1889 Ops/s	$\color{#35bf28}+0.97\%$
test_sac_speed[False-backward]	12.1506ms	10.9818ms	91.0600 Ops/s	90.4599 Ops/s	$\color{#35bf28}+0.66\%$
test_sac_speed[True-None]	2.6602ms	2.0935ms	477.6740 Ops/s	471.1064 Ops/s	$\color{#35bf28}+1.39\%$
test_sac_speed[True-backward]	4.3504ms	3.8378ms	260.5672 Ops/s	265.0584 Ops/s	$\color{#d91a1a}-1.69\%$
test_sac_speed[reduce-overhead-None]	2.5896ms	2.1016ms	475.8354 Ops/s	474.7573 Ops/s	$\color{#35bf28}+0.23\%$
test_sac_speed[reduce-overhead-backward]	3.9153ms	3.8176ms	261.9433 Ops/s	262.8223 Ops/s	$\color{#d91a1a}-0.33\%$
test_redq_speed[False-None]	14.8691ms	13.3388ms	74.9690 Ops/s	75.7892 Ops/s	$\color{#d91a1a}-1.08\%$
test_redq_speed[False-backward]	23.8230ms	22.8594ms	43.7457 Ops/s	44.1594 Ops/s	$\color{#d91a1a}-0.94\%$
test_redq_speed[True-None]	5.5661ms	4.9555ms	201.7948 Ops/s	195.7609 Ops/s	$\color{#35bf28}+3.08\%$
test_redq_speed[True-backward]	13.3599ms	12.5631ms	79.5984 Ops/s	77.5902 Ops/s	$\color{#35bf28}+2.59\%$
test_redq_speed[reduce-overhead-None]	6.1849ms	5.1366ms	194.6827 Ops/s	194.7702 Ops/s	$\color{#d91a1a}-0.04\%$
test_redq_speed[reduce-overhead-backward]	13.2352ms	12.5546ms	79.6522 Ops/s	76.8510 Ops/s	$\color{#35bf28}+3.64\%$
test_redq_deprec_speed[False-None]	14.9018ms	13.0821ms	76.4405 Ops/s	76.1479 Ops/s	$\color{#35bf28}+0.38\%$
test_redq_deprec_speed[False-backward]	21.9246ms	18.8657ms	53.0061 Ops/s	52.1427 Ops/s	$\color{#35bf28}+1.66\%$
test_redq_deprec_speed[True-None]	4.4671ms	3.9116ms	255.6525 Ops/s	200.8119 Ops/s	$\textbf{\color{#35bf28}+27.31\%}$
test_redq_deprec_speed[True-backward]	10.4768ms	8.9520ms	111.7068 Ops/s	116.4033 Ops/s	$\color{#d91a1a}-4.03\%$
test_redq_deprec_speed[reduce-overhead-None]	4.2842ms	3.8670ms	258.6001 Ops/s	258.6651 Ops/s	$\color{#d91a1a}-0.03\%$
test_redq_deprec_speed[reduce-overhead-backward]	9.4182ms	8.4895ms	117.7924 Ops/s	118.8772 Ops/s	$\color{#d91a1a}-0.91\%$
test_td3_speed[False-None]	8.4200ms	8.1747ms	122.3282 Ops/s	121.7917 Ops/s	$\color{#35bf28}+0.44\%$
test_td3_speed[False-backward]	12.4221ms	10.6584ms	93.8229 Ops/s	93.1054 Ops/s	$\color{#35bf28}+0.77\%$
test_td3_speed[True-None]	2.0252ms	1.7900ms	558.6595 Ops/s	556.9502 Ops/s	$\color{#35bf28}+0.31\%$
test_td3_speed[True-backward]	4.6119ms	3.4870ms	286.7814 Ops/s	291.1147 Ops/s	$\color{#d91a1a}-1.49\%$
test_td3_speed[reduce-overhead-None]	2.0413ms	1.7877ms	559.3738 Ops/s	559.1760 Ops/s	$\color{#35bf28}+0.04\%$
test_td3_speed[reduce-overhead-backward]	3.4837ms	3.4093ms	293.3118 Ops/s	292.8839 Ops/s	$\color{#35bf28}+0.15\%$
test_cql_speed[False-None]	40.2062ms	37.0945ms	26.9582 Ops/s	26.6069 Ops/s	$\color{#35bf28}+1.32\%$
test_cql_speed[False-backward]	50.6296ms	47.5040ms	21.0508 Ops/s	20.7944 Ops/s	$\color{#35bf28}+1.23\%$
test_cql_speed[True-None]	18.5668ms	16.2284ms	61.6204 Ops/s	61.3553 Ops/s	$\color{#35bf28}+0.43\%$
test_cql_speed[True-backward]	24.8546ms	22.9863ms	43.5042 Ops/s	43.6497 Ops/s	$\color{#d91a1a}-0.33\%$
test_cql_speed[reduce-overhead-None]	17.8980ms	16.1438ms	61.9434 Ops/s	60.5564 Ops/s	$\color{#35bf28}+2.29\%$
test_cql_speed[reduce-overhead-backward]	24.6440ms	22.9638ms	43.5468 Ops/s	42.3044 Ops/s	$\color{#35bf28}+2.94\%$
test_a2c_speed[False-None]	8.6922ms	7.2559ms	137.8186 Ops/s	134.2229 Ops/s	$\color{#35bf28}+2.68\%$
test_a2c_speed[False-backward]	15.0099ms	14.5538ms	68.7104 Ops/s	67.0312 Ops/s	$\color{#35bf28}+2.51\%$
test_a2c_speed[True-None]	5.2917ms	3.8145ms	262.1559 Ops/s	263.7465 Ops/s	$\color{#d91a1a}-0.60\%$
test_a2c_speed[True-backward]	10.5660ms	10.1697ms	98.3317 Ops/s	93.3463 Ops/s	$\textbf{\color{#35bf28}+5.34\%}$
test_a2c_speed[reduce-overhead-None]	4.1079ms	3.7581ms	266.0899 Ops/s	258.2304 Ops/s	$\color{#35bf28}+3.04\%$
test_a2c_speed[reduce-overhead-backward]	10.7545ms	10.3145ms	96.9513 Ops/s	93.9316 Ops/s	$\color{#35bf28}+3.21\%$
test_ppo_speed[False-None]	8.9221ms	7.6398ms	130.8927 Ops/s	129.3263 Ops/s	$\color{#35bf28}+1.21\%$
test_ppo_speed[False-backward]	16.9989ms	15.1018ms	66.2171 Ops/s	64.3792 Ops/s	$\color{#35bf28}+2.85\%$
test_ppo_speed[True-None]	4.7780ms	4.0995ms	243.9349 Ops/s	235.1746 Ops/s	$\color{#35bf28}+3.73\%$
test_ppo_speed[True-backward]	10.5354ms	10.0290ms	99.7111 Ops/s	89.2607 Ops/s	$\textbf{\color{#35bf28}+11.71\%}$
test_ppo_speed[reduce-overhead-None]	4.8741ms	4.1044ms	243.6418 Ops/s	233.1278 Ops/s	$\color{#35bf28}+4.51\%$
test_ppo_speed[reduce-overhead-backward]	11.1200ms	10.1310ms	98.7070 Ops/s	95.2981 Ops/s	$\color{#35bf28}+3.58\%$
test_reinforce_speed[False-None]	8.1278ms	6.6660ms	150.0146 Ops/s	146.4154 Ops/s	$\color{#35bf28}+2.46\%$
test_reinforce_speed[False-backward]	10.3877ms	10.0322ms	99.6791 Ops/s	96.9830 Ops/s	$\color{#35bf28}+2.78\%$
test_reinforce_speed[True-None]	3.8548ms	3.0740ms	325.3048 Ops/s	316.4977 Ops/s	$\color{#35bf28}+2.78\%$
test_reinforce_speed[True-backward]	9.3219ms	9.0311ms	110.7286 Ops/s	110.0069 Ops/s	$\color{#35bf28}+0.66\%$
test_reinforce_speed[reduce-overhead-None]	3.8839ms	3.0806ms	324.6130 Ops/s	321.2275 Ops/s	$\color{#35bf28}+1.05\%$
test_reinforce_speed[reduce-overhead-backward]	9.8189ms	9.0271ms	110.7771 Ops/s	109.4994 Ops/s	$\color{#35bf28}+1.17\%$
test_iql_speed[False-None]	34.9493ms	32.5477ms	30.7241 Ops/s	29.3239 Ops/s	$\color{#35bf28}+4.78\%$
test_iql_speed[False-backward]	50.2271ms	45.8335ms	21.8181 Ops/s	21.8039 Ops/s	$\color{#35bf28}+0.07\%$
test_iql_speed[True-None]	12.5664ms	11.3092ms	88.4238 Ops/s	86.1622 Ops/s	$\color{#35bf28}+2.62\%$
test_iql_speed[True-backward]	23.9083ms	22.7959ms	43.8676 Ops/s	43.4940 Ops/s	$\color{#35bf28}+0.86\%$
test_iql_speed[reduce-overhead-None]	12.6225ms	11.3311ms	88.2524 Ops/s	86.3564 Ops/s	$\color{#35bf28}+2.20\%$
test_iql_speed[reduce-overhead-backward]	24.1885ms	22.5783ms	44.2903 Ops/s	42.9081 Ops/s	$\color{#35bf28}+3.22\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.5128ms	4.8973ms	204.1931 Ops/s	197.3199 Ops/s	$\color{#35bf28}+3.48\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2.5144ms	0.5354ms	1.8678 KOps/s	1.8869 KOps/s	$\color{#d91a1a}-1.01\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.8047ms	0.5157ms	1.9391 KOps/s	2.0112 KOps/s	$\color{#d91a1a}-3.59\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.0331ms	4.7022ms	212.6651 Ops/s	213.7276 Ops/s	$\color{#d91a1a}-0.50\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.1599ms	0.5191ms	1.9263 KOps/s	1.9268 KOps/s	$\color{#d91a1a}-0.03\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7522ms	0.4922ms	2.0317 KOps/s	2.0178 KOps/s	$\color{#35bf28}+0.69\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	2.3393ms	1.6881ms	592.3916 Ops/s	596.3778 Ops/s	$\color{#d91a1a}-0.67\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.8112ms	1.5913ms	628.4290 Ops/s	633.2228 Ops/s	$\color{#d91a1a}-0.76\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.1781ms	4.7520ms	210.4355 Ops/s	208.0704 Ops/s	$\color{#35bf28}+1.14\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2.3309ms	0.6596ms	1.5160 KOps/s	1.5142 KOps/s	$\color{#35bf28}+0.12\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.9156ms	0.6323ms	1.5815 KOps/s	1.5840 KOps/s	$\color{#d91a1a}-0.15\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.4175ms	4.7211ms	211.8170 Ops/s	214.7078 Ops/s	$\color{#d91a1a}-1.35\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2.3445ms	0.5247ms	1.9057 KOps/s	1.7952 KOps/s	$\textbf{\color{#35bf28}+6.15\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.8072ms	0.5064ms	1.9749 KOps/s	1.9603 KOps/s	$\color{#35bf28}+0.74\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.4702ms	4.6660ms	214.3180 Ops/s	213.6244 Ops/s	$\color{#35bf28}+0.32\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.3846ms	0.5194ms	1.9253 KOps/s	1.9336 KOps/s	$\color{#d91a1a}-0.43\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7300ms	0.4882ms	2.0484 KOps/s	2.0282 KOps/s	$\color{#35bf28}+1.00\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.1808ms	4.8248ms	207.2633 Ops/s	211.2120 Ops/s	$\color{#d91a1a}-1.87\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.4787s	1.3441ms	744.0001 Ops/s	1.5149 KOps/s	$\textbf{\color{#d91a1a}-50.89\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8263ms	0.6363ms	1.5716 KOps/s	1.5745 KOps/s	$\color{#d91a1a}-0.19\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	5.5559ms	4.2563ms	234.9471 Ops/s	244.3263 Ops/s	$\color{#d91a1a}-3.84\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	7.9090ms	2.3714ms	421.6965 Ops/s	421.5039 Ops/s	$\color{#35bf28}+0.05\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	1.9273ms	1.3423ms	744.9785 Ops/s	722.3525 Ops/s	$\color{#35bf28}+3.13\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	5.6566ms	4.2927ms	232.9547 Ops/s	242.1526 Ops/s	$\color{#d91a1a}-3.80\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	5.9488ms	2.3801ms	420.1555 Ops/s	430.8502 Ops/s	$\color{#d91a1a}-2.48\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	6.0647ms	1.4445ms	692.2713 Ops/s	707.8488 Ops/s	$\color{#d91a1a}-2.20\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.4379s	13.1948ms	75.7874 Ops/s	32.7891 Ops/s	$\textbf{\color{#35bf28}+131.14\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	8.5752ms	2.7474ms	363.9817 Ops/s	400.3796 Ops/s	$\textbf{\color{#d91a1a}-9.09\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	5.7897ms	1.6134ms	619.7989 Ops/s	632.1373 Ops/s	$\color{#d91a1a}-1.95\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	14.2223ms	12.2122ms	81.8851 Ops/s	77.3982 Ops/s	$\textbf{\color{#35bf28}+5.80\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	19.2390ms	14.5030ms	68.9512 Ops/s	70.0617 Ops/s	$\color{#d91a1a}-1.59\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	22.5150ms	21.0125ms	47.5906 Ops/s	46.6781 Ops/s	$\color{#35bf28}+1.96\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	16.1045ms	14.6012ms	68.4876 Ops/s	68.2189 Ops/s	$\color{#35bf28}+0.39\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	23.2778ms	20.9544ms	47.7227 Ops/s	47.1472 Ops/s	$\color{#35bf28}+1.22\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	26.3936ms	16.1536ms	61.9059 Ops/s	62.6293 Ops/s	$\color{#d91a1a}-1.16\%$

[github-actions]

Result of GPU Benchmark Tests

Expand to view detailed results

Name	Max	Mean	Ops
test_simple	0.8844s	0.7910s	1.2643 Ops/s
test_transformed	1.4603s	1.3767s	0.7264 Ops/s
test_serial	2.3246s	2.2336s	0.4477 Ops/s
test_parallel	1.9351s	1.8520s	0.5400 Ops/s
test_step_mdp_speed[True-True-True-True-True]	0.1802ms	38.9522μs	25.6725 KOps/s
test_step_mdp_speed[True-True-True-True-False]	53.1700μs	22.9673μs	43.5402 KOps/s
test_step_mdp_speed[True-True-True-False-True]	54.6110μs	21.8053μs	45.8605 KOps/s
test_step_mdp_speed[True-True-True-False-False]	40.0900μs	12.6715μs	78.9172 KOps/s
test_step_mdp_speed[True-True-False-True-True]	74.3810μs	41.2419μs	24.2472 KOps/s
test_step_mdp_speed[True-True-False-True-False]	58.1910μs	25.1096μs	39.8254 KOps/s
test_step_mdp_speed[True-True-False-False-True]	55.9310μs	23.5774μs	42.4136 KOps/s
test_step_mdp_speed[True-True-False-False-False]	42.8610μs	14.9727μs	66.7881 KOps/s
test_step_mdp_speed[True-False-True-True-True]	76.4410μs	43.7125μs	22.8768 KOps/s
test_step_mdp_speed[True-False-True-True-False]	60.0110μs	27.4366μs	36.4476 KOps/s
test_step_mdp_speed[True-False-True-False-True]	68.8610μs	23.8947μs	41.8503 KOps/s
test_step_mdp_speed[True-False-True-False-False]	88.2610μs	15.1044μs	66.2058 KOps/s
test_step_mdp_speed[True-False-False-True-True]	74.4110μs	45.7924μs	21.8377 KOps/s
test_step_mdp_speed[True-False-False-True-False]	60.2310μs	29.4115μs	34.0003 KOps/s
test_step_mdp_speed[True-False-False-False-True]	53.3110μs	26.4290μs	37.8372 KOps/s
test_step_mdp_speed[True-False-False-False-False]	44.0900μs	17.1725μs	58.2326 KOps/s
test_step_mdp_speed[False-True-True-True-True]	73.9410μs	43.1753μs	23.1614 KOps/s
test_step_mdp_speed[False-True-True-True-False]	64.0610μs	27.4523μs	36.4268 KOps/s
test_step_mdp_speed[False-True-True-False-True]	55.9310μs	27.9207μs	35.8157 KOps/s
test_step_mdp_speed[False-True-True-False-False]	45.6300μs	16.7252μs	59.7899 KOps/s
test_step_mdp_speed[False-True-False-True-True]	76.9510μs	44.8344μs	22.3043 KOps/s
test_step_mdp_speed[False-True-False-True-False]	70.9710μs	29.7100μs	33.6587 KOps/s
test_step_mdp_speed[False-True-False-False-True]	3.0712ms	30.2248μs	33.0854 KOps/s
test_step_mdp_speed[False-True-False-False-False]	55.8300μs	18.9383μs	52.8030 KOps/s
test_step_mdp_speed[False-False-True-True-True]	82.6110μs	47.4181μs	21.0890 KOps/s
test_step_mdp_speed[False-False-True-True-False]	63.5510μs	32.0014μs	31.2486 KOps/s
test_step_mdp_speed[False-False-True-False-True]	66.0610μs	30.0754μs	33.2498 KOps/s
test_step_mdp_speed[False-False-True-False-False]	51.1210μs	18.8809μs	52.9634 KOps/s
test_step_mdp_speed[False-False-False-True-True]	77.0710μs	49.2966μs	20.2854 KOps/s
test_step_mdp_speed[False-False-False-True-False]	66.0510μs	33.9203μs	29.4809 KOps/s
test_step_mdp_speed[False-False-False-False-True]	62.5210μs	31.7578μs	31.4883 KOps/s
test_step_mdp_speed[False-False-False-False-False]	63.4010μs	21.1014μs	47.3902 KOps/s
test_values[generalized_advantage_estimate-True-True]	24.5704ms	23.9385ms	41.7737 Ops/s
test_values[vec_generalized_advantage_estimate-True-True]	94.9806ms	2.7861ms	358.9256 Ops/s
test_values[td0_return_estimate-False-False]	0.1109ms	77.3018μs	12.9363 KOps/s
test_values[td1_return_estimate-False-False]	54.8278ms	53.5999ms	18.6568 Ops/s
test_values[vec_td1_return_estimate-False-False]	1.2449ms	1.0602ms	943.2287 Ops/s
test_values[td_lambda_return_estimate-True-False]	86.3431ms	84.6623ms	11.8116 Ops/s
test_values[vec_td_lambda_return_estimate-True-False]	1.3034ms	1.0653ms	938.7236 Ops/s
test_gae_speed[generalized_advantage_estimate-False-1-512]	24.8575ms	24.4468ms	40.9052 Ops/s
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.0309ms	0.7258ms	1.3777 KOps/s
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7419ms	0.6561ms	1.5241 KOps/s
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.5096ms	1.4656ms	682.3374 Ops/s
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.7247ms	0.6720ms	1.4881 KOps/s
test_dqn_speed[False-None]	1.5660ms	1.4418ms	693.5872 Ops/s
test_dqn_speed[False-backward]	2.3650ms	2.0665ms	483.9064 Ops/s
test_dqn_speed[True-None]	0.5830ms	0.5295ms	1.8887 KOps/s
test_dqn_speed[True-backward]	1.2350ms	1.1972ms	835.2908 Ops/s
test_dqn_speed[reduce-overhead-None]	0.9599ms	0.5629ms	1.7764 KOps/s
test_dqn_speed[reduce-overhead-backward]	1.0922ms	1.0440ms	957.8242 Ops/s
test_ddpg_speed[False-None]	3.0672ms	2.6903ms	371.7085 Ops/s
test_ddpg_speed[False-backward]	4.4845ms	4.0592ms	246.3510 Ops/s
test_ddpg_speed[True-None]	1.7065ms	1.2919ms	774.0537 Ops/s
test_ddpg_speed[True-backward]	2.4183ms	2.3431ms	426.7774 Ops/s
test_ddpg_speed[reduce-overhead-None]	1.3745ms	1.2896ms	775.4084 Ops/s
test_ddpg_speed[reduce-overhead-backward]	2.0972ms	1.8842ms	530.7174 Ops/s
test_sac_speed[False-None]	8.2766ms	7.8683ms	127.0929 Ops/s
test_sac_speed[False-backward]	11.1843ms	10.6758ms	93.6695 Ops/s
test_sac_speed[True-None]	1.8376ms	1.7697ms	565.0762 Ops/s
test_sac_speed[True-backward]	3.5194ms	3.4633ms	288.7381 Ops/s
test_sac_speed[reduce-overhead-None]	20.7326ms	11.7125ms	85.3785 Ops/s
test_sac_speed[reduce-overhead-backward]	2.5069ms	1.5786ms	633.4617 Ops/s
test_redq_speed[False-None]	7.7939ms	7.3088ms	136.8207 Ops/s
test_redq_speed[False-backward]	11.7013ms	11.1813ms	89.4347 Ops/s
test_redq_speed[True-None]	2.3463ms	2.2473ms	444.9795 Ops/s
test_redq_speed[True-backward]	4.3149ms	3.9092ms	255.8070 Ops/s
test_redq_speed[reduce-overhead-None]	2.3572ms	2.2720ms	440.1380 Ops/s
test_redq_speed[reduce-overhead-backward]	4.3893ms	3.9289ms	254.5269 Ops/s
test_redq_deprec_speed[False-None]	9.1639ms	8.7567ms	114.1978 Ops/s
test_redq_deprec_speed[False-backward]	12.2104ms	11.7080ms	85.4118 Ops/s
test_redq_deprec_speed[True-None]	2.8698ms	2.5419ms	393.4102 Ops/s
test_redq_deprec_speed[True-backward]	4.2579ms	4.1880ms	238.7765 Ops/s
test_redq_deprec_speed[reduce-overhead-None]	2.6434ms	2.5388ms	393.8835 Ops/s
test_redq_deprec_speed[reduce-overhead-backward]	4.4752ms	4.3590ms	229.4097 Ops/s
test_td3_speed[False-None]	8.0482ms	7.7107ms	129.6892 Ops/s
test_td3_speed[False-backward]	10.7575ms	10.2538ms	97.5251 Ops/s
test_td3_speed[True-None]	1.5895ms	1.5659ms	638.6019 Ops/s
test_td3_speed[True-backward]	3.2604ms	3.2226ms	310.3081 Ops/s
test_td3_speed[reduce-overhead-None]	48.7727ms	24.8278ms	40.2774 Ops/s
test_td3_speed[reduce-overhead-backward]	1.4847ms	1.4372ms	695.7983 Ops/s
test_cql_speed[False-None]	16.5302ms	16.0850ms	62.1698 Ops/s
test_cql_speed[False-backward]	22.0875ms	21.5580ms	46.3865 Ops/s
test_cql_speed[True-None]	3.5000ms	3.1205ms	320.4606 Ops/s
test_cql_speed[True-backward]	5.6158ms	5.4794ms	182.5033 Ops/s
test_cql_speed[reduce-overhead-None]	20.8950ms	12.8173ms	78.0197 Ops/s
test_cql_speed[reduce-overhead-backward]	1.8794ms	1.7617ms	567.6372 Ops/s
test_a2c_speed[False-None]	3.1731ms	3.0221ms	330.8957 Ops/s
test_a2c_speed[False-backward]	6.6034ms	5.9194ms	168.9368 Ops/s
test_a2c_speed[True-None]	1.6046ms	1.2998ms	769.3497 Ops/s
test_a2c_speed[True-backward]	2.9919ms	2.8391ms	352.2220 Ops/s
test_a2c_speed[reduce-overhead-None]	16.1760ms	8.7166ms	114.7232 Ops/s
test_a2c_speed[reduce-overhead-backward]	1.5323ms	1.4251ms	701.7288 Ops/s
test_ppo_speed[False-None]	3.7190ms	3.5184ms	284.2199 Ops/s
test_ppo_speed[False-backward]	7.0327ms	6.5563ms	152.5246 Ops/s
test_ppo_speed[True-None]	1.4288ms	1.3619ms	734.2625 Ops/s
test_ppo_speed[True-backward]	3.0087ms	2.9740ms	336.2524 Ops/s
test_ppo_speed[reduce-overhead-None]	1.0354ms	0.9450ms	1.0582 KOps/s
test_ppo_speed[reduce-overhead-backward]	1.4698ms	1.3702ms	729.8441 Ops/s
test_reinforce_speed[False-None]	2.2716ms	2.1572ms	463.5669 Ops/s
test_reinforce_speed[False-backward]	3.2799ms	3.1417ms	318.3009 Ops/s
test_reinforce_speed[True-None]	1.3697ms	1.2470ms	801.9025 Ops/s
test_reinforce_speed[True-backward]	2.8975ms	2.8330ms	352.9814 Ops/s
test_reinforce_speed[reduce-overhead-None]	17.4091ms	9.6604ms	103.5155 Ops/s
test_reinforce_speed[reduce-overhead-backward]	1.5500ms	1.4620ms	683.9826 Ops/s
test_iql_speed[False-None]	9.3173ms	8.8817ms	112.5908 Ops/s
test_iql_speed[False-backward]	12.9615ms	12.4484ms	80.3319 Ops/s
test_iql_speed[True-None]	2.2039ms	2.1289ms	469.7229 Ops/s
test_iql_speed[True-backward]	4.8505ms	4.6164ms	216.6210 Ops/s
test_iql_speed[reduce-overhead-None]	0.4829s	12.5742ms	79.5282 Ops/s
test_iql_speed[reduce-overhead-backward]	2.0528ms	2.0043ms	498.9227 Ops/s
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.3987ms	5.9832ms	167.1355 Ops/s
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5260ms	0.3193ms	3.1323 KOps/s
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5026ms	0.2722ms	3.6740 KOps/s
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.9187ms	5.7015ms	175.3926 Ops/s
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.1539ms	0.3197ms	3.1276 KOps/s
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.4465ms	0.2372ms	4.2158 KOps/s
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.4613ms	1.2427ms	804.6961 Ops/s
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.4797ms	1.2806ms	780.9029 Ops/s
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.0016ms	5.8828ms	169.9877 Ops/s
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2.2531ms	0.4388ms	2.2791 KOps/s
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6081ms	0.3895ms	2.5674 KOps/s
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.8138ms	5.6939ms	175.6265 Ops/s
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2.0702ms	0.3468ms	2.8831 KOps/s
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5350ms	0.2780ms	3.5970 KOps/s
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	8.1433ms	5.7144ms	174.9959 Ops/s
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.8122ms	0.3220ms	3.1060 KOps/s
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.5915ms	0.3168ms	3.1561 KOps/s
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.0450ms	5.8874ms	169.8531 Ops/s
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.9635ms	0.4324ms	2.3124 KOps/s
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6659ms	0.4538ms	2.2036 KOps/s
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	6.9906ms	5.3770ms	185.9782 Ops/s
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	6.2125ms	2.0009ms	499.7811 Ops/s
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	8.9320ms	1.2475ms	801.6319 Ops/s
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.4531s	14.4316ms	69.2925 Ops/s
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	9.0545ms	1.9363ms	516.4379 Ops/s
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	7.9509ms	1.2179ms	821.0779 Ops/s
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	7.5651ms	5.6264ms	177.7320 Ops/s
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	8.6124ms	2.1903ms	456.5652 Ops/s
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	7.3366ms	1.3848ms	722.1085 Ops/s
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	13.3929ms	13.1424ms	76.0895 Ops/s
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	18.5307ms	16.5502ms	60.4224 Ops/s
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	18.3191ms	18.0144ms	55.5112 Ops/s
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	18.8066ms	16.8999ms	59.1718 Ops/s
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	18.2971ms	17.9322ms	55.7657 Ops/s
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	19.2695ms	18.1194ms	55.1896 Ops/s