You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: evaluation/badges.html
+2-2
Original file line number
Diff line number
Diff line change
@@ -233,7 +233,7 @@ <h1 class="title">Badges</h1>
233
233
<p>This page evaluates the extent to which the author-published research artefacts meet the criteria of badges related to reproducibility from various organisations and journals.</p>
<ahref="../evaluation/posts/2024_06_04/index.html" class="no-external">Evaluated study against badge criteria and recommendations for sharing of research artefacts.</a>
270
+
<ahref="../evaluation/posts/2024_06_04/index.html" class="no-external">Evaluated study against guidelines (badges, sharing artefacts, and starting on STRESS-DES reporting).</a>
<ahref="../evaluation/posts/2024_06_03/index.html" class="no-external">Chose a seed that produced results fairly visually similar to the paper, and decided that each figure had now been successfully reproduced. Total time used: 7h 6m (17.8%).…</a>
303
+
<ahref="../evaluation/posts/2024_06_03/index.html" class="no-external">Chose a seed that produced fairly similar figures to paper. Decided that each figure had now been successfully reproduced. Total time used: 7h 6m (17.8%).</a>
<p>Exploring methods for overlaying figures. Not timed as not about reproduction of this study, but about how we are going to do this each time when reproducing.</p>
291
291
<p>Decided that it’s not helpful to do this - spend more time fiddling around with getting them to resize and overlay correctly - and that the simplest option here would be to compare by eye.</p>
Copy file name to clipboardexpand all lines: evaluation/posts/2024_06_03/index.html
+2-2
Original file line number
Diff line number
Diff line change
@@ -257,7 +257,7 @@ <h2 id="toc-title">On this page</h2>
257
257
</div>
258
258
</div>
259
259
<divclass="callout-body-container callout-body">
260
-
<p>Chose a seed that produced results fairly visually similar to the paper, and decided that each figure had now been successfully reproduced. Total time used: 7h 6m (17.8%). Reproduction stage complete.</p>
260
+
<p>Chose a seed that produced fairly similar figures to paper. Decided that each figure had now been successfully reproduced. Total time used: 7h 6m (17.8%).</p>
<li>Remote execution/Online coding environment - Zenodo doesn’t but GitLab does. I was largely working with Zenodo as that was the archived version, but in this case, did say met criteria via GitLab. Importance of which code is chosen when there are two versions. I didn’t initially think it met criteria until checked the GitLab too. Have recommended both are referred to. But do they meet criteria if changes are made to repository after publication? After submission? For example, if we asked them to add a license.</li>
299
299
</ul>
300
300
<p>Compared my decisions for the best practice audit against those made in Monks and Harper. Don’t think I would normally plan to do this. Their GitHub is <ahref="https://github.com/TomMonks/des_sharing_lit_review">TomMonks/des_sharing_lit_review</a>, which provides the file <code>share_sim_data_extract.zip</code>. I used the provided code to clean this and saved it as <code>share_sim_data_extract_clean.csv</code>, which can then view here:</p>
<p>These were evaluated based ONLY on the journal article and supplementary materials (and not on the linked code repository).</p>
453
+
<p>STRESS uncertanties:</p>
454
+
<ul>
455
+
<li>Model outputs - “Specify how and when they are calculated during the model run along with how any measures of error such as confidence intervals are calculated” - I find this criteria quite hard to apply? Here, it’s just N patients and additional time during model, mediane and extreme. Does it need more detail than that to meet it though?</li>
456
+
<li>Experiementation aims - is it applicable? The DES model, as I understand, just tests one scenario - a worst case scenario - so it’s not really “scenario based analysis” comparing scenarios - but it does have A scenario and justification for it?</li>
457
+
<li>Algorithms - is this applicable in this example? Parameters are just from sampling from distributions, or proportions. Assuming its applicable, and that distribing those distributions answers this question?</li>
458
+
<li>Queues - wasn’t certain about this, as I don’t really know what order they get allocated to units. but that’s not a queue? but it kind of is - in that, at the start of the day, they are waiting to be assigned?</li>
459
+
<li>Entry/exit points - they are in model on days they need dialysis, and then leave again? unless get unallocated? and then they never return if their state turns to dead? this isn’t really spelled out though.</li>
<p>Tom mentioned how this study they provided the filled out STRESS checklist to the journal, but it wasn’t published alongside the article, but that would have been beneficial. This is a good suggestion to think about.</p>
472
+
<p>Also, it has proven quite time consuming to evaluate against the checklist - hence, moreso reason to include it with the article, to make it easier for readers to spot these things.</p>
<li>Suggest to detail those uncertainties within logbook</li>
473
483
<li>Any advice can give following uncertainties raised above</li>
474
484
<li>If there are multiple code locations (e.g. code repository and archive), then refer to both when assessing against criteria</li>
485
+
<li>Recommended sources for evaluation (e.g. reporting guidelines is based on article, badges and sharing of research artefacts are based on code)</li>
486
+
<li>Given how long it is taking, would it be relevant to time these steps - in particular with the ambition of timing the evaluation against reporting guidelines - to illustrate how long it can take to find this information? And hence, provide rationale for the benefit of including it with the paper, clearly spelled out? Not much to learn from timing the badges or sharing artefacts evaluation though.</li>
0 commit comments