Create samples_single_gemini.jsonl for BioChartQA -lyg

evals/registry/data/05_biochart/samples_single_gemini.jsonl

@@ -0,0 +1,15 @@
+{"file_name": "../bio_chart_data/10.48550_arXiv.2310.00926_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Figure 3,  which has a higher accurate score, with the graph encoder or without? \n\na) with graph encoder \nb) w/o graph encoder"}], "ideal": "a) with graph encoder"}
+{"file_name": "../bio_chart_data/10.1093_bib_bbab147_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Figure 2 part A, what contributes most to the gain in cell similarity? \n\na) Gene+KEGG \nb) Gene+Reactome \nc) Gene+de novo pathway \nd) Gene+Wikipathways"}], "ideal": "d) Gene+Wikipathways"}
+{"file_name": "../bio_chart_data/10.1038_s41592-023-01938-4_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Fig. 4, in the PCA analysis, which group of cell lines is classified as intermediate? \n\na) MM001 \nb) MM031 \nc) MM057 \nd) M074"}], "ideal": "c)MM057"}
+{"file_name": "../bio_chart_data/10.1371_journal.pcbi.1008379_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Fig. 3.d, which method performs the best (has the highest AUC)? \n\na) PCOR \nb) PROB \nc) LEAP \nd) CLR"}], "ideal": "b) PROB"}
+{"file_name": "../bio_chart_data/10.1101_2022.08.20.504663_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Supp Fig 1, which number is closest to the number of cells that received two perturbations? \n\na) 10000 \nb) 40000 \nc) 50000 \nd) 60000"}], "ideal": "b) 40000"}
+{"file_name": "../bio_chart_data/10.1016_j.crmeth.2023.100411_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Figure 2 part A, what is the Average Rank (F1-score) of ForSyn? \n\na) 2 \nb) 4 \nc) 5 \nd) 10"}], "ideal": "a) 2"}
+{"file_name": "../bio_chart_data/10.1101_gr.277488.122_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Figure 3B, which network backbone gets the best AUPR for pre-mRNA? \n\na) Linear \nb) cycle \nc) bifurcating \nd) converging"}], "ideal": "d) converging"}
+{"file_name": "../bio_chart_data/10.1101_gr.275870.121_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Figure 2 part A, which method has a lower NLL value? \n\na) pi-CNN  \nb) Epi-GraphReg"}], "ideal": "b) Epi-GraphReg"}
+{"file_name": "../bio_chart_data/10.1016_j.cels.2020.11.013_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "Figure 3 part A: Under the single-to-combo condition, which method has the lowest model performance (correlation)? \n\na) NN \nb) Co-exp \nc) BP \nd) CellBox"}], "ideal": "c) BP"}
+{"file_name": "../bio_chart_data/10.1101_2023.03.24.533888_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "What is the impact of increasing the Kh value on the predicted signaling strengths of the gene interactions as shown in Figure S1 part A? \na) It significantly increases the predicted signaling strengths. \nb) It does not significantly change the predicted signaling strengths. \nc) It significantly decreases the predicted signaling strengths. \nd) It causes the predicted signaling strengths to fluctuate unpredictably."}], "ideal": "c) It significantly decreases the predicted signaling strengths."}
+{"file_name": "../bio_chart_data/10.1093_bib_bbab366_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "For Figure 3 C in this paper, when the gene number is 8000, which one has the highest ARI score? \n\na) Raw \nb) SCORE \nc) CSN \nd) SCENIC"}], "ideal": "b) SCORE"}
+{"file_name": "../bio_chart_data/10.15252_msb.202211176_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "For Figure 2 part A in this paper, what trend does Uy exhibit as Ux increases? \n\na) Decrease \nb) Increase"}], "ideal": "b) Increase"}
+{"file_name": "../bio_chart_data/10.1101_2022.04.17.488600_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "For Figure 5 part d in this paper, which distance is greater, Alpha, Beta, or Delta? \n\na) Alpha \nb) Beta \nc) Delta"}], "ideal": "b) Beta"}
+{"file_name": "../bio_chart_data/10.21203_rs.3.rs-2675584_v1_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Fig. SR1 of this paper, in the validation MSE (Mean Squared Error) results for the SIM350 system, which model had the lowest average MSE under high noise conditions? \n\na) PHOENIX \nb) Unregularized PHOENIX \nc) OOTB (tanh) \nd) OOTB (sigmoid) \ne) OOTB (ReLU)"}], "ideal": "a) PHOENIX"}
+{"file_name": "../bio_chart_data/10.1371_journal.pcbi.1007909_singlepage.pdf", "input": [{"role": "system", "content": "You are a highly intelligent biology scientist who answers the following multiple choice question correctly.\nOnly write the options and values down, such as 'b) 2045.1'."}, {"role": "user", "content": "In Fig 4 of this paper, Which of the following would AKTi activate, according to the signalling pathway shown in the image? \n\na) MEKi \nb) RAFi \nc) STAT6-pY641 \nd) c-JUN"}], "ideal": "c) STAT6-pY641"}