Skip to content

Commit 4128a37

Browse files
ChriunChriun
committed
jupyter notebook docs
1 parent 37c0fea commit 4128a37

File tree

3 files changed

+246
-0
lines changed

3 files changed

+246
-0
lines changed

‎_docs/instructor/autograding/jupyter_notebook.md

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
---
2+
title: Jupyter Notebooks
3+
category: Instructor > Autograding
4+
---
5+
6+
### Example Configuration
7+
8+
The following configuration can be found on the [Submitty GitHub](https://github.com/Submitty/Submitty/tree/main/more_autograding_examples/jupyter_notebook_autograding).
9+
10+
```json
11+
{
12+
"autograding" : {
13+
"submission_to_runner" : [ "*.ipynb", "*.png" ],
14+
"work_to_details" : [ "**/*.ipynb", "**/*.png", "**/*.txt", "**/*.err" ]
15+
},
16+
"autograding_method": "docker",
17+
"resource_limits" : {
18+
"RLIMIT_NPROC" : 32,
19+
"RLIMIT_FSIZE": 20971520 // 20 MB
20+
},
21+
"max_submission_size": 10485760, // 10 MB
22+
"container_options": {
23+
"container_image": "submitty/jupyter:latest"
24+
},
25+
"allow_system_calls": [
26+
"COMMUNICATIONS_AND_NETWORKING_INTERPROCESS_COMMUNICATION",
27+
"COMMUNICATIONS_AND_NETWORKING_KILL",
28+
"COMMUNICATIONS_AND_NETWORKING_SIGNALS",
29+
"COMMUNICATIONS_AND_NETWORKING_SOCKETS",
30+
"COMMUNICATIONS_AND_NETWORKING_SOCKETS_MINIMAL",
31+
"FILE_MANAGEMENT_MOVE_DELETE_RENAME_FILE_DIRECTORY",
32+
"FILE_MANAGEMENT_RARE",
33+
"PROCESS_CONTROL_ADVANCED",
34+
"PROCESS_CONTROL_NEW_PROCESS_THREAD",
35+
"PROCESS_CONTROL_MEMORY_ADVANCED",
36+
"PROCESS_CONTROL_SYNCHRONIZATION",
37+
"PROCESS_CONTROL_SCHEDULING",
38+
"FILE_MANAGEMENT_PERMISSIONS",
39+
"UNKNOWN"
40+
],
41+
"testcases": [
42+
{
43+
"title": "Executes successfully",
44+
// Although wildcard is used, the grader will only expect one notebook file. Use if filename cannot be determined.
45+
"command": "jupyter_notebook_grader -i *.ipynb -o executed.ipynb",
46+
"points": 1,
47+
"validation": [
48+
{
49+
"method": "fileExists",
50+
"actual_file": "executed.ipynb",
51+
"show_actual" : "always",
52+
"show_message" : "always"
53+
}
54+
]
55+
},
56+
{
57+
"pre_commands" : [
58+
{
59+
"command" : "cp",
60+
"testcase" : "test01",
61+
"source" : "cell1*.*",
62+
"destination" : "./"
63+
}
64+
],
65+
"title": "STDOUT",
66+
"points": 1,
67+
"validation": [
68+
{
69+
"method": "diff",
70+
"actual_file": "cell1_stdout.txt",
71+
"expected_string" : "hello world!"
72+
},
73+
{
74+
"method" : "warnIfEmpty",
75+
"actual_file" : "cell1_stderr.txt",
76+
"show_actual" : "always",
77+
"deduction" : 0.0
78+
},
79+
{
80+
"method" : "warnIfEmpty",
81+
"actual_file" : "cell1_source.txt",
82+
"show_actual" : "always",
83+
"deduction" : 0.0
84+
},
85+
{
86+
"method" : "errorIfNotEmpty",
87+
"actual_file" : "cell1.err",
88+
"show_actual" : "always",
89+
"deduction" : 0.0
90+
}
91+
]
92+
},
93+
{
94+
"pre_commands" : [
95+
{
96+
"command" : "cp",
97+
"testcase" : "test01",
98+
"source" : "cell2*.*",
99+
"destination" : "./"
100+
}
101+
],
102+
"title": "Markdown",
103+
"points": 1,
104+
"validation": [
105+
{
106+
"method": "diff",
107+
"actual_file": "cell2.txt",
108+
"expected_string" : "## 2. Create IPython images with the given \"one.png\" and \"two.png\".\nUse `from IPython.display import Image`"
109+
},
110+
{
111+
"method" : "errorIfNotEmpty",
112+
"actual_file" : "cell2.err",
113+
"show_actual" : "always",
114+
"deduction" : 0.0
115+
}
116+
]
117+
}
118+
}
119+
```
120+
### Required Fields
121+
122+
For autograding Jupyter Notebooks, the following fields must be included in your autograding configuration.
123+
124+
```json
125+
{
126+
"autograding" : {
127+
"submission_to_runner" : [ "*.ipynb", "*.png" ],
128+
"work_to_details" : [ "**/*.ipynb", "**/*.png", "**/*.txt", "**/*.err" ]
129+
},
130+
"autograding_method": "docker",
131+
"container_options": {
132+
"container_image": "submitty/jupyter:latest"
133+
},
134+
"allow_system_calls": [
135+
"COMMUNICATIONS_AND_NETWORKING_INTERPROCESS_COMMUNICATION",
136+
"COMMUNICATIONS_AND_NETWORKING_KILL",
137+
"COMMUNICATIONS_AND_NETWORKING_SIGNALS",
138+
"COMMUNICATIONS_AND_NETWORKING_SOCKETS",
139+
"COMMUNICATIONS_AND_NETWORKING_SOCKETS_MINIMAL",
140+
"FILE_MANAGEMENT_MOVE_DELETE_RENAME_FILE_DIRECTORY",
141+
"FILE_MANAGEMENT_RARE",
142+
"PROCESS_CONTROL_ADVANCED",
143+
"PROCESS_CONTROL_NEW_PROCESS_THREAD",
144+
"PROCESS_CONTROL_MEMORY_ADVANCED",
145+
"PROCESS_CONTROL_SYNCHRONIZATION",
146+
"PROCESS_CONTROL_SCHEDULING",
147+
"FILE_MANAGEMENT_PERMISSIONS",
148+
"UNKNOWN"
149+
]
150+
}
151+
```
152+
153+
You may also need to pass in resource limit values or a max submission size.
154+
155+
```json
156+
"resource_limits" : {
157+
"RLIMIT_NPROC" : 32, //
158+
"RLIMIT_FSIZE": 20971520 // 20 MB
159+
},
160+
"max_submission_size": 10485760, // 10 MB
161+
```
162+
163+
`RLIMIT_NPROC` will allow the necessary resources for the script to execute a submitted Jupyter Notebook.
164+
165+
`RLIMIT_FSIZE` allows the script to save the notebook in Submitty. In this case, it allows at most a 20 MB file to be executed and saved.
166+
167+
### Precommands
168+
169+
Because of the saved outputs, you will need to apply precommands to each testcase that requires files generated from the validation
170+
method.
171+
172+
```json
173+
{
174+
"pre_commands" : [
175+
{
176+
"command" : "cp",
177+
// Assuming the first testcase is the case where the validation method was run
178+
"testcase" : "test01",
179+
"source" : "{filename}*.*",
180+
"destination" : "./"
181+
}
182+
],
183+
...
184+
},
185+
```
186+
187+
### Validation Method
188+
189+
We offer a validation method that parses the cells in Jupyter Notebooks for autograding. The script requires a single specified
190+
input notebook (the student's submission) and outputs an executed version of the notebook. If the student has no restrictions on
191+
the naming convention for their submitted notebook, you can use a wildcard to expect any input.
192+
193+
```
194+
"command": "jupyter_notebook_grader -i *.ipynb -o executed.ipynb"
195+
```
196+
197+
The possible saved files based on parsed output are highlighted below.
198+
199+
| Cell Type | File Pattern | Description |
200+
| --------- | ----------------------- | --------------------------------------------------- |
201+
| Markdown | {filename}.txt | Contains raw markdown source from cell |
202+
| Code | {filename}_source.txt | Contains Python source code from cell |
203+
| | {filename}_err.txt | Contains traceback if error occurs, otherwise empty |
204+
| | {filename}_stdout.txt | Captures standard output (e.g., print() statements) |
205+
| | {filename}_stderr.txt | Captures standard error streams |
206+
| | {filename}_result.txt | Text representation of cell's result |
207+
| | {filename}.png | Generated image outputs |
208+
209+
By default, the filenames are generated based on the index of the cell in the notebook, starting from 1.
210+
211+
### Submitty IDs
212+
213+
Instructors can add Submitty IDs to mark specific cells in Jupyter Notebooks to grade. These specified IDs will replace the
214+
default filename of the saved output (e.g. cell1.txt --> {submitty_id}.txt). If students modify their provided Jupyter Notebook
215+
to be out of order, this method will correctly find the cell to grade. Note that your autograding configuration will also need
216+
to match the name of the saved file.
217+
218+
```json
219+
{
220+
"pre_commands" : [
221+
{
222+
"command" : "cp",
223+
"testcase" : "test01",
224+
"source" : "{submitty_id}*.*",
225+
"destination" : "./"
226+
}
227+
],
228+
"title": "STDOUT",
229+
"points": 1,
230+
"validation": [
231+
{
232+
"method": "diff",
233+
"actual_file": "{submitty_id}_stdout.txt",
234+
"expected_string" : "hello world!"
235+
}
236+
]
237+
}
238+
```
239+
240+
Jupyter Notebook provides an option to easily edit cell metadata, introduced in [7.1.0](https://jupyter-notebook.readthedocs.io/en/stable/changelog.html#id116). In the `Edit` dropdown, you can find
241+
`Edit Notebook Metadata`. An editor panel will open on the right side of your screen to easily type in the metadata.
242+
243+
![](/images/jupyter_metadata.png)
244+
245+
An alternative option would be to manually edit the Jupyter Notebook JSON itself to include the `submitty_id` property in a cell's metadata.

‎images/jupyter_metadata.png

86.4 KB
Loading

‎navtreedata.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,7 @@ var NAVTREE =
135135
[ "Hidden Testcase Examples", "/instructor/autograding/hidden_testcase_examples", null ],
136136
[ "Docker Images", "/instructor/autograding/docker_images", null ],
137137
[ "Docker UI", "/instructor/autograding/docker_ui", null ],
138+
[ "Jupyter Notebook", "/instructor/autograding/jupyter_notebook", null ],
138139
[ "Static Analysis", "/instructor/autograding/static_analysis/index", [
139140
[ "Overview", "/instructor/autograding/static_analysis/index", null ],
140141
[ "Count", "/instructor/autograding/static_analysis/count", null ],

0 commit comments

Comments
 (0)