Skip to content

Commit 76460f2

Browse files
DSouzaMDSouzaM
committed
More work to migrate to Bytecode DSL.
Commits have been squashed to simplify rebase efforts. The original commit messages are: Get the DSL to generate code without errors Get the generated code to compile without errors re-add code to enable Operations interpreter Fix rebase issues + start factoring common validation out to AbstractCompiler.java Fix kwarg validation Fix implementation of Try Add named exception unbinding Fix implementation of With Minor fixes to f-strings and for-else blocks Refactor to support nested root node compilation without FunctionFinder Begin refactoring generators Partly fix generator implementation Support plain one-way generators Fix bug with empty generator expressions Support generator send (non-exceptional cases) Fully support enhanced generators Remove intermediate root node in generator implementation Add branch profile to generator resume code Fix prepareCall to handle generator calling convention merge generator logic back into a single PGenerator class Fix argument validation Fix unbound local checking and qualified names for functions Encapsulate compiler state in an OperationCompilerContext Support async def, async with, and await Get more JUnit tests running with the Operation interpreter Minor improvements to ResumeGenerator code Get JUnit tests using only the Operation interpreter Set catching frame reference for try-finally and with statements Fix stack walking to support continuation frames Minor location + integer constant fixes Convert FrameInfo to an interface and provide separate impls for bytecode and operation interpreters Make CodeUnit an abstract class with Bytecode and Operation implementations Allocate locals contiguously from slot 0 & avoid redundant allocations Misc. bug fixes (incl. name mangling and unbound local checking) Fix class cell reads and docstrings Fix bugs with keyword arguments + augmented assignments Remove AbstractCompiler abstraction, add more syntax checks Support type annotations Implement co_consts and other code fields Respect 'optimize' flag for compilations, constant fold unary negation Fix constants implementation + parser bugs Set exception __context__ when it is raised Properly handle __future__ imports Convert host StackOverflowError to RecursionError Use findBci to compute frame.f_lineno Use interceptTruffleException to set catchBci for exceptions fix Operation DSL API usage Move exception chaining to intercept hook Make future annotations inherited by exec Support interop method calls Fix finallyTry, local accesses, and continuation API usages after DSL updates Save + restore exception state on generator exit + entry Use getLocals/copyLocals APIs instead of hard-coded local offset Store bci into frame when potentially escaping root node use Proxyable for OperationProxy nodes Fix test setup after rebase Update with operation->bytecode package rename Rename POperationXYZ -> PBytecodeDSLXYZ Refactor compiler to return a CodeUnit and use a unique Builder for each root node Rework Marshal to use DataOutput and DataInput APIs Basic serialization implementation Avoid serialize+deserialize step for AST parsing Only build root node once for each MakeFunction/Generator/Coroutine operation Enable frozen modules, don't actually serialize Sources Cleanup style errors. Fix source creation, unintentional root node adoption, and parsing bugs Fix bciToLine Always use character-based sources Add separate DSL config for tests bump CI overlay bump overlay, dsl -> bytecode-dsl in gate tags peg imports to bytecode dsl branch Fix some compilation issues and undo temporary test hacks Break constant collection creation into separate operations Re-enable pattern matching tests and start a DSL implementation Add support for sequence pattern matching Move EnableBytecodeDSLInterpreter from an option to a system property; fix NI compilation bump overlay, add SVM DSL unittests config (wip) attempt to create separate native standalone for the DSL interpreter Pass bytecode DSL flag to builder when building standalone disable fail fast for DSL tests Fix inplace shifts; interactive string expressions Fix bug with empty module __init__ files Fix class decorators, disable pdb test on DSL interpreter, re-enable traceback test Fix super() implementation & fix bug with shadowed cell/free variables Update short-circuit ops, fix super() generator frame lookup Correctly mangle kwargs; implement co_lines; fix imod implementation Parse __future__ annotations before scope analysis to ensure bad annotations rejected Skip boolean coercion for boolean constants Use getLocal API for readClassCell/readSelf Clean up marshal code Use a PBytecodeDSLGeneratorFunctionRootNode instead of creating a bytecode node to instantiate generators Remove extra import Fix NI compilation by putting SourceSection method behind boundary; compute internal the same way as PBytecodeRootNode Fix rebase issue + manual bc frozen module issue bump ci overlay Fix another rebase issue with mx configuration bump overlay update to use new Bytecode DSL APIs (wip) remove bciToLine(int), set bytecode node when intercepting Truffle exceptions Replace instanceof checks with PythonOptions.ENABLE_BYTECODE_DSL_INTERPRETER checks getSourceLocation -> findSourceLocation Undo PythonTests.newBuilder now that DSL is enabled by a system property remove unreachable PKeyword marshalling logic
1 parent 1574ca6 commit 76460f2

File tree

81 files changed

+10452
-6195
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+10452
-6195
lines changed

‎graalpython/com.oracle.graal.python.frozen/freeze_modules.py

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ def relpath_for_posix_display(path, base):
156156
#######################################
157157
# specs
158158

159-
def parse_frozen_specs():
159+
def parse_frozen_specs(suffix):
160160
seen = {}
161161
for section, specs in FROZEN:
162162
parsed = _parse_specs(specs, section, seen)
@@ -165,7 +165,7 @@ def parse_frozen_specs():
165165
try:
166166
source = seen[frozenid]
167167
except KeyError:
168-
source = FrozenSource.from_id(frozenid, pyfile)
168+
source = FrozenSource.from_id(frozenid, suffix, pyfile)
169169
seen[frozenid] = source
170170
else:
171171
assert not pyfile or pyfile == source.pyfile, item
@@ -273,11 +273,11 @@ def iter_subs():
273273
class FrozenSource(namedtuple('FrozenSource', 'id pyfile frozenfile deepfreezefile')):
274274

275275
@classmethod
276-
def from_id(cls, frozenid, pyfile=None):
276+
def from_id(cls, frozenid, suffix, pyfile=None):
277277
if not pyfile:
278278
pyfile = os.path.join(STDLIB_DIR, *frozenid.split('.')) + '.py'
279279
#assert os.path.exists(pyfile), (frozenid, pyfile)
280-
frozenfile = resolve_frozen_file(frozenid, FROZEN_MODULES_DIR)
280+
frozenfile = resolve_frozen_file(frozenid, FROZEN_MODULES_DIR, suffix)
281281
return cls(frozenid, pyfile, frozenfile, STDLIB_DIR)
282282

283283
@classmethod
@@ -313,7 +313,7 @@ def isbootstrap(self):
313313
return self.id in BOOTSTRAP
314314

315315

316-
def resolve_frozen_file(frozenid, destdir):
316+
def resolve_frozen_file(frozenid, destdir, suffix):
317317
"""Return the filename corresponding to the given frozen ID.
318318
319319
For stdlib modules the ID will always be the full name
@@ -326,7 +326,7 @@ def resolve_frozen_file(frozenid, destdir):
326326
raise ValueError(f'unsupported frozenid {frozenid!r}')
327327
# We use a consistent naming convention for all frozen modules.
328328
frozen_symbol = FrozenSource.resolve_symbol(frozenid)
329-
frozenfile = f"Frozen{frozen_symbol}.bin"
329+
frozenfile = f"Frozen{frozen_symbol}.{suffix}"
330330

331331
if not destdir:
332332
return frozenfile
@@ -636,11 +636,17 @@ def main():
636636
STDLIB_DIR = os.path.abspath(parsed_args.python_lib)
637637
FROZEN_MODULES_DIR = os.path.abspath(parsed_args.binary_dir)
638638

639+
if __graalpython__.is_bytecode_dsl_interpreter:
640+
suffix = "bin_dsl"
641+
assert os.path.isdir(parsed_args.binary_dir), "Frozen modules for the DSL should be built after the manual bytecode interpreter."
642+
else:
643+
suffix = "bin"
644+
shutil.rmtree(parsed_args.binary_dir, ignore_errors=True)
645+
os.makedirs(parsed_args.binary_dir)
646+
639647
# create module specs
640-
modules = list(parse_frozen_specs())
648+
modules = list(parse_frozen_specs(suffix))
641649

642-
shutil.rmtree(parsed_args.binary_dir, ignore_errors=True)
643-
os.makedirs(parsed_args.binary_dir)
644650
# write frozen module binary files containing the byte code and class files
645651
# used for importing the binary files
646652
for src in _iter_sources(modules):

‎graalpython/com.oracle.graal.python.pegparser/src/com/oracle/graal/python/pegparser/scope/Scope.java

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,11 +229,15 @@ public boolean isNested() {
229229
}
230230

231231
public HashMap<String, Integer> getSymbolsByType(EnumSet<DefUse> expectedFlags, int start) {
232+
return getSymbolsByType(expectedFlags, EnumSet.noneOf(DefUse.class), start);
233+
}
234+
235+
public HashMap<String, Integer> getSymbolsByType(EnumSet<DefUse> expectedFlags, EnumSet<DefUse> unexpectedFlags, int start) {
232236
int i = start;
233237
HashMap<String, Integer> mapping = new HashMap<>();
234238
for (String key : getSortedSymbols()) {
235239
EnumSet<DefUse> keyFlags = getUseOfName(key);
236-
if (!Collections.disjoint(expectedFlags, keyFlags)) {
240+
if (!Collections.disjoint(expectedFlags, keyFlags) && Collections.disjoint(unexpectedFlags, keyFlags)) {
237241
mapping.put(key, i++);
238242
}
239243
}

‎graalpython/com.oracle.graal.python.pegparser/src/com/oracle/graal/python/pegparser/scope/ScopeEnvironment.java

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ public class ScopeEnvironment {
9898
final HashMap<SSTNode, Scope> blocks = new HashMap<>();
9999
final ErrorCallback errorCallback;
100100
final EnumSet<FutureFeature> futureFeatures;
101+
final HashMap<Scope, Scope> parents = new HashMap<>();
101102

102103
public static ScopeEnvironment analyze(ModTy moduleNode, ErrorCallback errorCallback, EnumSet<FutureFeature> futureFeatures) {
103104
return new ScopeEnvironment(moduleNode, errorCallback, futureFeatures);
@@ -128,6 +129,14 @@ public Scope lookupScope(SSTNode node) {
128129
return blocks.get(node);
129130
}
130131

132+
public Scope lookupParent(Scope scope) {
133+
return parents.get(scope);
134+
}
135+
136+
public Scope getTopScope() {
137+
return topScope;
138+
}
139+
131140
private void analyzeBlock(Scope scope, HashSet<String> bound, HashSet<String> free, HashSet<String> global) {
132141
HashSet<String> local = new HashSet<>();
133142
HashMap<String, DefUse> scopes = new HashMap<>();
@@ -328,6 +337,7 @@ private void enterBlock(String name, Scope.ScopeType type, SSTNode ast) {
328337
if (type == Scope.ScopeType.Annotation) {
329338
return;
330339
}
340+
env.parents.put(scope, prev);
331341
if (prev != null) {
332342
prev.children.add(scope);
333343
}

‎graalpython/com.oracle.graal.python.test.integration/src/com/oracle/graal/python/test/integration/generator/GeneratorTests.java

Lines changed: 116 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ public void desugared() {
7272
}
7373

7474
@Test
75-
public void testYieldFrom() {
75+
public void testYieldFromSimple() {
7676
String source = "def gen1():\n" +
7777
" yield 1\n" +
7878
" yield 2\n" +
@@ -83,4 +83,119 @@ public void testYieldFrom() {
8383
"print(list(gen2()))\n";
8484
assertPrints("[1, 2]\n", source);
8585
}
86+
87+
@Test
88+
public void testYieldFromIterable() {
89+
// yield from should extract an iterator from a non-generator argument
90+
String source = "class Foo:\n" +
91+
" def __init__(self, wrapped):\n" +
92+
" self.wrapped = wrapped\n" +
93+
" def __iter__(self):\n" +
94+
" return iter(self.wrapped)\n" +
95+
"def gen():\n" +
96+
" foo = Foo([1,2,3])\n" +
97+
" yield from foo\n" +
98+
"\n" +
99+
"print(list(gen()))\n";
100+
assertPrints("[1, 2, 3]\n", source);
101+
}
102+
103+
@Test
104+
public void testYieldFromReturn() {
105+
String source = "def gen1():\n" +
106+
" yield 1\n" +
107+
" yield 2\n" +
108+
" return 3\n" +
109+
"\n" +
110+
"def gen2():\n" +
111+
" final = yield from gen1()\n" +
112+
" yield final\n" +
113+
"\n" +
114+
"print(list(gen2()))\n";
115+
assertPrints("[1, 2, 3]\n", source);
116+
}
117+
118+
@Test
119+
public void testYieldFromSend() {
120+
String source = "def gen1(x):\n" +
121+
" yield (yield (yield x))\n" +
122+
"\n" +
123+
"def gen2():\n" +
124+
" yield from gen1(2)\n" +
125+
" yield 8\n" +
126+
"\n" +
127+
"gen = gen2()\n" +
128+
"print(gen.send(None))\n" +
129+
"print(gen.send(4))\n" +
130+
"print(gen.send(6))\n" +
131+
"print(gen.send(42))\n";
132+
assertPrints("2\n4\n6\n8\n", source);
133+
}
134+
135+
@Test
136+
public void testYieldFromThrowCaught() {
137+
String source = "def gen1():\n" +
138+
" try:\n" +
139+
" x = 1\n" +
140+
" while True:\n" +
141+
" x = yield x\n" +
142+
" except ValueError:\n" +
143+
" yield 42\n" +
144+
"\n" +
145+
"def gen2():\n" +
146+
" yield from gen1()\n" +
147+
"\n" +
148+
"gen = gen2()\n" +
149+
"print(gen.send(None))\n" +
150+
"print(gen.send(2))\n" +
151+
"print(gen.send(3))\n" +
152+
"print(gen.throw(ValueError))\n";
153+
assertPrints("1\n2\n3\n42\n", source);
154+
}
155+
156+
@Test
157+
public void testYieldFromThrowUncaught() {
158+
String source = "def gen1():\n" +
159+
" x = 1\n" +
160+
" while True:\n" +
161+
" x = yield x\n" +
162+
"\n" +
163+
"def gen2():\n" +
164+
" yield from gen1()\n" +
165+
"\n" +
166+
"gen = gen2()\n" +
167+
"print(gen.send(None))\n" +
168+
"print(gen.send(2))\n" +
169+
"print(gen.send(3))\n" +
170+
"try:\n" +
171+
" gen.throw(ValueError)\n" +
172+
" print('error')\n" +
173+
"except ValueError:\n" +
174+
" print('success')\n";
175+
assertPrints("1\n2\n3\nsuccess\n", source);
176+
}
177+
178+
@Test
179+
public void testYieldFromClose() {
180+
String source = "def gen1():\n" +
181+
" x = 1\n" +
182+
" try:\n" +
183+
" while True:\n" +
184+
" x = yield x\n" +
185+
" except GeneratorExit:\n" +
186+
" print('gen1 exit')\n" +
187+
"\n" +
188+
"def gen2():\n" +
189+
" try:\n" +
190+
" yield from gen1()\n" +
191+
" except GeneratorExit:\n" +
192+
" print('gen2 exit')\n" +
193+
"\n" +
194+
"gen = gen2()\n" +
195+
"print(gen.send(None))\n" +
196+
"print(gen.send(2))\n" +
197+
"print(gen.send(3))\n" +
198+
"gen.close()\n";
199+
assertPrints("1\n2\n3\ngen1 exit\ngen2 exit\n", source);
200+
}
86201
}

‎graalpython/com.oracle.graal.python.test.integration/src/com/oracle/graal/python/test/integration/grammar/AsyncTests.java

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
package com.oracle.graal.python.test.integration.grammar;
2+
3+
import static com.oracle.graal.python.test.integration.PythonTests.assertPrints;
4+
5+
import org.junit.Test;
6+
7+
public class AsyncTests {
8+
@Test
9+
public void nativeCoroutine() {
10+
String source = "import asyncio\n" +
11+
"async def foo():\n" +
12+
" return 42\n" +
13+
"async def main():\n" +
14+
" print(await foo())\n" +
15+
"asyncio.run(main())";
16+
assertPrints("42\n", source);
17+
}
18+
19+
@Test
20+
public void asyncWith() {
21+
String source = "import asyncio\n" +
22+
"class AsyncContextManager:\n" +
23+
" async def __aenter__(self):\n" +
24+
" await asyncio.sleep(0.01)\n" +
25+
" print(\"entered\")\n" +
26+
" async def __aexit__(self, exc_type, exc_value, traceback):\n" +
27+
" await asyncio.sleep(0.01)\n" +
28+
" if exc_type:\n" +
29+
" print(\"exited exceptionally\")\n" +
30+
" else:\n" +
31+
" print(\"exited normally\")\n" +
32+
" return True\n" +
33+
"async def main(shouldRaise):\n" +
34+
" async with AsyncContextManager():\n" +
35+
" print(\"inside\")\n" +
36+
" if shouldRaise:\n" +
37+
" raise ValueError\n" +
38+
"asyncio.run(main(%s))";
39+
assertPrints("entered\ninside\nexited normally\n", String.format(source, "False"));
40+
assertPrints("entered\ninside\nexited exceptionally\n", String.format(source, "True"));
41+
}
42+
43+
@Test
44+
public void asyncWithExceptional() {
45+
String source = "import asyncio\n" +
46+
"class AsyncContextManager:\n" +
47+
" async def __aenter__(self):\n" +
48+
" await asyncio.sleep(0.01)\n" +
49+
" print(\"entered\")\n" +
50+
" async def __aexit__(self, exc_type, exc_value, traceback):\n" +
51+
" await asyncio.sleep(0.01)\n" +
52+
" print(\"exited\")\n" +
53+
" return False\n" + // don't handle exception
54+
"async def main(shouldRaise):\n" +
55+
" async with AsyncContextManager():\n" +
56+
" print(\"inside\")\n" +
57+
" if shouldRaise:\n" +
58+
" raise ValueError\n" +
59+
"try:\n" +
60+
" asyncio.run(main(%s))\n" +
61+
"except ValueError:\n" +
62+
" print(\"rethrew\")\n";
63+
assertPrints("entered\ninside\nexited\n", String.format(source, "False"));
64+
assertPrints("entered\ninside\nexited\nrethrew\n", String.format(source, "True"));
65+
}
66+
}

‎graalpython/com.oracle.graal.python.test.integration/src/com/oracle/graal/python/test/integration/grammar/ClassTests.java

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,4 +205,21 @@ public void multipleInheritance() {
205205
assertPrints("common\n", source);
206206
}
207207

208+
@Test
209+
public void classDecorator() {
210+
String source = "def wrapper(cls):\n" + //
211+
" orig_init = cls.__init__\n" + //
212+
" def new_init(self):\n" + //
213+
" print('wrapper')\n" + //
214+
" orig_init(self)\n" + //
215+
" cls.__init__ = new_init\n" + //
216+
" return cls\n" + //
217+
"@wrapper\n" + //
218+
"class Foo:\n" + //
219+
" def __init__(self):\n" + //
220+
" print('Foo')\n" + //
221+
"Foo()\n";
222+
assertPrints("wrapper\nFoo\n", source);
223+
}
224+
208225
}

‎graalpython/com.oracle.graal.python.test.integration/src/com/oracle/graal/python/test/integration/grammar/TryTests.java

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,4 +290,49 @@ public void testExceptionState6() {
290290
"print(repr(sys.exc_info()[1]))\n";
291291
assertPrints("None\nNone\n", source);
292292
}
293+
294+
@Test
295+
public void testNamedExceptionDeleted() {
296+
String source = "ex = 42\n" +
297+
"try:\n" +
298+
" raise NameError\n" +
299+
"except BaseException as ex:\n" +
300+
" pass\n" +
301+
"try:\n" +
302+
" print(ex)\n" +
303+
" print(\"expected NameError\")\n" +
304+
"except NameError:\n" +
305+
" print(\"hit NameError\")\n";
306+
assertPrints("hit NameError\n", source);
307+
}
308+
309+
@Test
310+
public void testNamedExceptionNotDeleted() {
311+
String source = "ex = 42\n" +
312+
"try:\n" +
313+
" print(\"nothing thrown\")\n" +
314+
"except BaseException as ex:\n" +
315+
" pass\n" +
316+
"try:\n" +
317+
" print(ex)\n" +
318+
"except NameError:\n" +
319+
" print(\"hit unexpected NameError\")\n";
320+
assertPrints("nothing thrown\n42\n", source);
321+
}
322+
323+
@Test
324+
public void testNamedExceptionDeletedByHandler() {
325+
String source = "ex = 42\n" +
326+
"try:\n" +
327+
" raise NameError\n" +
328+
"except BaseException as ex:\n" +
329+
" print(\"deleting exception\")\n" +
330+
" del ex\n" +
331+
"try:\n" +
332+
" print(ex)\n" +
333+
" print(\"expected NameError\")\n" +
334+
"except NameError:\n" +
335+
" print(\"hit NameError\")\n";
336+
assertPrints("deleting exception\nhit NameError\n", source);
337+
}
293338
}

0 commit comments

Comments
 (0)