Skip to content

Commit 74798ff

Browse files
committed
add code highlight, fixes #9
1 parent 5108a4b commit 74798ff

File tree

5 files changed

+219
-142
lines changed

5 files changed

+219
-142
lines changed

Diff for: _includes/head.html

+4-1
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,12 @@
66
<meta name="description" content="{{site.description}}">
77
<!-- google fonts -->
88
<link defer async href="https://fonts.googleapis.com/css?family=Fira+Sans|Sen:800&display=swap" rel="stylesheet">
9-
<!-- stylesheet and webmanifest -->
9+
<!-- stylesheet -->
1010
<link defer async rel="stylesheet" href="{{ "/assets/style.css" | relative_url }}">
1111
<link defer async rel="stylesheet" href="{{ "/assets/media.css" | relative_url }}">
12+
<link defer async rel="stylesheet" href="{{ "/assets/monokai.css" | relative_url }}">
13+
14+
<!-- webmanifest -->
1215
<link rel="manifest" href="{{site.baseurl}}/manifest.webmanifest">
1316

1417
{%- seo -%}

Diff for: _layouts/post.html

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
<!-- header section -->
66
<section class="bg-light">
7-
<div class="w-80 m-auto">
7+
<div class="w-md-80 m-auto">
88
{%- assign date_format = site.minima.date_format | default: "%b %-d, %Y" -%}
99
<mark class="post-date">{{ page.date | date: date_format }}</mark>
1010

Diff for: _posts/2020-03-12-thing-explainer-parser.markdown

+14-15
Original file line numberDiff line numberDiff line change
@@ -32,13 +32,12 @@ This list of steps introduces some new concepts like: tokens and abstract syntax
3232
### Step 1: parsing source code into tokens
3333

3434
The fastest way to understand what tokens are, is to see them. Conveniently, Python comes with a [tokenizer](https://docs.python.org/3/library/tokenize.html). Here is what happen if I run the tokenizer on the function that I created earlier.
35-
`$ python -m tokenize file.py`
36-
3735

36+
`$ python -m tokenize file.py`
3837

3938
`file.py` has the function that I used in the previous example.
4039

41-
```
40+
```python
4241
def add(x,y):
4342
return x+y
4443
```
@@ -51,7 +50,7 @@ A picture IS worth a thousand word 😛 Those are the tokens. They are the basic
5150

5251
If you want to sound fancy:
5352
- The tokens are the basic "lexical components"
54-
- The parsing process is called "Lexical Analysis"
53+
- The parsing process is called "lexical analysis"
5554
- The thing that does the process is a "lexer"
5655

5756
Here is the link to the RustPython lexer.
@@ -71,34 +70,34 @@ In the previous step, if you add random stuff to your function and tokenize it,
7170

7271
So don't hate on the whole interpreter when you get error messages! or at least don't hate on the tokenizer!
7372

74-
To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined "grammar" or set of rules. The official reference is on [this link](https://docs.python.org/3/reference/grammar.html). There, you will find a machine readable file. You may read a book to know the rules of python, but words are too "fluffy", the machine needs a very strict set of rules encoded in a file. [This video](https://www.youtube.com/watch?v=KGMFvy2d5OI) explains the notation and the Python grammar.
73+
To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined "grammar" or set of rules. The official reference is on [this link](https://docs.python.org/3/reference/grammar.html). There, you will find a machine readable file. You may read a book to know the rules of python, but words are too "fluffy", an algorithm that verifies if rules are followed needs a very strict set of rules encoded in a file. [This video](https://www.youtube.com/watch?v=KGMFvy2d5OI) explains the Python grammar and the file's notation.
7574
As the presenter puts it, this is the spirit of the beast (python) and it is only ~10KB 😭 (compare that to the size of python books you had to read!)
7675

7776
So, we have the rules or grammar of a programming language in a machine encoded format... now we need to write something that verifies that those rules were followed... This sounds like something that other people could use and like something that should exist as an open source project! 🤔
7877

7978
Sure enough, there is a whole Rust framework called `LALRPOP`. It takes the tokens generated by the lexer, verifies the syntax and turns the tokens into an AST (Abstract Syntax Tree). More information and a tutorial can be found in the [LALRPOP book](https://lalrpop.github.io/lalrpop/README.html).
8079

81-
RustPython does one nice extra thing on top of `LALRPOP `. It masks the errors and provides you with safer, nicer errors. You can see the code for this in `RustPython/parser/src/error.rs`
80+
RustPython does one nice extra thing on top of `LALRPOP`. It masks the errors and provides you with safer, nicer errors. You can see the code for this in `RustPython/parser/src/error.rs`
8281

8382
**Using RustPython to generate an AST**
8483

8584
You can do:
8685

87-
```
86+
```rust
8887
use rustpython_parser::{parser, ast};
8988
let python_source = "print('Hello world')";
9089
let python_ast = parser::parse_expression(python_source).unwrap();
9190
```
9291

9392
## Recap 🥴 🥵
9493

95-
As a recap, when you write a line of python code and "run it", here is what happens:
94+
As a recap, when you write a line of python code and "run it", here is what the RustPython interpreter does:
9695

97-
**your code** (in `file.py` or interactive shell)
98-
parse the line of source code into tokens
99-
determine if the tokens are valid syntax
100-
create an Abstract Syntax Tree (AST)
101-
compile the AST into bytecode
102-
**bytecode** (in `__pycache__/file.pyc` or in memory)
96+
**INPUT: your code** (in `file.py` or interactive shell)
97+
⬇️ parse the line of source code into tokens
98+
⬇️ determine if the tokens are valid syntax
99+
⬇️ create an Abstract Syntax Tree (AST)
100+
⬇️ compile the AST into bytecode
101+
**OUTPUT: bytecode** (in `__pycache__/file.pyc` or in memory)
103102

104-
The compiler is under **`RustPython/compiler`** we'll dive into the details in a next post. In the meantime, check out the parser source code in [RustPython/parser/](https://github.com/RustPython/RustPython/tree/master/parser).
103+
The compiler is under **`RustPython/compiler`**. Keep an eye on the blog for a future post about the details or the compiler. In the meantime, check out the parser source code in [RustPython/parser/](https://github.com/RustPython/RustPython/tree/master/parser).

Diff for: assets/monokai.css

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
.highlight pre { background-color: #272822; }
2+
.highlight .hll { background-color: #272822; }
3+
.highlight .c { color: #75715e } /* Comment */
4+
.highlight .err { color: #960050; background-color: #1e0010 } /* Error */
5+
.highlight .k { color: #66d9ef } /* Keyword */
6+
.highlight .l { color: #ae81ff } /* Literal */
7+
.highlight .n { color: #f8f8f2 } /* Name */
8+
.highlight .o { color: #f92672 } /* Operator */
9+
.highlight .p { color: #f8f8f2 } /* Punctuation */
10+
.highlight .cm { color: #75715e } /* Comment.Multiline */
11+
.highlight .cp { color: #75715e } /* Comment.Preproc */
12+
.highlight .c1 { color: #75715e } /* Comment.Single */
13+
.highlight .cs { color: #75715e } /* Comment.Special */
14+
.highlight .ge { font-style: italic } /* Generic.Emph */
15+
.highlight .gs { font-weight: bold } /* Generic.Strong */
16+
.highlight .kc { color: #66d9ef } /* Keyword.Constant */
17+
.highlight .kd { color: #66d9ef } /* Keyword.Declaration */
18+
.highlight .kn { color: #f92672 } /* Keyword.Namespace */
19+
.highlight .kp { color: #66d9ef } /* Keyword.Pseudo */
20+
.highlight .kr { color: #66d9ef } /* Keyword.Reserved */
21+
.highlight .kt { color: #66d9ef } /* Keyword.Type */
22+
.highlight .ld { color: #e6db74 } /* Literal.Date */
23+
.highlight .m { color: #ae81ff } /* Literal.Number */
24+
.highlight .s { color: #e6db74 } /* Literal.String */
25+
.highlight .na { color: #a6e22e } /* Name.Attribute */
26+
.highlight .nb { color: #f8f8f2 } /* Name.Builtin */
27+
.highlight .nc { color: #a6e22e } /* Name.Class */
28+
.highlight .no { color: #66d9ef } /* Name.Constant */
29+
.highlight .nd { color: #a6e22e } /* Name.Decorator */
30+
.highlight .ni { color: #f8f8f2 } /* Name.Entity */
31+
.highlight .ne { color: #a6e22e } /* Name.Exception */
32+
.highlight .nf { color: #a6e22e } /* Name.Function */
33+
.highlight .nl { color: #f8f8f2 } /* Name.Label */
34+
.highlight .nn { color: #f8f8f2 } /* Name.Namespace */
35+
.highlight .nx { color: #a6e22e } /* Name.Other */
36+
.highlight .py { color: #f8f8f2 } /* Name.Property */
37+
.highlight .nt { color: #f92672 } /* Name.Tag */
38+
.highlight .nv { color: #f8f8f2 } /* Name.Variable */
39+
.highlight .ow { color: #f92672 } /* Operator.Word */
40+
.highlight .w { color: #f8f8f2 } /* Text.Whitespace */
41+
.highlight .mf { color: #ae81ff } /* Literal.Number.Float */
42+
.highlight .mh { color: #ae81ff } /* Literal.Number.Hex */
43+
.highlight .mi { color: #ae81ff } /* Literal.Number.Integer */
44+
.highlight .mo { color: #ae81ff } /* Literal.Number.Oct */
45+
.highlight .sb { color: #e6db74 } /* Literal.String.Backtick */
46+
.highlight .sc { color: #e6db74 } /* Literal.String.Char */
47+
.highlight .sd { color: #e6db74 } /* Literal.String.Doc */
48+
.highlight .s2 { color: #e6db74 } /* Literal.String.Double */
49+
.highlight .se { color: #ae81ff } /* Literal.String.Escape */
50+
.highlight .sh { color: #e6db74 } /* Literal.String.Heredoc */
51+
.highlight .si { color: #e6db74 } /* Literal.String.Interpol */
52+
.highlight .sx { color: #e6db74 } /* Literal.String.Other */
53+
.highlight .sr { color: #e6db74 } /* Literal.String.Regex */
54+
.highlight .s1 { color: #e6db74 } /* Literal.String.Single */
55+
.highlight .ss { color: #e6db74 } /* Literal.String.Symbol */
56+
.highlight .bp { color: #f8f8f2 } /* Name.Builtin.Pseudo */
57+
.highlight .vc { color: #f8f8f2 } /* Name.Variable.Class */
58+
.highlight .vg { color: #f8f8f2 } /* Name.Variable.Global */
59+
.highlight .vi { color: #f8f8f2 } /* Name.Variable.Instance */
60+
.highlight .il { color: #ae81ff } /* Literal.Number.Integer.Long */
61+
62+
.highlight .gh { } /* Generic Heading & Diff Header */
63+
.highlight .gu { color: #75715e; } /* Generic.Subheading & Diff Unified/Comment? */
64+
.highlight .gd { color: #f92672; } /* Generic.Deleted & Diff Deleted */
65+
.highlight .gi { color: #a6e22e; } /* Generic.Inserted & Diff Inserted */

0 commit comments

Comments
 (0)