Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proofread blog posts #34

Merged
merged 3 commits into from
Jan 29, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# rustpython.github.io
# [rustpython.github.io](https://rustpython.github.io)

<img src="./logo.png" width="125" height="125" align="right" />

Expand Down Expand Up @@ -67,4 +67,4 @@ The color scheme is that of the RustPython Logo:

![#306998](https://via.placeholder.com/15/306998/000000?text=+) `#306998` or `rgb(48, 105, 152)`

![#ffd43b](https://via.placeholder.com/15/ffd43b/000000?text=+) `#ffd43b` or `rgb(255, 212, 59)`
![#ffd43b](https://via.placeholder.com/15/ffd43b/000000?text=+) `#ffd43b` or `rgb(255, 212, 59)`
4 changes: 2 additions & 2 deletions _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
# in the templates via {{ site.myvariable }}.

title: "RustPython"
description: "An open source Python-3 (CPython >= 3.5.0) Interpreter written in Rust 🐍 😱 🤘"
description: "An open source Python 3 (CPython >= 3.5.0) interpreter written in Rust 🐍 😱 🤘"
# baseurl: "/" # the subpath of your site, e.g. /blog
url: "https://rustpython.github.io" # the base hostname & protocol for your site, e.g. http://example.com
github_username: RustPython
Expand All @@ -29,7 +29,7 @@ docs: https://github.com/RustPython/docs/
gitter: https://gitter.im/rustpython/Lobby
show_excerpts: true
contributor_excerpt: "" # TODO: write something here, goes right under "Contributors" heading
blog-intro: Create an issue if you read something wrong. Edit posts or create new ones via PR on <a target="_blank" href="https://github.com/RustPython/rustpython.github.io">github.com/RustPython/rustpython.github.io</a>
blog-intro: Create an issue if you see something wrong. Edit posts or create new ones via PR on <a target="_blank" href="https://github.com/RustPython/rustpython.github.io">github.com/RustPython/rustpython.github.io</a>

navigation:
- title: Blog
Expand Down
26 changes: 13 additions & 13 deletions _posts/2020-03-12-thing-explainer-parser.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,21 @@ date: 2020-04-02 11:34:01 -0400

This post goes over the RustPython parser. You can see the source code at [RustPython/parser/](https://github.com/RustPython/RustPython/tree/master/parser).

When you write code in python and run it, an interpreter, such as the RustPython interpreter, acts as the translator between you and your machine.
When you write code in Python and run it, an interpreter, such as the RustPython interpreter, acts as the translator between you and your machine.

The interpreter has the job of turning your human code into byte code that a python virtual machine can run. Bytecode is an intermediate code between source code and machine code. This makes it portable across multiple hardware and operating systems. Bytecode "works" as long as you implement a virtual machine(vm) that can run it. There is a performance penalty for this flexibility. RustPython has a vm under [RustPython/vm/](https://github.com/RustPython/RustPython/tree/master/vm). Other posts, will go into the details of that vm but now let's figure out how to turn code into bytecode.
The interpreter has the job of turning your human code into bytecode that a Python virtual machine can run. Bytecode is an intermediate code between source code and machine code. This makes it portable across multiple hardware and operating systems. Bytecode "works" as long as you implement a virtual machine (vm) that can run it. There is a performance penalty for this flexibility. RustPython has a vm under [RustPython/vm/](https://github.com/RustPython/RustPython/tree/master/vm). Other posts will go into the details of that vm but now let's figure out how to turn code into bytecode.


## How does bytecode look like
## What bytecode looks like

Seeing is believing. To see what bytecode looks like, you can use a Python module called [`dis`](https://docs.python.org/3/library/dis.html). dis stands for disassembler. You can write source code then see how its bytecode looks like. Here is an example:
Seeing is believing. To see what bytecode looks like, you can use a Python module called [`dis`](https://docs.python.org/3/library/dis.html). "dis" is short of for _dis_assembler. You can write source code then see how its bytecode looks like. Here is an example:

![bytecode](/assets/media/bytecode.jpg)


## How RustPython turns your code to bytecode
## How RustPython turns your code into bytecode

Here are the main steps that RustPython currently does:
Here are the main steps that RustPython currently goes through:
- parse the line of source code into tokens
- determine if the tokens have a valid syntax
- create an Abstract Syntax Tree (AST)
Expand All @@ -31,7 +31,7 @@ This list of steps introduces some new concepts like: tokens and abstract syntax

### Step 1: parsing source code into tokens

The fastest way to understand what tokens are, is to see them. Conveniently, Python comes with a [tokenizer](https://docs.python.org/3/library/tokenize.html). Here is what happen if I run the tokenizer on the function that I created earlier.
The fastest way to understand what tokens are, is to see them. Conveniently, Python comes with a [tokenizer](https://docs.python.org/3/library/tokenize.html). Here is what happens if I run the tokenizer on the function that I created above.

`$ python -m tokenize file.py`

Expand All @@ -46,20 +46,20 @@ def add(x,y):
![tokenzizing](/assets/media/tokenizing.jpg)


A picture IS worth a thousand word 😛 Those are the tokens. They are the basic "units" in the programming language. They are the keywords and operators that you typed. Even new lines and identations count.
A picture IS worth a thousand words 😛 Those are the tokens. They are the basic "units" of the programming language. They are the keywords and operators that you typed. Even new lines and identation count.

If you want to sound fancy:
- The tokens are the basic "lexical components"
- The parsing process is called "lexical analysis"
- The thing that does the process is a "lexer"
- The thing that does this is a "lexer"

Here is the link to the RustPython lexer.

**`RustPython/parser/lexer.rs`** >>
[source code](https://github.com/RustPython/RustPython/blob/master/parser/src/lexer.rs)


If you want dive into the details of lexical analysis, check out [Python in a nutshell / Lexical structure](https://learning.oreilly.com/library/view/python-in-a/9781491913833/ch03.html#python_language-id00003)
If you want to dive into the details of lexical analysis, check out [Python in a nutshell / Lexical structure](https://learning.oreilly.com/library/view/python-in-a/9781491913833/ch03.html#python_language-id00003)


### Step 2 : determine if the tokens are valid syntax
Expand All @@ -70,8 +70,8 @@ In the previous step, if you add random stuff to your function and tokenize it,

So don't hate on the whole interpreter when you get error messages! or at least don't hate on the tokenizer!

To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined "grammar" or set of rules. The official reference is on [this link](https://docs.python.org/3/reference/grammar.html). There, you will find a machine readable file. You may read a book to know the rules of python, but words are too "fluffy", an algorithm that verifies if rules are followed needs a very strict set of rules encoded in a file. [This video](https://www.youtube.com/watch?v=KGMFvy2d5OI) explains the Python grammar and the file's notation.
As the presenter puts it, this is the spirit of the beast (python) and it is only ~10KB 😭 (compare that to the size of python books you had to read!)
To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined "grammar" or set of rules. The official reference is on [this link](https://docs.python.org/3/reference/grammar.html). There, you will find a machine readable file. You may read a book to know the rules of Python, but words are too "fluffy", an algorithm that verifies if the rules are followed needs a very strict set of rules encoded in a file. [This video](https://www.youtube.com/watch?v=KGMFvy2d5OI) explains the Python grammar and the file's notation.
As the presenter puts it, this is the spirit of the beast (Python) and it is only ~10KB 😭 (compare that to the size of the Python books you had to read!)

So, we have the rules or grammar of a programming language in a machine encoded format... now we need to write something that verifies that those rules were followed... This sounds like something that other people could use and like something that should exist as an open source project! 🤔

Expand All @@ -91,7 +91,7 @@ You can do:

## Recap 🥴 🥵

As a recap, when you write a line of python code and "run it", here is what the RustPython interpreter does:
As a recap, when you write a line of Python code and "run it", here is what the RustPython interpreter does:

**INPUT: your code** (in `file.py` or interactive shell)
⬇️ parse the line of source code into tokens
Expand Down
47 changes: 23 additions & 24 deletions _posts/2020-04-05-how-to-contribute-by-cpython-unittest.markdown
Original file line number Diff line number Diff line change
@@ -1,45 +1,44 @@
---
layout: post
title: "How to contribute to RustPython by CPython unittest"
title: "How to contribute to RustPython using CPython's unit tests"
date: 2020-04-05 01:45:00 +0900
categories: [guideline, featured]
permalink: guideline/2020/04/04/how-to-contribute-by-cpython-unittest.html
---

At the very end of 2019, we finally reached one of the short-term goals: CPython unittest support. Due to this enhancement, finding CPython compatibility is easier than before.
Probably this will be the major source of contribution spots for the new contributors this year. Here is a simple guideline.
At the very end of 2019, we finally reached one of our short-term goals: CPython `unittest` support which makes finding CPython compatibility errors easier than ever.
This will probably be the major source of contributions for new contributors this year. Here is a simple guideline.

## Fix known compatibility bugs
Let's find an incompatibility issue and fix it.

1. See `Lib/test` directory of the project. There are many `test_` prefixed files like `test_unicode.py`.
1. Look at the `Lib/test` directory of the project. There are many `test_` prefixed files like `test_unicode.py`.
2. Try to open one of them. It might look just fine at a glance - but search for `TODO: RUSTPYTHON` in the files. There are tons of skipped, marked as an expected failure or commented out tests.
1. Alternatively, try looking at the [regression tests results]({% link pages/regression-tests-results.markdown %}) to find skipped or expected failure tests; some of them have
1. Alternatively, try looking at the [regression test results]({% link pages/regression-tests-results.markdown %}) to find skipped or expected failure tests; some of them have
notes for a way to resolve the issue.
3. Choose one or two interesting bugs. Remove the test blocker - skip, expectedFailure or comments.
3. Choose one or two interesting bugs. Remove the test blocker - `skip`, `expectedFailure` or comments.
4. Try to fix them.

Here is a quick tip to run single unittest file.
Here's how you run a single unittest file:

`$ RUSTPYTHONPATH=Lib cargo run --release Lib/test/test_unicode.py`
` $ RUSTPYTHONPATH=Lib cargo run --release Lib/test/test_unicode.py`

## Add a new unittest file
Because CPython unittest is not perfectly working in RustPython, we are doing this one by one with editings.
1. Download CPython source code.
2. Check out a specific version of CPython. For now, 3.8.2 is recommended. (We are testing against CPython 3.8 and 3.8.2 is the most recent version for now)
3. Copy a file from CPython `Lib/test`
4. Commit the file without editing. Specify copied CPython version to commit message.
Because CPython unittest doesn't work work perfectly in RustPython, we are adding test files one by one. Here's how:
1. Download the CPython source code with `git clone https://github.com/python/cpython.git`.
2. Check out a specific version of CPython. We test against CPython 3.8, so the most recent release of 3.8 ([currently 3.8.7](https://www.python.org/doc/versions/)) is recommended.
3. Copy a file from CPython's `Lib/test`
4. Commit the file without editing it. Specify the CPython version you copied from in the commit message.
5. Try to edit it until it runs without a crash or failure.
6. Commit the changes to make it run. This is the core contribution.

Because RustPython is not perfect, "try to edit it until it runs" doesn't mean to make 100% successful running. The common editing methods here:
1. At least it must be able to start to run the test. Fix the test code or bug until it runs at least a single unit of the test. Typically, unimplemented stdlib or missing files of unittest can make issues. Sometimes RustPython bugs make issues too.
2. If any test is not loadable by `SyntaxError`, that part is required to be commented out.
3. If any test leads to a crash of RustPython, this code is not possible to run. Mark the test to skip.
4. If any test is run but fails, this is an incompatibility issue. Mark the test as an expected failure.
Because RustPython is not perfect, "try to edit it until it runs" doesn't mean to make it run 100% of the tests successfully. The common methods to make the test file pass are:
1. It must at least be able to start to run the tests. Fix the test code or bug until it runs at least a single unit of the test. Typically, unimplemented stdlib or missing files in `unittest` can cause issues. Sometimes RustPython bugs cause issues too.
2. If any test can't be loaded because of a `SyntaxError`, you'll have to comment that part out.
3. If any test leads to a crash of RustPython, this code can't be run. Mark the test with `@unittest.skip('TODO: RUSTPYTHON')` to skip it.
4. If any test runs but fails, this is an incompatibility issue. Mark the test as an expected failure with `@unittest.expectedFailure`.

We prefer the reversed order of upper methods. The later the more strict so easy to detect any progress or regression.
When we temporarily disable parts of unittest due to RustPython caveats, we mark them to find it out easily later. Please check the examples below or search for `TODO: RUSTPYTHON` in `Lib/test` directory to check actual usage.
We prefer the reversed order of above methods. The later, the more strict, so it's easier to detect any progress or regression.
When we temporarily disable parts of `unittest` due to RustPython caveats, we mark them to make it easier to find (and re-enable!) them later. Please see the examples below or search for `TODO: RUSTPYTHON` in the `Lib/test` directory to check actual usage.

Comment out:
```python
Expand All @@ -48,18 +47,18 @@ Comment out:
# def ... # commented out tests
```

skip:
`skip`:
```python
@unittest.skip("TODO: RUSTPYTHON")
def ... # skipped tests
```

expectedFailure:
`expectedFailure`:
```python
# TODO: RUSTPYTHON
@unittest.expectedFailure
def ... # failed tests
```

## Development guide
For the general source of the development, please visit the [RustPython development guide](https://github.com/RustPython/RustPython/blob/master/DEVELOPMENT.md)
For a general introduction to RustPython development, please visit the [RustPython development guide](https://github.com/RustPython/RustPython/blob/master/DEVELOPMENT.md)
8 changes: 4 additions & 4 deletions index.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ installation:

goals:
- goal:
Full Python-3 environment entirely in Rust (not CPython bindings), with
Full Python 3 environment entirely in Rust (not CPython bindings), with
a clean implementation and no compatiblity hacks.
# TODO: integrate this into the "goals" boxes
progress:
Expand All @@ -53,10 +53,10 @@ There are many implementations of Python. For example:
- [PyPy](https://www.pypy.org/) (Python)
- [Stackless](http://www.stackless.com/)

Each of these implementations offer some benefits: Jython, for example, compiles Python source code to Java byte code, then routes it to the Java Virtual Machine. Because Python code is translated to Java byte code, it looks and feels like a true Java program at runtime and so it integrates well with Java applications.
Each of these implementations offer some benefits: Jython, for example, compiles Python 2 source code to Java byte code, then routes it to the Java Virtual Machine. Because Python code is translated to Java bytecode, it looks and feels like a true Java program at runtime and so it integrates well with Java applications.

IronPython is well-integrated with .NET, which means IronPython can use the .NET framework and Python libraries or vice versa.
IronPython is well-integrated with .NET, which means IronPython can use the .NET framework and Python 2 libraries or vice versa.

We want to unlock the same possibilities that Jython and IronPython enable, but for the Rust programming language. In addition, thanks to Rusts' minimal runtime, we're able to compile RustPython to WebAssembly and allow users to run their Python code easily in the browser.

Check the "learn more" section for an explainer of all those jargon-y words, or read the blog for more in-depth technical discussion.
Check out the "learn more" section for an explainer of all those jargon-y words, or read the blog for more in-depth technical discussion.
2 changes: 1 addition & 1 deletion manifest.webmanifest
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@
"start_url": ".",
"display": "standalone",
"theme_color": "#F74C00",
"description": "An open source Python-3 (CPython >= 3.5.0) Interpreter written in Rust 🐍 😱 🤘"
"description": "An open source Python 3 (CPython >= 3.5.0) Interpreter written in Rust 🐍 😱 🤘"
}