Skip to content

[WIP] "Preparation" and "Writing the script" for Basics and Workflow #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 45 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
4fd66fb
Update descale Akanesasu Shoujo example
LightArrowsEXE Jul 1, 2019
10dcfee
Wrote the "preparation" and "writing the script" parts for the basics…
LightArrowsEXE Jul 1, 2019
d97372b
Add semantics linefeeds to basics and workflow
LightArrowsEXE Jul 1, 2019
7c96009
Update basics and workflow, upload assets
LightArrowsEXE Jul 2, 2019
8fe35e2
Update basics-and-workflow.md
LightArrowsEXE Jul 2, 2019
a57527a
Minor expansion and fixes to basics and workflow
LightArrowsEXE Jul 2, 2019
76f9d41
Update x264 related links
cshmnyfy Sep 29, 2019
f091c6a
Mention chroma shift when resizing subsampled video
Lypheo Oct 2, 2019
3625196
Remove "officially published" note
FichteFoll Oct 3, 2019
90728e9
Remove the top notice entirely
FichteFoll Oct 3, 2019
ce1e619
Write basics of H.264 encoding guide
shssoichiro Oct 5, 2019
262a859
Fix typos
shssoichiro Oct 5, 2019
6b6cd58
Add recap section
shssoichiro Oct 6, 2019
e25873a
Fix the name of Apple's proprietary operating system
shssoichiro Oct 7, 2019
06c55f5
Part 1 of PR fixes
shssoichiro Oct 7, 2019
947fac7
Split lines better
shssoichiro Oct 7, 2019
3b60e39
More PR fixes
shssoichiro Oct 7, 2019
71c5889
Fix weird copy paste mistake
shssoichiro Oct 7, 2019
e88ef98
Remove outdated x264 build info
shssoichiro Oct 8, 2019
0c2da86
Fix spacing before headers
OrangeChannel Oct 9, 2019
cc752bc
More PR fixes
shssoichiro Oct 10, 2019
c85bcc0
Write basics of x264 guide (#41)
FichteFoll Oct 10, 2019
669589a
CONTRIBUTING: update ordered list formatting
OrangeChannel Oct 10, 2019
66349bb
CSS: small fix for portrait screens
OrangeChannel Oct 10, 2019
78fee3d
Add gitbook plug-in "katex" for math rendering
OrangeChannel Oct 9, 2019
c23eee6
CSS: update KaTeX formatting
OrangeChannel Oct 11, 2019
628b201
x264: Change bitrate math to TeX based rendering
OrangeChannel Oct 10, 2019
3cfe498
masking-limiting-etc: change mask formulas to TeX
OrangeChannel Oct 10, 2019
44ca844
resampling: change bicubic, spline, chroma-shifting math to TeX
OrangeChannel Oct 10, 2019
c43f03d
CONTRIBUTING: explain KaTeX syntax usage
OrangeChannel Oct 10, 2019
9af616e
video-artifacts: change footnote to TeX
OrangeChannel Oct 10, 2019
83ccede
Fix unnecessary delimiters
OrangeChannel Oct 11, 2019
063303c
CONTRIBUTING: get rid of delimiter info
OrangeChannel Oct 11, 2019
56f785e
Use katex plugin to pre-render math (#42)
FichteFoll Oct 12, 2019
0821b6b
Update x264 related links once again
cshmnyfy Oct 18, 2019
bde20b0
Replace ADC with working tracker
cshmnyfy Oct 18, 2019
03f6667
Mention exclusive parameters for x264 t_mod (fix #45)
kageru Oct 19, 2019
c769596
Add download link for x264 t_mod
kageru Oct 19, 2019
b4ab07a
Fixes
OrangeChannel Oct 28, 2019
7badaa3
Fix bitrate calculation
OrangeChannel Oct 29, 2019
c968f40
Units should use singular
FichteFoll Oct 30, 2019
93b43b9
Remove doubled words
cshmnyfy Dec 13, 2019
f96f737
Replace (flagged) YouTube videos with streamable embeds
FichteFoll Feb 5, 2020
94d31da
Fix missing paranthesis
LightArrowsEXE Mar 2, 2020
001a5dc
Merge branch 'master' of https://github.com/LightArrowsEXE/guide.enco…
LightArrowsEXE Mar 2, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
247 changes: 242 additions & 5 deletions encoding/basics-and-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,245 @@

## Preparation

downloading a source, looking at the video, some decisions
(resolution(s) for the release, audio codec, group-specific
requirements)
It is important to have a good idea
of what you want the result of the encode
to be like.
What will your video source be?
What resolution should you encode it to?
Do you use AAC or FLAC?
8bit colors or 10bit colors?
All of these are things
that have to be accounted for
and decided early on.

Some groups have set standards.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except in germany. Because in germany having a standard set by a group implies that there is a group to be a part of to begin with.

For example, you may have a group
that always encodes their audio in AAC.
Others will always release their videos in 1080p.
Some may even mix it up
and use FLAC audio for 1080p releases,
but AAC for 720p.
They might even only go for
8bit colors in the latter case!
If you're unsure what standards a group has,
contact your group leader.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to have a sentence that reiterates how this guide focuses on best practices and should generally be preferred over sticking to some weird standards (scene releases any1?).


There are various sources
that are often used for the video.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "often" in this sentence makes it seem pretentious.

Suggested change
that are often used for the video.
that can be used for video and audio material.

You will most commonly work with websources
like Crunchyroll,
as those are easy to acquire,
or Blu-rays, for the improved quality and fixes.
As a beginner, you might not
have access to private trackers with BDMVs yet.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence seems out of context because you didn't explain beforehand that BDMVs are frequently uploaded to private trackers.

A good alternative until then
is to use the raws provided by groups like Reinforce:
those that have not been filtered,
just re-encoded from the BDMV.

## Writing the Script

imports, source filter (mention lsmash, ffms2), examples for resizing,
debanding, AA. with images if possible
In order to start filtering,
first you must learn how to write a VapourSynth script.

```Py
import vapoursynth
core = vs.core
```

First, you must import the VapourSynth module.
You can also import various other modules,
like the many "funcs"[^1] that people have written.
Here are some common examples:

```Py
import kagefunc as kgf
import fvsfunc as fvf
import havsfunc as haf
```

These will import various functions
that may prove useful for filtering,
like masks, descaling wrappers, anti-aliasing functions, etc.

Now, to start filtering,
we must first import a video clip.

```Py
src = core.ffms2.Source("source")
```
```Py
src = core.lsmas.LWLibavSource("source")
```

There are two common import filters available.
Those are L-SMASH and ffms2.
L-SMASH is typically used for m2ts files
(the kind you'll find in BMDVs),
while ffms2 is used for everything else.
The reason for this is that ffms2
can't accurately read m2ts files,
which L-SMASH can.
However, L-SMASH is slower than ffms2,
and some people prefer to be able to
index the source files faster.

It is common practice to call the source clip `src`.
This way it's easy to tell what the original video is,
and it can be referenced for some masks or for comparisons.

Let's now look at a couple of common filters
that you will find yourself using often.

```Py
src16 = fvf.Depth(src, 16)
```

The color depth of a clip
indicates the number of bits used
to give a pixel its color.
A lot of filters work with high bitdepths,
so this is typically set at the start of the script.
Your average `src` will be 8bit,
but you might occasionally run into 10bit sources as well.

```Py
trim = core.std.Trim(src, first=100, last=1000)
```
```Py
trim = src[100:1001]
```

`Trim` is used for trimming the video.
It's inclusive,
which means that it also adds the `last` frame given.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"includes", not "adds".

Another way to trim is by using Python Slices.
These are exclusive however,
so you need to take that into account.

Inclusive and exclusive work like this:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo it should be enough to just claim that the above two trims are equivalent (and skipp the following paragraphs), or do people have problems with the two terms "inclusive" and "exclusive"?


Let's say you have a number range.
This is what they'd look like with both methods:
```
Inclusive from 1 to 10:
1 2 3 4 5 6 7 8 9 10

Exclusive from 1 to 10:
1 2 3 4 5 6 7 8 9
```

Inclusive also **inclu**des
the final number in the given range,
whilst exclusive **exclu**des them.
Both of these have their pros and cons,
so it all comes down to preference.

```Py
scaled = core.resize.Spline36(src, width=1280, height=720)
```

`resize` has multiple "kernels"
that are used for [resampling](resampling.md#upsampling) the clip
to the given resolution.
Common resolutions are 1920x1080, 1600x900, 1280x720, 1024x576 and 848x480.
Nowadays most sources will be available in 1080p,
but there are advantages to downscaling,
like smaller filesizes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should mention the advantages of not downscaling.


You will often find grain or noise on a lot of sources.
You can get rid of that by denoising them.

![Manaria Friends — 04](images/basics_noise.png)

Look at her hair. There is a lot of grain on there, but some people may prefer that to be smooth instead.

```Py
import mvsfunc as mvf

denoise = mvf.BM3D(src, sigma=[4,2])
```

![Manaria Friends — 04 (denoised)](images/basics_denoised.png)

There are many denoisers available to use,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would start by explaining what a denoiser is.

and they all have their strong and weak points.
Some will have better detail retention,
whilst others are a lot faster.

Just be aware that getting rid of grain
will often result in more problems popping up,
like the very obvious banding in this case.

Banding is caused by gradients
breaking up during compression.
In this picture,
you can see the gradient breaking up
very clearly in her hair.
To fix this, you use a debanding filter.

```Py
import mvsfunc as mvf
import kagefunc as kgf

denoise = mvf.BM3D(src, sigma=[4,2])
deband = core.f3kdb.Deband(denoise, range=18, y=40, cb=32, cr=32, grainy=12, grainc=0, output_depth=16)
grain = kgf.adaptive_grain(deband, strength=0.1)
```
![Manaria Friends — 04 (debanded and grained)](images/basics_debanded.png)

The banding is now less obvious,
and has been further hidden by adding grain.

Grain is something that some people like,
and others hate with a passion.
There are various ways that it can be used,
ranging from adding a particular mood to a scene
to preventing other artifacts from showing up.
It's also common for encoders to
add their own grain after denoising.

![Fairy Tail — NCOP20 (cleaned)](images/basics_clean.png)

This frame looks fairly clean overall,
but the encoding may introduce some banding
in the darker areas of this frame.
To combat this, we're adding grain
to just the darker areas.

```Py
import kagefunc as kgf

grain = kgf.adaptive_grain(deband, strength=0.3, luma_scaling=8)
```
![Fairy Tail — NCOP20 (grained)](images/basics_grained.png)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a blank line above (and has a few too many below).




Another common issue is aliasing.
Aliasing an artifact that is commonly caused
by scaling or compression.

![Kimetsu no Yaiba — OP1](images/basics_aliasing.png)

Note the windows to the left.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't you describe aliasing with a bit more detail?

You'll want to use anti-aliasing filters
to deal with this.

```Py
import vsTAAmbk as taa

aa = taa.TAAmbk(src aatype='Eedi3', opencl=True)
```

![Kimetsu no Yaiba - OP1 (anti-aliased)](images/basics_anti-aliased.png)

There are a couple of different anti-aliasing filters,
some faster than others.
Try out lighter AA if you can first,
like `Nnedi3`.



## Encoding the Result

Expand All @@ -35,3 +265,10 @@ examples for qaac, flac
## Muxing

mkvtoolnix

***

[^1]: "funcs" are a combination of wrappers written by people to perform different tasks.
Most of these can be found in the [VapourSynth Database][vsdb]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Iirc you cannot use linebreaks in footnotes. @OrangeChannel?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, that's correct.


[vsdb]: http://vsdb.top/
Binary file added encoding/images/basics_aliasing.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added encoding/images/basics_anti-aliased.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added encoding/images/basics_clean.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added encoding/images/basics_debanded.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added encoding/images/basics_denoised.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added encoding/images/basics_grained.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added encoding/images/basics_noise.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified encoding/images/descale_akanesasu_rescaled.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified encoding/images/descale_akanesasu_src.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.