diff --git a/website/src/pages/internals.mdx b/website/src/pages/internals.mdx new file mode 100644 index 0000000..5b88f3a --- /dev/null +++ b/website/src/pages/internals.mdx @@ -0,0 +1,245 @@ +--- +title: Recap +layout: ../layouts/doc.astro +--- + +import { MiniRepl } from "../components/MiniRepl"; +import { Icon } from "../components/Icon"; +import Box from "../components/Box.astro"; +import DocLayout from "../layouts/doc.astro"; + +# How kabelsalat works + +This site explains how kabelsalat works under the hood. **You don't need to know this to use it!** + +## Overview + +The understand how kabelsalat works, we need to understand how the user code +is transformed to produce audio in the end. These are the basic steps: + +1. User evaluates code +2. The code creates a `Node` representing the audio graph +3. The `Node` is compiled to an optimized chunk audio callback code +4. The code is sent to the audio thread +5. The audio thread creates the required `AudioNode`'s and runs the code for each audio sample + +## From Code to Graph + +Let's look at this example: + +JSON.stringify(n,null,2)) +.out()`} + client:only="solid" +/> + +If you run the above patch, the `log` method will output this: + +```json +{ + "type": "Sine", + "ins": [ + { + "type": "n", + "value": 200, + "ins": [] + } + ] +} +``` + +Here we see the basic structure of a `Node` in kabelsalat: + +- type: which kind of node is it? +- ins: which `Node`'s are connected to it? + +We can also see that the number passed to `sine` is converted to a `Node` of type `n`, which is the constant number node. +It is special because it also has a `value` property. + +The clever thing about this data structure is that you only need to know the last `Node` (before .out) to know the whole graph! + +### Method Chaining vs Function Calling + +kabelsalat relies heavily on method chaining, which is only syntax sugar for regular function calls. These 2 variants are equivalent: + +```js +mul(sine(200), 0.5); +// is equal to +sine(200).mul(0.5); +``` + +This will work for any node (except `out`, which is a special case). More generally, you can say + +``` +a.x(b) = x(a,b) +``` + +This syntax sugar reduces the level of parenthesis nesting alot and is also more natural way to express arithmetic: + +```js +3 * 4; // infix notation +// = +n(3).mul(4); // method chaining +// = +mul(3, 4); // polish notation +``` + +So you could say the method chaining syntax acts as an infix operator replacement. + +## Compiling the Graph + +In the next stage, we pass the graph to the compiler, which turns it into a sequence of instructions that runs well on the audio thread. +Before actual compilation there is preprocessing step: + +1. flatten the `Node`s +2. apply topological sort + +### Flatten Node's + +This step takes the last `Node` and turns it into an Array of `Node`'s where, the `ins` are replaced with indices: + + + +This is the console output: + +```json +[ + // 0 + { + "type": "mul", + "ins": ["1", "3"] + }, + // 1 + { + "type": "Sine", + "ins": ["2"] + }, + // 2 + { + "type": "n", + "ins": [], + "value": 200 + }, + // 3 + { + "type": "n", + "ins": [], + "value": 0.5 + } +] +``` + +This representation of the graph contains the same info, but structured in a way suitable for... + +### Topological Sort + +[Topological Sorting](https://en.wikipedia.org/wiki/Topological_sorting) makes sure the `Node`s are ordered so that nodes without dependencies are first. + + + +This gives us: + +```json +["3", "2", "1", "0"] +``` + +Reordering the flattened nodes in this order: + +```json +[ + // 3 + { + "type": "n", + "ins": [], + "value": 0.5 + } + // 2 + { + "type": "n", + "ins": [], + "value": 200 + }, + // 1 + { + "type": "Sine", + "ins": ["2"] + }, + // 0 + { + "type": "mul", + "ins": ["1", "3"] + }, +] +``` + +We can now observe that the `ins` of each `Node` preceed the `Node` itself, which means it's time to generate some code.. + +### Compilation + +The compiler can now generate a chunk of code, where each line roughly corresponds to the value of one Node: + + + +The compiler gives us this compiled `unit`: + +```js +{ + "src": "const n1 = nodes[0].update(200, 0); /* Sine */\nconst n0 = n1 * 0.5; /* Sine * n */\nreturn [(0*0.3), (0*0.3)]", + "audioThreadNodes": ["Sine"] +} +``` + +Let's make the `src` more readable: + +```js +const n1 = nodes[0].update(200, 0); /* Sine */ +const n0 = n1 * 0.5; /* Sine * n */ +return [0 * 0.3, 0 * 0.3]; +``` + +This is now the actual code that runs on the audio thread. + +- The `n` variable numbers correspond to the original indices of the topologically sorted Array. +- To shorten the output, the compiler inlines arithmetic Node's (add, mul,...) and constant numbers (n). +- For `AudioNode`'s like `Sine`, the compiler inserts an `update` call with its inputs as arguments (they could also be variables). +- In the end we just get 2 numbers for the left and right speaker. + +## Running the Compiled Code + +The `unit` returned by the compiler is sent to our `GraphWorklet` which hosts an instance of `AudioGraph`. +The `AudioGraph` creates an `AudioNode` for each `type` in the `audioThreadNodes` Array. +Each `AudioNode` has the ability to keep its own state, for example, `Sine` will keep track of its phase. +The compiled `src` is used as the body of the `_genSample` function, which is ultimately called in `GraphWorklet.process` for each sample. + +## Credits + +This is the basic mechanism! It is heavily influenced by the amazing [noisecraft](https://noisecraft.app/) by Maxime Chevalier-Boisvert. +The audio DSP code is mostly borrowed as is, but I've written a new compiler + implemented feedback and multichannel expansion + removed UI related features. +You can read more about the specific changes in this [discussion](https://github.com/maximecb/noisecraft/discussions/109) + +## More + +What I haven't described so far: + +- how feedback is resolved +- how multichannel expansion works +- how the viz works + +I might write about it in the future