Skip to content

Commit

Permalink
Update files
Browse files Browse the repository at this point in the history
  • Loading branch information
leikareipa committed Dec 4, 2024
1 parent 81f4230 commit f233a2a
Show file tree
Hide file tree
Showing 9 changed files with 525 additions and 4 deletions.
72 changes: 70 additions & 2 deletions blog/briefly-evaluating-qwq-preview/content.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Alibaba has somewhat stunned the AI community by their releasing a relatively capable o1 competitor as an open model, mere months after the unveiling of o1. The model is called QwQ Preview, or QwQ-32B-Preview if you will. You can learn more about it on google.

Let's briefly evaluate its coding chops, using the Q4_K_M variant running via Ollama.
Let's briefly evaluate its coding chops, using the Q4_K_M variant running via Ollama. Note that I won't be comparing it to o1 directly since my OAI API credit is at 0 and I don't want to put more in there right now.

## Add comments to C++ code

Expand Down Expand Up @@ -707,6 +707,74 @@ int main(void)
A small ray tracer. Qwen 2.5 Coder gets further than QwQ; neither gets it right.
## Spatial reasoning
<x-prompt>
<p>
This code draws the likeness of a 1990s ray traced scene of two spheres. It doesn't quite look physically correct though.
</p>
> Code
```js
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Simple Scene</title>
</head>
<body>
<canvas id="sceneCanvas" width="400" height="300"></canvas>
<script>
const canvas = document.getElementById('sceneCanvas');
const ctx = canvas.getContext('2d');
// Draw red sphere in the middle
ctx.fillStyle = 'red';
ctx.beginPath();
ctx.arc(200, 150, 50, 0, Math.PI * 2);
ctx.fill();
// Draw blue sphere to the left
ctx.fillStyle = 'blue';
ctx.beginPath();
ctx.arc(100, 170, 48, 0, Math.PI * 2);
ctx.fill();
// Draw reflection of red sphere on blue sphere (simplified)
ctx.globalAlpha = 0.5; // Semi-transparent for the reflection effect
ctx.beginPath();
ctx.arc(138, 206, 19, 0, Math.PI * 2);
ctx.fillStyle = 'red';
ctx.fill();
</script>
</body>
</html>
```
</x-prompt>

> Original
![{iframe}{inline-class:model-response}](./spat-o.html)

> QwQ Preview 32B
![{iframe}{inline-class:model-response}](./spat-qwq.html)

> Claude 3.5 Sonnet (October 2024)
![{iframe}{inline-class:model-response}](./spat-s35.html)

> Mistral Large 123B
![{iframe}{inline-class:model-response}](./spat-misla.html)

> Grok 2
![{iframe}{inline-class:model-response}](./spat-grok2.html)

> Qwen 2.5 72B
![{iframe}{inline-class:model-response}](./spat-q25.html)

> Llama 3.2 Vision 90B
![{iframe}{inline-class:model-response}](./spat-llama-vision-90.html)

In the original image, the reflection of the red sphere on the blue sphere is incorrectly placed. QwQ was the only model that clearly understood this and fixed it. Grok 2 might also have had a sniff of this issue; the other models not so much.

## Use a GUI framework

<x-prompt>
Expand Down Expand Up @@ -781,5 +849,5 @@ Both QwQ and Qwen 2.5 Coder added a game-over condition, though I'd say QwQ's is

## Conclusions

QwQ Preview does a decent job, often rising above Qwen 2.5 Coder and sometimes matching Claude 3.5 Sonnet; but on the flip side eating up quite a few more tokens. Interesting to see the full QwQ down the road.
QwQ Preview does a decent job, often rising above Qwen 2.5 Coder and sometimes even Claude 3.5 Sonnet; but on the flip side eating up quite a few more tokens. Interesting to see the full QwQ down the road.

66 changes: 64 additions & 2 deletions blog/briefly-evaluating-qwq-preview/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
<post-date date="4 December 2024"></post-date>
<dokki-topic title="Briefly evaluating QwQ Preview">
<p>Alibaba has somewhat stunned the AI community by their releasing a relatively capable o1 competitor as an open model, mere months after the unveiling of o1. The model is called QwQ Preview, or QwQ-32B-Preview if you will. You can learn more about it on google.</p>
<p>Let's briefly evaluate its coding chops, using the Q4_K_M variant running via Ollama.</p>
<p>Let's briefly evaluate its coding chops, using the Q4_K_M variant running via Ollama. Note that I won't be comparing it to o1 directly since my OAI API credit is at 0 and I don't want to put more in there right now.</p>
<dokki-subtopic title="Add comments to C++ code">
<x-prompt>
<p>
Expand Down Expand Up @@ -778,6 +778,68 @@
<dokki-image src="./rend-qc.png" width="479" height="320" inline-class="model-response" thumbnail-src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAC4jAAAuIwF4pT92AAAB2UlEQVR4nJ2STctxURSG9zk+MhBCScZS5kaUAXO/wU8wkImRH2BkIhn7CwoTEwnJRIlEKR8R8h2539Z6nfOch+cZvO+u1dnn7LWv+15rHXG9XvE/cblcOGi1Wi3EYjH4/X6I96TT6YTz+fwriM4PhwOOxyPvi8UinE4nhBCQJAmCPirJt9uNFR+PB8PfYST0fD75nFY2m2WQTqfjkGUZgg4Jer/fsVgsEI1GWZUWuVBKIxitTqeDdDqNTCaDUCjEQAYJ8Tf6/b5awnA4hM1mQyqVYiebzQa73Y6fJJzP59XLVB6FLMvfgW63G4PBAOv1GsvlEvv9nt1st1vM53N2TdButwu73c6X9Ho9l0h7o9EIl8v1BYxEIiCX0+kUq9UKvV4PlUqFAfQ+Ho8ZXigU1H6pl1+hOCUhUa/XMZlMMJvNkEgkYLFYOIlKTyaTGI1GfKYAeZI/ACXlOymGw2HE4/GPJHpS80mQhuHz+T6GIL3yHA4H/4tCC9E2mO0LgWAwiGaziVqthlwuB4PB8OHKarWiXC6zsNBOSluOsqef1uv1wmw2w+PxqELaHJPJhFKphHa7/d3hv4YCJBHqcaPR+AJSKeRCm0jOqDdKO94HorQnEAigWq1y2X8AXP1GWIbOxsgAAAAASUVORK5CYII="><template #caption>Qwen 2.5 32B</template>
</dokki-image>
<p>A small ray tracer. Qwen 2.5 Coder gets further than QwQ; neither gets it right.</p>
</dokki-subtopic><dokki-subtopic title="Spatial reasoning">
<x-prompt>
<p>
This code draws the likeness of a 1990s ray traced scene of two spheres. It doesn't quite look physically correct though.
</p>


<dokki-code syntax="js"><template #caption>Code</template>
<template #code>
<pre>&lt;!DOCTYPE html&gt;
&lt;html lang=&quot;en&quot;&gt;
&lt;head&gt;
&lt;meta charset=&quot;UTF-8&quot;&gt;
&lt;title&gt;Simple Scene&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;canvas id=&quot;sceneCanvas&quot; width=&quot;400&quot; height=&quot;300&quot;&gt;&lt;/canvas&gt;
&lt;script&gt;
const canvas = document.getElementById('sceneCanvas');
const ctx = canvas.getContext('2d');

// Draw red sphere in the middle
ctx.fillStyle = 'red';
ctx.beginPath();
ctx.arc(200, 150, 50, 0, Math.PI * 2);
ctx.fill();

// Draw blue sphere to the left
ctx.fillStyle = 'blue';
ctx.beginPath();
ctx.arc(100, 170, 48, 0, Math.PI * 2);
ctx.fill();

// Draw reflection of red sphere on blue sphere (simplified)
ctx.globalAlpha = 0.5; // Semi-transparent for the reflection effect
ctx.beginPath();
ctx.arc(138, 206, 19, 0, Math.PI * 2);
ctx.fillStyle = 'red';
ctx.fill();
&lt;/script&gt;
&lt;/body&gt;
&lt;/html&gt;
</pre>
</template>
</dokki-code>

</x-prompt>
<dokki-iframe src="./spat-o.html" inline-class="model-response"><template #caption>Original</template>
</dokki-iframe>
<dokki-iframe src="./spat-qwq.html" inline-class="model-response"><template #caption>QwQ Preview 32B</template>
</dokki-iframe>
<dokki-iframe src="./spat-s35.html" inline-class="model-response"><template #caption>Claude 3.5 Sonnet (October 2024)</template>
</dokki-iframe>
<dokki-iframe src="./spat-misla.html" inline-class="model-response"><template #caption>Mistral Large 123B</template>
</dokki-iframe>
<dokki-iframe src="./spat-grok2.html" inline-class="model-response"><template #caption>Grok 2</template>
</dokki-iframe>
<dokki-iframe src="./spat-q25.html" inline-class="model-response"><template #caption>Qwen 2.5 72B</template>
</dokki-iframe>
<dokki-iframe src="./spat-llama-vision-90.html" inline-class="model-response"><template #caption>Llama 3.2 Vision 90B</template>
</dokki-iframe>
<p>In the original image, the reflection of the red sphere on the blue sphere is incorrectly placed. QwQ was the only model that clearly understood this and fixed it. Grok 2 might also have had a sniff of this issue; the other models not so much.</p>
</dokki-subtopic><dokki-subtopic title="Use a GUI framework">
<x-prompt>
<p>
Expand Down Expand Up @@ -845,7 +907,7 @@
</dokki-iframe>
<p>Both QwQ and Qwen 2.5 Coder added a game-over condition, though I'd say QwQ's is better. They both also implemented right-clicking to place flags, but since the UI framework only supports left-click, that can't be tested. Overall, I wouldn't call either version finished, but getting there.</p>
</dokki-subtopic><dokki-subtopic title="Conclusions">
<p>QwQ Preview does a decent job, often rising above Qwen 2.5 Coder and sometimes matching Claude 3.5 Sonnet; but on the flip side eating up quite a few more tokens. Interesting to see the full QwQ down the road.</p>
<p>QwQ Preview does a decent job, often rising above Qwen 2.5 Coder and sometimes even Claude 3.5 Sonnet; but on the flip side eating up quite a few more tokens. Interesting to see the full QwQ down the road.</p>
</dokki-subtopic></dokki-topic>

</dokki-topics>
Expand Down
66 changes: 66 additions & 0 deletions blog/briefly-evaluating-qwq-preview/spat-grok2.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Improved Ray Traced Scene</title>
<style>
body { background-color: #f0f0f0; margin: 0; overflow: hidden; }
canvas { display: block; margin: auto; }
</style>
</head>
<body>
<canvas id="sceneCanvas" width="400" height="300"></canvas>
<script>
const canvas = document.getElementById('sceneCanvas');
const ctx = canvas.getContext('2d');

// Light source position
const lightSource = {x: 300, y: 50};

// Function to draw a sphere with shading
function drawSphere(x, y, radius, color, lightReflection = true) {
const gradient = ctx.createRadialGradient(x, y, 0, x, y, radius);
gradient.addColorStop(0, 'white'); // Highlight from light source
gradient.addColorStop(0.1, color);
gradient.addColorStop(1, 'black'); // Shadow on the opposite side

ctx.fillStyle = gradient;
ctx.beginPath();
ctx.arc(x, y, radius, 0, Math.PI * 2);
ctx.fill();

if (lightReflection) {
// Add a specular highlight
const highlightX = x - radius / 3;
const highlightY = y - radius / 3;
ctx.fillStyle = 'rgba(255,255,255,0.5)';
ctx.beginPath();
ctx.arc(highlightX, highlightY, radius / 5, 0, Math.PI * 2);
ctx.fill();
}
}

// Draw red sphere with shading
drawSphere(200, 150, 50, 'red');

// Draw blue sphere with shading
drawSphere(100, 170, 48, 'blue');

// Improved reflection of red sphere on blue sphere
ctx.globalAlpha = 0.7;
ctx.fillStyle = 'rgba(255,0,0,0.5)';
ctx.beginPath();
// The reflection should be elliptical due to perspective and sphere curvature
ctx.ellipse(130, 180, 10, 15, Math.PI / 4, 0, Math.PI * 2);
ctx.fill();

// Simple shadow for depth (this is very basic, real ray tracing would calculate this)
ctx.globalAlpha = 0.3;
ctx.fillStyle = 'black';
ctx.beginPath();
ctx.ellipse(205, 205, 40, 15, 0, 0, Math.PI * 2); // Shadow for red sphere
ctx.ellipse(105, 220, 38, 15, 0, 0, Math.PI * 2); // Shadow for blue sphere
ctx.fill();
</script>
</body>
</html>
53 changes: 53 additions & 0 deletions blog/briefly-evaluating-qwq-preview/spat-llama-vision-90.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Simple Scene</title>
</head>
<body>
<canvas id="sceneCanvas" width="400" height="300"></canvas>
<script>
const canvas = document.getElementById('sceneCanvas');
const ctx = canvas.getContext('2d');

// Light properties
const lightDirection = { x: -0.5, y: -1 };
const lightIntensity = 1;
const ambientLight = 0.2;

function diffuseLighting(point) {
const dotProduct = (point.x * lightDirection.x + point.y * lightDirection.y);
return Math.max(dotProduct, ambientLight) * lightIntensity;
}

// Draw red sphere in the middle
ctx.fillStyle = 'red';
ctx.beginPath();
ctx.arc(200, 150, 50, 0, Math.PI * 2);

const gradientRed = ctx.createRadialGradient(200, 150, 0, 200, 150, 50);
gradientRed.addColorStop(0, 'rgb(255, 0, 0)'); // Surface color
gradientRed.addColorStop(1, 'rgba(255, 0, 0, 0.2)'); // Lighten towards the edge
ctx.fillStyle = gradientRed;
ctx.fill();

// Draw blue sphere to the left
ctx.beginPath();
ctx.arc(100, 170, 48, 0, Math.PI * 2);

const gradientBlue = ctx.createRadialGradient(100, 170, 0, 100, 170, 48);
gradientBlue.addColorStop(0, 'rgb(0, 0, 255)'); // Surface color
gradientBlue.addColorStop(1, 'rgba(0, 0, 255, 0.2)'); // Lighten towards the edge
ctx.fillStyle = gradientBlue;
ctx.fill();

// Draw reflection of red sphere on blue sphere (simplified)
ctx.globalAlpha = 0.5; // Semi-transparent for the reflection effect
ctx.beginPath();
ctx.arc(138, 206, 19, 0, Math.PI * 2);
ctx.fillStyle = 'rgba(255, 0, 0, 0.3)'; // Reflection color with alpha
ctx.fill();

</script>
</body>
</html>
46 changes: 46 additions & 0 deletions blog/briefly-evaluating-qwq-preview/spat-misla.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Simple Scene</title>
<style>
body {
background-color: #000; /* Black background for better contrast */
display: flex;
justify-content: center;
align-items: center;
height: 100vh;
margin: 0;
}
</style>
</head>
<body>
<canvas id="sceneCanvas" width="400" height="300"></canvas>
<script>
const canvas = document.getElementById('sceneCanvas');
const ctx = canvas.getContext('2d');

// Function to draw a sphere with shading and highlight
function drawSphere(x, y, radius, color) {
const grd = ctx.createRadialGradient(x, y, 0, x, y, radius);
grd.addColorStop(0, 'white'); // Highlight at the center
grd.addColorStop(1, color); // Main color at the edge
ctx.fillStyle = grd;
ctx.beginPath();
ctx.arc(x, y, radius, 0, Math.PI * 2);
ctx.fill();
}

// Draw red sphere in the middle with shading and highlight
drawSphere(200, 150, 50, 'red');

// Draw blue sphere to the left with shading and highlight
drawSphere(138, 170, 46, 'blue');

// Simplified reflection of red sphere on blue sphere
ctx.globalAlpha = 0.5; // Semi-transparent for the reflection effect
drawSphere(129, 195, 18, 'red');
</script>
</body>
</html>

33 changes: 33 additions & 0 deletions blog/briefly-evaluating-qwq-preview/spat-o.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Simple Scene</title>
</head>
<body>
<canvas id="sceneCanvas" width="400" height="300"></canvas>
<script>
const canvas = document.getElementById('sceneCanvas');
const ctx = canvas.getContext('2d');

// Draw red sphere in the middle
ctx.fillStyle = 'red';
ctx.beginPath();
ctx.arc(200, 150, 50, 0, Math.PI * 2);
ctx.fill();

// Draw blue sphere to the left
ctx.fillStyle = 'blue';
ctx.beginPath();
ctx.arc(100, 170, 48, 0, Math.PI * 2);
ctx.fill();

// Draw reflection of red sphere on blue sphere (simplified)
ctx.globalAlpha = 0.5; // Semi-transparent for the reflection effect
ctx.beginPath();
ctx.arc(138, 206, 19, 0, Math.PI * 2);
ctx.fillStyle = 'red';
ctx.fill();
</script>
</body>
</html>
Loading

0 comments on commit f233a2a

Please sign in to comment.