Skip to content

Create digest-stream.md #4919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
simonhaenisch wants to merge 1 commit into from
Closed

Create digest-stream.md #4919

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions content/workers/examples/digest-stream.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
type: example
summary: Generate an ETag from a ReadableStream via DigestStream.
tags:
- WebCrypto
pcx-content-type: configuration
title: ETag Generation with DigestStream
weight: 1001
layout: example
---

```ts
interface Env {
KV: KVNamespace;
}

const onFetch: ExportedHandlerFetchHandler<Env> = async (request, env) => {
const encoder = new TextEncoder();

const etagStream = new crypto.DigestStream('SHA-1');
const { readable, writable } = new TransformStream();

@jasnell jasnell Jul 7, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to use new IdentityTransformStream() here.

@simonhaenisch simonhaenisch Oct 20, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry it's been too long, looking at this again now. Can you explain the difference, please?

@jasnell jasnell Oct 31, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is using the TransformStream essentially as an identity passthrough. It is not actually transforming any bytes. This is the exact use case for IdentityTransformStream. Soon, the default implementation of new TransformStream() is going to change to be more spec compliant. It will still be an identity passthrough by default but the underlying implementation is a bit more complex. To maintain the very simple use case you have here, IdentityTransformStream is the right choice.


const etagWriter = etagStream.getWriter();
const resultWriter = writable.getWriter();

@jasnell jasnell Jul 7, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative approach here is to tee the streams.

const { writable, readable } = new IdentityTransformStream();
const [ branch1, branch2 ] = readable.tee();

env.KV.put('result', branch1);

async getDigest(readable) {
  const etagStream = new crypto.DigestStream('SHA-1');
  await readable.pipeTo(etagStream);
  return etagStream.digest;
}

const writer = writable.getWriter();
writer.write('hello');
writer.close();

const etag = await getDigest(branch2);
// ...


/**
* We're not awaiting this because we'll start streaming into
* the writable side of the transform stream after.
*/
env.KV.put('result', readable);

/**
* `processStream` would be some utility function that reads
* a ReadableStream, decoding each chunk and passing the data
* to the callback.
*/
await processStream(request.body, (data: string) => {
// ... do whatever with the data here

const encoded = encoder.encode(data);

etagWriter.write(encoded);
resultWriter.write(encoded);
}

await Promise.all([etagWriter.close(), resultWriter.close()]);

const etagBuffer = await etagStream.digest;

const etag = Array.from(new Uint8Array(etagBuffer))
.map((byte) => byte.toString(16).padStart(2, '0'))
.join('');

await env.KV.put('etag', etag);

return new Response('done');
}

const worker: ExportedHandler<Env> = {
fetch: onFetch,
};

export default worker;
```