Skip to content

Commit c930513

Browse files
authored
feat: Add standalone query and scan iterators with support for suspending/resuming queries (awslabs#61)
* Bump Jest dependency * Add a standalone query/scan iterator package * Integrate the standalone query/scan abstractions to provide easy access to metadata from the async iterable returned by parallelScan, query, or scan * Ensure Symbol.asyncIterator is always polyfilled before classes implementing the AsyncIterable protocol are declared * Remove inferred LastEvaluatedKey generation * Add documentation for query/scan metadata+pagination * Update changelog * Add tests for "limit" parameter * Remove package-lock.json from committed artifacts
1 parent 0912ab9 commit c930513

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+4438
-438
lines changed

.travis.yml

+3
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,6 @@ sudo: false
1010

1111
before_script:
1212
- npm run bootstrap
13+
14+
script:
15+
- npm test -- --runInBand

README.md

+1
Original file line numberDiff line numberDiff line change
@@ -236,3 +236,4 @@ by viewing those packages directly.
236236
* [Amazon DynamoDB DataMapper Annotations](packages/dynamodb-data-mapper-annotations/)
237237
* [Amazon DynamoDB Data Marshaller](packages/dynamodb-data-marshaller/)
238238
* [Amazon DynamoDB Expressions](packages/dynamodb-expressions/)
239+
* [Amazon DynamoDB Query Iterator](packages/dynamodb-query-iterator/)

jest.config.js

-3
This file was deleted.

package.json

+3-3
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@
1313
},
1414
"license": "Apache-2.0",
1515
"devDependencies": {
16-
"@types/jest": "^22",
16+
"@types/jest": "^23",
1717
"@types/node": "^8.0.4",
18-
"jest": "^22",
19-
"lerna": "^2.5.1",
18+
"jest": "^23",
19+
"lerna": "^2.11.0",
2020
"typedoc": "^0.11.0",
2121
"typescript": "^2.7"
2222
},

packages/dynamodb-auto-marshaller/package.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,10 @@
2828
},
2929
"license": "Apache-2.0",
3030
"devDependencies": {
31-
"@types/jest": "^22",
31+
"@types/jest": "^23",
3232
"@types/node": "^8.0.4",
3333
"aws-sdk": "^2.7.0",
34-
"jest": "^22",
34+
"jest": "^23",
3535
"typedoc": "^0.11.0",
3636
"typescript": "^2.7"
3737
},

packages/dynamodb-batch-iterator/package.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,10 @@
2828
},
2929
"license": "Apache-2.0",
3030
"devDependencies": {
31-
"@types/jest": "^22",
31+
"@types/jest": "^23",
3232
"@types/node": "^8.0.4",
3333
"aws-sdk": "^2.7.0",
34-
"jest": "^22",
34+
"jest": "^23",
3535
"typedoc": "^0.11.0",
3636
"typescript": "^2.7"
3737
},

packages/dynamodb-batch-iterator/src/BatchOperation.ts

+3-1
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@ import {
77
} from './types';
88
import DynamoDB = require('aws-sdk/clients/dynamodb');
99

10-
require('./asyncIteratorSymbolPolyfill');
10+
if (Symbol && !Symbol.asyncIterator) {
11+
(Symbol as any).asyncIterator = Symbol.for("__@@asyncIterator__");
12+
}
1113

1214
export abstract class BatchOperation<
1315
Element extends TableStateElement

packages/dynamodb-batch-iterator/src/asyncIteratorSymbolPolyfill.ts

-8
This file was deleted.

packages/dynamodb-data-mapper-annotations/package.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,11 @@
3030
},
3131
"license": "Apache-2.0",
3232
"devDependencies": {
33-
"@types/jest": "^22",
33+
"@types/jest": "^23",
3434
"@types/node": "^8.0.4",
3535
"@types/uuid": "^3.0.0",
3636
"aws-sdk": "^2.7.0",
37-
"jest": "^22",
37+
"jest": "^23",
3838
"typedoc": "^0.11.0",
3939
"typescript": "^2.7"
4040
},

packages/dynamodb-data-mapper/CHANGELOG.md

+17
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.
44
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
55
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
66

7+
## [0.6.0]
8+
### Fixed
9+
- Update `DataMapper` for TypeScript 2.9 compatibility.
10+
11+
### Added
12+
- Use purpose-built async iterable objects as the return value for `query`,
13+
`scan`, and `parallelScan`.
14+
- Report the `count`, `scannedCount`, and `consumedCapacity` tallied over the
15+
lifetime of a `query`, `scan`, or `parallelScan` as properties on the
16+
returned iterable.
17+
- Provide a method to get the underlying paginator for a `query`, `scan`, or
18+
`parallelScan` iterator. The paginator may be used to suspend and resume
19+
iteration at any page boundary.
20+
- Add `limit` parameter to `scan` and `query` to automatically cease iteration
21+
once a certain number of items have been returned or the results have been
22+
exhausted, whichever comes first.
23+
724
## [0.5.0]
825
### Fixed
926
- Add default message to `ItemNotFoundException`

packages/dynamodb-data-mapper/README.md

+181-6
Original file line numberDiff line numberDiff line change
@@ -326,9 +326,9 @@ Inserts an item into a DynamoDB table. Takes two parameters:
326326

327327
### `query`
328328

329-
Retrieves multiple values from a table based on the primary key attributes.
330-
Queries must target a single partition key value but may read multiple items
331-
with different range keys.
329+
Retrieves multiple values from a table or index based on the primary key
330+
attributes. Queries must target a single partition key value but may read
331+
multiple items with different range keys.
332332

333333
This method is implemented as an async iterator and the results can be consumed
334334
with a `for-await-of` loop. If you are using TypeScript, you will need to
@@ -380,7 +380,9 @@ Takes three parameters:
380380
* `indexName` - The name of the index against which to execute this query.
381381
If not specified, the query will be executed against the base table.
382382

383-
* `pageSize` - The maximum number of items to return.
383+
* `limit` - The maximum number of items to return.
384+
385+
* `pageSize` - The maximum number of items to return **per page of results**.
384386

385387
* `projection` - A projection expression directing DynamoDB to return a
386388
subset of any fetched item's attributes. Please refer to the
@@ -398,9 +400,61 @@ Takes three parameters:
398400
* `startKey` - The primary key of the first item that this operation will
399401
evaluate.
400402

403+
#### Query metadata
404+
405+
The iterator returned by `query` will keep track of the number of items yielded
406+
and the number of items scanned via its `count` and `scannedCount` properties:
407+
408+
```typescript
409+
const iterator = mapper.query(
410+
MyClass,
411+
{partitionKey: 'foo', rangeKey: between(0, 10)}
412+
);
413+
for await (const record of iterator) {
414+
console.log(record, iterator.count, iterator.scannedCount);
415+
}
416+
```
417+
418+
#### Pagination
419+
420+
If you wish to perform a resumable query, you can use the `.pages()` method of
421+
the iterator returned by `query` to access the underlying paginator. The
422+
paginator differs from the iterator in that it yields arrays of unmarshalled
423+
records and has a `lastEvaluatedKey` property that may be provided to a new
424+
call to `mapper.query` to resume the query later or in a separate process:
425+
426+
```typescript
427+
const paginator = mapper.query(
428+
MyClass,
429+
{partitionKey: 'foo', rangeKey: between(0, 10)},
430+
{
431+
// automatically stop after 25 items or the entire result set has been
432+
// fetched, whichever is smaller
433+
limit: 25
434+
}
435+
).pages();
436+
437+
for await (const page of paginator) {
438+
console.log(
439+
paginator.count,
440+
paginator.scannedCount,
441+
paginator.lastEvaluatedKey
442+
);
443+
}
444+
445+
const newPaginator = mapper.query(
446+
MyClass,
447+
{partitionKey: 'foo', rangeKey: between(0, 10)},
448+
{
449+
// start this new paginator where the previous one stopped
450+
startKey: paginator.lastEvaluatedKey
451+
}
452+
).pages();
453+
```
454+
401455
### `scan`
402456

403-
Retrieves all values in a table.
457+
Retrieves all values in a table or index.
404458

405459
This method is implemented as an async iterator and the results can be consumed
406460
with a `for-await-of` loop. If you are using TypeScript, you will need to
@@ -433,6 +487,8 @@ Takes two parameters:
433487

434488
* `limit` - The maximum number of items to return.
435489

490+
* `pageSize` - The maximum number of items to return **per page of results**.
491+
436492
* `projection` - A projection expression directing DynamoDB to return a
437493
subset of any fetched item's attributes. Please refer to the
438494
documentation for the `@aws/dynamodb-expressions` package for guidance
@@ -452,6 +508,52 @@ Takes two parameters:
452508
divided (if this scan is being performed as part of a parallel scan
453509
operation).
454510

511+
#### Scan metadata
512+
513+
The iterator returned by `scan` will keep track of the number of items yielded
514+
and the number of items scanned via its `count` and `scannedCount` properties:
515+
516+
```typescript
517+
const iterator = mapper.scan(MyClass);
518+
for await (const record of iterator) {
519+
console.log(record, iterator.count, iterator.scannedCount);
520+
}
521+
```
522+
523+
#### Pagination
524+
525+
If you wish to perform a resumable scan, you can use the `.pages()` method of
526+
the iterator returned by `scan` to access the underlying paginator. The
527+
paginator differs from the iterator in that it yields arrays of unmarshalled
528+
records and has a `lastEvaluatedKey` property that may be provided to a new
529+
call to `mapper.scan` to resume the scan later or in a separate process:
530+
531+
```typescript
532+
const paginator = mapper.scan(
533+
MyClass,
534+
{
535+
// automatically stop after 25 items or the entire result set has been
536+
// fetched, whichever is smaller
537+
limit: 25
538+
}
539+
).pages();
540+
for await (const page of paginator) {
541+
console.log(
542+
paginator.count,
543+
paginator.scannedCount,
544+
paginator.lastEvaluatedKey
545+
);
546+
}
547+
548+
const newPaginator = mapper.scan(
549+
MyClass,
550+
{
551+
// start this new paginator where the previous one stopped
552+
startKey: paginator.lastEvaluatedKey
553+
}
554+
).pages();
555+
```
556+
455557
### `parallelScan`
456558
457559
Retrieves all values in a table by dividing the table into segments, all of
@@ -488,7 +590,7 @@ Takes three parameters:
488590
* `indexName` - The name of the index against which to execute this query.
489591
If not specified, the query will be executed against the base table.
490592
491-
* `limit` - The maximum number of items to return.
593+
* `pageSize` - The maximum number of items to return **per page of results**.
492594
493595
* `projection` - A projection expression directing DynamoDB to return a
494596
subset of any fetched item's attributes. Please refer to the
@@ -502,6 +604,53 @@ Takes three parameters:
502604
* `startKey` - The primary key of the first item that this operation will
503605
evaluate.
504606
607+
#### Scan metadata
608+
609+
The iterator returned by `parallelScan` will keep track of the number of items
610+
yielded and the number of items scanned via its `count` and `scannedCount`
611+
properties:
612+
613+
```typescript
614+
const iterator = mapper.parallelScan(MyClass, 4);
615+
for await (const record of iterator) {
616+
console.log(record, iterator.count, iterator.scannedCount);
617+
}
618+
```
619+
620+
#### Pagination
621+
622+
If you wish to perform a resumable parallel scan, you can use the `.pages()`
623+
method of the iterator returned by `parallelScan` to access the underlying
624+
paginator. The paginator differs from the iterator in that it yields arrays of
625+
unmarshalled records and has a `scanState` property that may be provided
626+
to a new call to `mapper.parallelScan` to resume the scan later or in a separate
627+
process:
628+
629+
```typescript
630+
const paginator = mapper.parallelScan(
631+
MyClass,
632+
4
633+
).pages();
634+
for await (const page of paginator) {
635+
console.log(
636+
paginator.count,
637+
paginator.scannedCount,
638+
paginator.lastEvaluatedKey
639+
);
640+
641+
break;
642+
}
643+
644+
const newPaginator = mapper.parallelScan(
645+
MyClass,
646+
4,
647+
{
648+
// start this new paginator where the previous one stopped
649+
scanState: paginator.scanState
650+
}
651+
).pages();
652+
```
653+
505654
### `update`
506655
507656
Updates an item in a DynamoDB table. Will leave attributes not defined in the
@@ -528,3 +677,29 @@ Takes two parameters:
528677
529678
* `skipVersionCheck` - Whether to forgo creating a condition expression
530679
based on a defined `versionAttribute` in the schema.
680+
681+
### `executeUpdateExpression`
682+
683+
Executes a custom update expression. This method will **not** automatically
684+
apply a version check, as the current state of the object being updated is not
685+
known.
686+
687+
Takes four parameters:
688+
689+
* The expression to execute. Please refer to the documentation for the
690+
`@aws/dynamodb-expressions` package for guidance on creating update
691+
expression objects.
692+
693+
* The key of the item being updated.
694+
695+
* The constructor for the class mapped to the table against which the expression
696+
should be run. Must have a prototype with a table name accessible via a
697+
property identified with the `DynamoDbTable` symbol and a schema accessible
698+
via a property identified with the `DynamoDbSchema` symbol.
699+
700+
* (Optional) An object specifying any of the following options:
701+
702+
* `condition` - A condition expression whose assertion must be satisfied in
703+
order for the update operation to be executed. Please refer to the
704+
documentation for the `@aws/dynamodb-expressions` package for guidance
705+
on creating condition expression objects.

packages/dynamodb-data-mapper/package.json

+3-2
Original file line numberDiff line numberDiff line change
@@ -30,10 +30,10 @@
3030
},
3131
"license": "Apache-2.0",
3232
"devDependencies": {
33-
"@types/jest": "^22",
33+
"@types/jest": "^23",
3434
"@types/node": "^8.0.4",
3535
"aws-sdk": "^2.7.0",
36-
"jest": "^22",
36+
"jest": "^23",
3737
"typedoc": "^0.11.0",
3838
"typescript": "^2.7"
3939
},
@@ -42,6 +42,7 @@
4242
"@aws/dynamodb-batch-iterator": "^0.5.0",
4343
"@aws/dynamodb-data-marshaller": "^0.5.0",
4444
"@aws/dynamodb-expressions": "^0.5.0",
45+
"@aws/dynamodb-query-iterator": "^0.5.0",
4546
"tslib": "^1.8.1"
4647
},
4748
"peerDependencies": {

0 commit comments

Comments
 (0)