Skip to content

Commit 5da5a8c

Browse files
nickytonlinetaty2010eduardoboucasLukas Holzerpieh
authored
feat: add blob storage (#2287)
* feat: adding inital blob setup * feat: added get and set for ssr files * chore: prettier * chore: prettier * updated tsconfig and added dynamic imports * chore: reverted some changes to bring project back to commonjs * chore: some cleanup * chore: ran formatter * chore: more cleanup related to going back to commonjs * chore: reverting some more stuff from async * chore: getting blob storage in a usable state in commonjs land * chore: testing blob storage * chore: memoized getting blob storage * chore: small refactor * chore: trigger build * chore: cleaned up blob storage instantiation * chore: fixed types for getting blob storage * chore: removed todo comment * chore: removed a TODO comment * chore: refactored getBlobStorage * chore: putting up test for Eduardo * chore: add debug logging * refactor: rebuild * refactor: remove debug logging * chore: wip * chore: updated package lock * chore: testing out blob storage for prerendered content * chore: added a check for deploy ID for blob storage * chore: small refactor * chore: reverted some refactor not related to the PR * chore: undid encoding of blob storage keys * chore: removed unnecessary comment * chore: added some logging for blob storage * chore: testing/logging blob storage * chore: stripping starting slash for blob storage keys * chore: using netliblob.setJSON now instead of .set * chore: cleaned up setting blob storage * chore: fixed issue with ssrDependencies not being iterable * chore: some logging cleanup * chore: reverted some unnecessary changes from 01f44e3 * chore: reverted some unnecessary changes from 01f44e3 * chore: reverted some unnecessary changes * chore: reverted change not related to work in PR * chore: updated package lock * chore: added a comment about starting slashes removed from blob storage keys * chore: added some more comments * chore: trigger build * chore: removed logging feature flags * chore: testing blob storage flag * chore: cleaned up blob storage check * chore: trigger build * chore: added some temp logging * Revert "chore: added some temp logging" This reverts commit 217bd9a. * chore: fixed blob storage check condition 🙃 * chore: bumped @netlify/build to 29.20.15 * chore: trigger build * feat: implemented incremental cache * chore: some cleanup * chore: removed a TODO comment * chore: wip * chore: refactored getBlobStorage * chore: added incremental cache with memory cache for testing key and values in cache * add new incremental cache file + store blob * removing blob type for now * updated file location for incremental-cache.js * updated filepath * adding publish dir intead * use publish dir for config * feat: moving incremental cache to functions * chore: creating cache file right before config set * chore: added require.resolve for incremental cache * chore: uncomment import that got commented out during merge * chore: no longer infer the type for Blobs * chore: updated @netlify/build to 29.21.0 * chore: formatting fixes * chore: cleaned up how netliblob is determined for SSR lambdas * chore: now incremental cache handler gets generated if blob storage is available * chore: fixed resolving file path for blobStorage.js * chore: trigger build * chore: cleaned up getting blob credentials from context * chore: added some temporary logging * chore: added another log statement in incremental cache handler * chore: trigger build * chore: stripped starting slash for keys for now * chore: changed where incremental cache handler gets put * chore: removed test incremental cache handler from demo site * chore: update * chore: update * chore: updates * chore: make blob storage available during runtime * chore: update * chore: use the blob store during request time differently * chore: update * chore: test * chore: update logging * chore: update * chore: update * chore: update * chore: fix * chore: update * chore: improve logging * chore: update * chore: update * chore: update * chore: update * chore: change encoding of key * refactor: remove incremental cache handler injection * refactor: eslint ignore file via .eslintignore and not config * feat: bump @netlify/blobs to 2.0.0 (key hashing) * refactor: allow @netlify/blobs to do the key hashing * refactor: some debugging * test: will this be set? * test: will this be set?2 * test: will this be set?3 * test: will this be set?4 * test: will this be set?5 * test: will this be set?6 * test: will this be set?7 * test: will this be set?8 * test: will this be set?9 * fix: lambda response is base64 encoded * test: check how things look in moveStaticPages * test: check how things look in moveStaticPages2 * test: check how things look in moveStaticPages3 * test: check if we can skip key normalization already * fix: don't compile blob helpers for function bundling, copy files instead * feat: cleanup * fix: clean up lambda a bit * fix: clean up some logs when used locally * fix: move esbuild back to deps, as it's used for middleware compilation * feat: don't warm up blobs when configuring functions (we will do this when moving static assets) * fix: more cleanup * feat: start uploading to blob store during static pages moves * feat: log number of assets moved to blob storage and delete those files * test: check simple blob keys ... again * test: restore key normalization ... again * feat: apply trailing slash to blob keys * feat: initial default locale handling * chore: some logs * fix: commit exported isLocalized * fix: fix isLocalized import * chore: expose router push * feat: blob manifest for i18n aliases * chore: log blob manifest * fix: fix blobs-manifest filenames * chore: a bit more type safety * chore: bump cypress to latest v12 * test: test fix1 * chore: drop some test code * fix: remove blobs-manifest before publishing to cdn * feat: initial headers setting * feat: initial headers setting2 * chore: some more visible rsc-data logs * chore: some more visible rsc-data logs2 * chore: some more visible rsc-data logs3 * fix: normalize path after blob store checks * fix: strip trailing slash for .rsc/ paths before checking blob store * test: log maybe more details * test: maybe i will get more details without --json? * test: inject global fetch polyfill * test: inject global fetch polyfill2 * test: restore json output * test: remove logs from fetch polyfill that were breaking ntl deploy output parsing * chore: some cleanup * chore: cleanup lambda handler * chore: drop mime-types dep as it's not used * test: let's see basepath * chore: revert basepath for now * chore: try out slightly different approach * chore: show time on index page * chore: fix var name * chore: set to array fix * test: try pulling from blob store in fs augment * test: maybe fix blob type * chore: stuff * chore: stuff2 * chore: add deployID to logs * chore: log missed blobs * chore: log filepath * chore: is cached file being read? * chore: is blobbing erroring out? * chore: ensure directory stuff * chore: oneline log for easier grepping * chore: more oneliners * chore: don't use asynclocalstorage ...! chore: don't use asynclocalstorage ...! * chore: log served message * chore: log served message2 * chore: what if we use asynclocalstorage later * chore: cleanups * chore: cleanups2 * chore: cleanups3 * chore: cleanups4 * chore: cleanups5 * chore: cleanups6 * chore: cleanups7 * chore: test basePath * chore: test basePath, no i18n * chore: test no basePath no i18n * chore: restore default demo config * chore: cleanups8 * test: fix unit tests, maybe fix API functions * test: maybe fix API functions2 * chore: try to make fallback:true act as fallback:blocking * chore: fix some logic in fallback: blocking forcing and add comments * chore: update test comments * chore: move some imports so comment make sense * chore: strip node: from compiled @netlify/blobs * chore: restore revalidation on non-first odb request * chore: add a feature flag for using blobs for isr pages * chore: sync package-lock.json * fix: import type from package entry and not internals * chore: bump @netlify/blobs, don't build CJS version anymore --------- Co-authored-by: taty2010 <[email protected]> Co-authored-by: Eduardo Bouças <[email protected]> Co-authored-by: Lukas Holzer <[email protected]> Co-authored-by: Michal Piechowiak <[email protected]>
1 parent e3bb71d commit 5da5a8c

File tree

17 files changed

+305
-104
lines changed

17 files changed

+305
-104
lines changed

Diff for: cypress/e2e/default/dynamic-routes.cy.ts

+8-5
Original file line numberDiff line numberDiff line change
@@ -61,12 +61,13 @@ describe('Dynamic Routing', () => {
6161
expect(res.body).to.contain('Under the Dome')
6262
})
6363
})
64-
it('renders fallback page via ODB on a non-prerendered dynamic route with fallback: true', () => {
64+
it('does not render fallback page via ODB on a non-prerendered dynamic route with fallback: true', () => {
65+
// unfortunately there is a problem with `fallback: true` in ODB context - the fallback would be cached indefinitely
66+
// so visits to those pages would always render the fallback first and browser client would later re-render with correct
67+
// content. As this is not ideal the `fallback: true` is treated as `fallback: blocking`.
6568
cy.request({ url: '/getStaticProps/withFallback/3/', headers: { 'x-nf-debug-logging': '1' } }).then((res) => {
6669
expect(res.status).to.eq(200)
67-
// expect 'odb' until https://github.com/netlify/pillar-runtime/issues/438 is fixed
6870
expect(res.headers).to.have.property('x-nf-render-mode', 'odb')
69-
// expect 'Bitten' until the above is fixed and we can test for fallback 'Loading...' message
7071
expect(res.body).to.contain('Bitten')
7172
})
7273
})
@@ -115,12 +116,14 @@ describe('Dynamic Routing', () => {
115116
},
116117
)
117118
})
118-
it('renders fallback page via ODB on a non-prerendered dynamic route with revalidate and fallback: true', () => {
119+
it('does not render fallback page via ODB on a non-prerendered dynamic route with revalidate and fallback: true', () => {
120+
// unfortunately there is a problem with `fallback: true` in ODB context - the fallback would be cached indefinitely
121+
// so visits to those pages would always render the fallback first and browser client would later re-render with correct
122+
// content. As this is not ideal the `fallback: true` is treated as `fallback: blocking`.
119123
cy.request({ url: '/getStaticProps/withRevalidate/withFallback/3/', headers: { 'x-nf-debug-logging': '1' } }).then(
120124
(res) => {
121125
expect(res.status).to.eq(200)
122126
expect(res.headers).to.have.property('x-nf-render-mode', 'odb ttl=60')
123-
// expect 'Bitten' until https://github.com/netlify/pillar-runtime/issues/438 is fixed
124127
expect(res.body).to.contain('Bitten')
125128
},
126129
)

Diff for: package-lock.json

+42-40
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: packages/runtime/package.json

+1
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"manifest.yml"
1313
],
1414
"dependencies": {
15+
"@netlify/blobs": "^2.2.0",
1516
"@netlify/esbuild": "0.14.39",
1617
"@netlify/functions": "^1.6.0",
1718
"@netlify/ipx": "^1.4.5",

Diff for: packages/runtime/src/constants.ts

+1
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ export const HIDDEN_PATHS = destr(process.env.NEXT_KEEP_METADATA_FILES)
3333
'/prerender-manifest.js',
3434
'/required-server-files.json',
3535
'/static-manifest.json',
36+
'/blobs-manifest.json',
3637
]
3738

3839
export const ODB_FUNCTION_PATH = `/.netlify/builders/${ODB_FUNCTION_NAME}`

Diff for: packages/runtime/src/helpers/config.ts

+3
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,9 @@ const defaultFailBuild = (message: string, { error }): never => {
3737
export const getNextConfig = async function getNextConfig({
3838
publish,
3939
failBuild = defaultFailBuild,
40+
}: {
41+
publish: string
42+
failBuild?: (message: string, { error }) => never
4043
}): Promise<NextConfig> {
4144
try {
4245
const { config, appDir, ignore }: RequiredServerFiles = await readJSON(join(publish, 'required-server-files.json'))

0 commit comments

Comments
 (0)