No dataset.ndjson being created

[essejmclean]

Hi, I am attempting to use this to move a project from Contentful to Sanity and unfortunately running into an issue.

I have created a clean Sanity project and then run npx contentful-to-sanity@latest -s <space-id> -t <management-token> -a <access-token> ./migrate with my proper Space ID, Management Token, and API Key.

Everything runs fine by the looks of it, though maybe only up to a point, because my ./migrate directory only contains:

• contentful.json
• contentful.published.json
• schema.ts

As far as I understand it should also include a dataset.ndjson and sanity.config.ts.

Here is what my terminal logs:

  ✔ Initialize client (1s)
  ✔ Fetching data from space
    ✔ Connecting to space (1s)
    ✔ Fetching content types data (1s)
    ✔ Fetching tags data (1s)
    ✔ Fetching editor interfaces data (6s)
    ✔ Fetching content entries data (5s)
    ✔ Fetching assets data (2s)
    ✔ Fetching locales data (1s)
    ↓ Fetching webhooks data [skipped]
    ↓ Fetching roles data [skipped]
  ↓ Download assets [skipped]
  ✔ Write export log file
    ✔ Lookup directory to store the logs
    ✔ Create log directory
    ✔ Writing data to file
┌──────────────────────────┐
│ Exported entities        │
├───────────────────┬──────┤
│ Content Types     │ 30   │
├───────────────────┼──────┤
│ Tags              │ 1    │
├───────────────────┼──────┤
│ Editor Interfaces │ 30   │
├───────────────────┼──────┤
│ Entries           │ 1581 │
├───────────────────┼──────┤
│ Assets            │ 688  │
├───────────────────┼──────┤
│ Locales           │ 1    │
└───────────────────┴──────┘
The export took less than a minute (17s)

Stored space data to json file at: /Users/jessemclean/GitHub/ian-port/migrate/contentful.json


The following 0 errors and 8 warnings occurred:

13:02:58 - Rate limit error occurred. Waiting for 1586 ms before retrying...
13:02:58 - Rate limit error occurred. Waiting for 1690 ms before retrying...
13:02:58 - Rate limit error occurred. Waiting for 1632 ms before retrying...
13:02:58 - Rate limit error occurred. Waiting for 1684 ms before retrying...
13:02:58 - Rate limit error occurred. Waiting for 1614 ms before retrying...
13:02:58 - Rate limit error occurred. Waiting for 1600 ms before retrying...
13:03:00 - Rate limit error occurred. Waiting for 1650 ms before retrying...
13:03:00 - Rate limit error occurred. Waiting for 1623 ms before retrying...
The export was successful.
  ✔ Initialize client (1s)
  ✔ Fetching data from space
    ✔ Connecting to space (2s)
    ✔ Fetching content types data (1s)
    ✔ Fetching tags data (1s)
    ✔ Fetching editor interfaces data (4s)
    ✔ Fetching content entries data (2s)
    ✔ Fetching assets data (1s)
    ✔ Fetching locales data (1s)
    ↓ Fetching webhooks data [skipped]
    ↓ Fetching roles data [skipped]
  ↓ Download assets [skipped]
  ✔ Write export log file
    ✔ Lookup directory to store the logs
    ✔ Create log directory
    ✔ Writing data to file
┌──────────────────────────┐
│ Exported entities        │
├───────────────────┬──────┤
│ Content Types     │ 30   │
├───────────────────┼──────┤
│ Tags              │ 1    │
├───────────────────┼──────┤
│ Editor Interfaces │ 30   │
├───────────────────┼──────┤
│ Entries           │ 1414 │
├───────────────────┼──────┤
│ Assets            │ 660  │
├───────────────────┼──────┤
│ Locales           │ 1    │
└───────────────────┴──────┘
The export took less than a minute (10s)

Stored space data to json file at: /Users/jessemclean/GitHub/ian-port/migrate/contentful.published.json


The following 0 errors and 6 warnings occurred:

13:03:16 - Rate limit error occurred. Waiting for 1681 ms before retrying...
13:03:16 - Rate limit error occurred. Waiting for 1600 ms before retrying...
13:03:16 - Rate limit error occurred. Waiting for 1608 ms before retrying...
13:03:16 - Rate limit error occurred. Waiting for 1699 ms before retrying...
13:03:16 - Rate limit error occurred. Waiting for 1667 ms before retrying...
13:03:16 - Rate limit error occurred. Waiting for 1641 ms before retrying...
The export was successful.
Custom True and False labels are not supported by default (hideNonFirmInsights)
The Contentful content type "image" is a reserved name in Sanity. Renaming to "contentful_image"
Unique validation only supported on slug fields by default. Consider adding custom validation for other unique fields or changing field to 'slug'
Custom True and False labels are not supported by default (showContact)

Help on this would be amazing as we are trying to move a client over from Contentful to Sanity (we ♥ Sanity).

p.s. I also came across what I think may be a separate and unrelated issue in the schema.ts where contentful_imageType is added to the types object as contentfulImageType.

[alex-mbp]

I'm also experiencing the same issue

[alex-mbp]

Just having another look at this and I've found that if I run contentful-to-sanity dataset . I get the following error:

Error: No transformer found for node type "table"
@dist\contentful-rich-text-to-portable-text.cjs.development.js:302:11

It seems that there is no support for transforming tables in rich text content. It would seem logical that an option should be provided to skip over instances like these?

[DimaBur]

Have the same issue. Is there any updates how to solve it?

[nardomarquez]

Error: No transformer found for node type "table"
@dist\contentful-rich-text-to-portable-text.cjs.development.js:302:11

@alex-mbp Had that same issue as well. Were you able to bypass this?

[alex-mbp]

Error: No transformer found for node type "table"
@dist\contentful-rich-text-to-portable-text.cjs.development.js:302:11

@alex-mbp Had that same issue as well. Were you able to bypass this?

I didn't pursue it any further but imagine that the possible solutions include removing table nodes from the Contentful export if they aren't critical, or creating a transformer for the node type.

[leomanlapera]

Have the same issue today, is there any solution for this?

[raff-wakflo]

Mine does'nt even generate the schemas ts file

[mealCode]

@essejmclean how did you resolve this issue?

[nardomarquez]

@mealCode I checked @alex-mbp's issue and confirmed it was the case. I bypassed it by modifying the package that runs it. Please note that this is not a fix and you wont have any data for your tables in portable texts. But you'll be able to create your dataset.ndjson

To fix:

1. Open file that runs the transform: /node_modules/@portabletext/contentful-rich-text-to-portable-text/dist/contentful-rich-text-to-portable-text.cjs.development.js.

2. at function toPortableText, change if (!transformer) condition to this:

  if (!transformer) {
    // throw new Error("No transformer found for node type \"" + data.nodeType + "\"");
    return [];
  }

3. Run migration scripts again and it wont throw error anymore. dataset.ndjson will be produced.

You can manually clean your data afterwards.

[mealCode]

@mealCode I checked @alex-mbp's issue and confirmed it was the case. I bypassed it by modifying the package that runs it. Please note that this is not a fix and you wont have any data for your tables in portable texts. But you'll be able to create your dataset.ndjson

To fix:

1. Open file that runs the transform: /node_modules/@portabletext/contentful-rich-text-to-portable-text/dist/contentful-rich-text-to-portable-text.cjs.development.js.
2. at function toPortableText, change if (!transformer) condition to this:

  if (!transformer) {
    // throw new Error("No transformer found for node type \"" + data.nodeType + "\"");
    return [];
  }

3. Run migration scripts again and it wont throw error anymore. dataset.ndjson will be produced.

You can manually clean your data afterwards.

thank you @nardomarquez for swift response. It's awesome that you figured out on how to generate the dataset.ndjson file this way. But, i need the data comes with the .ndjson file generated too. Manually, adding the data would take me forever. I am talking about thousands of data.

[nardomarquez]

thank you @nardomarquez for swift response. It's awesome that you figured out on how to generate the dataset.ndjson file this way. But, i need the data comes with the .ndjson file generated too. Manually, adding the data would take me forever. I am talking about thousands of data.

@mealCode All data will still be migrated to your dataset.ndjson. Only tables in rich texts wont be transformed. Although if you have thousands of tables in your rich texts, you may have to write a custom transformer to convert those to portable text.

In my case, I really don't have tables on my rich text contents so bypassing it is enough.

[mealCode]

thank you @nardomarquez for swift response. It's awesome that you figured out on how to generate the dataset.ndjson file this way. But, i need the data comes with the .ndjson file generated too. Manually, adding the data would take me forever. I am talking about thousands of data.

@mealCode All data will still be migrated to your dataset.ndjson. Only tables in rich texts wont be transformed. Although if you have thousands of tables in your rich texts, you may have to write a custom transformer to convert those to portable text.

In my case, I really don't have tables on my rich text contents so bypassing it is enough.

Thank you for the help. <3 Unfortunately every content we have in contentful uses rich text. We might go with another cms provider like payload. I also wrote a json-to-ndjson-converter simple node api but it doesn't help i'll post it here as it might help others.

import { NextApiRequest, NextApiResponse } from 'next'
import fs from 'fs'

export default function handler(req: NextApiRequest, res: NextApiResponse) {
  fs.readFile('./migrate/contentful.json', 'utf8', (err, data) => {
    if (err) {
      console.error('Error reading the JSON file:', err)
      throw err
    }

    try {
      const jsonData = JSON.parse(data)

      const ndjsonStream = fs.createWriteStream('./migrate/contentful.ndjson')

      if (Array.isArray(jsonData)) {
        jsonData.forEach((item) => {
          ndjsonStream.write(JSON.stringify(item) + '\n')
        })
      } else {
        ndjsonStream.write(JSON.stringify(jsonData) + '\n')
      }

      ndjsonStream.end(() => {
        console.log('Conversion to NDJSON completed successfully.')
      })
    } catch (parseError) {
      console.error('Error parsing JSON data:', parseError)
    }
  })
}

Thank again <3