Fix initial ingestion failing due to Intercom API timeouts

[spastorelli]

Note: ⚠️ This is only a mitigation for now but I believe we should be revisiting the criteria we use for initially retrieving (at install time) the closed conversations from Intercom. Maybe by adding a filter to search for conversations that were closed in the last 30 days or similar to help optimize the search query instead of the broader criteria we use right now.
Alternatively we could allow customers in the integration configuration screen specify a query that could help filters the ones they are really interested in using for ingestion.

This PR addresses Intercom API timeouts occurring during the initial fetch of closed conversations.

Currently, we request 100 items per page, which causes each search call to take around 18 seconds. After fetching about 5–6 pages, the API begins timing out:

To mitigate these timeouts, this PR reduces the number of conversations fetched per page, bringing the latency per search call down to 6–8 seconds:

Note: Since we previously fixed worker execution timeouts in #1025, by offloading parts of the workload (e.g. fetching conversation details, processing ~700 conversations, and ingestion) into batched task executions, we should now be in a good position regarding Intercom API rate limits.

[changeset-bot]

⚠️ No Changeset found

Latest commit: a5d1cd3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[pkg-pr-new]

Open in StackBlitz

bun add https://pkg.pr.new/GitbookIO/integrations/@gitbook/api@1044

commit: a5d1cd3