-
Notifications
You must be signed in to change notification settings - Fork 192
Labels
Description
Part of #2393
Description
The confluence connector currently swallows errors during its paginated_api_call() function. See:
connectors/connectors/sources/confluence.py
Lines 194 to 198 in 8c8d4a3
| except Exception as exception: | |
| self._logger.warning( | |
| f"Skipping data for type {url_name} from {url}. Exception: {exception}." | |
| ) | |
| break |
It would be better for these to be re-raised if they can't be handled, and for callers to be able to determine how significant the error is. Otherwise, critical calls (like the first attempt to list all pages) could fail, leading to a mostly empty sync that thinks it's successful.
Acceptance Criteria
-
paginated_api_callcatches some types of errors, and retries/handles the ones it can - if
paginated_api_callcannot handle an error, it re-raises it - calling code of
paginated_api_callhandles errors as makes sense for the use case. We balance the need to "fail fast" with the need to not crash a large sync because of a small data issue.