When schema properties are empty, the target attempts to create a table with no columns which results in SQL error

[VMois]

Details

When no tables are found, the target will create a new table in Snowflake. But if a schema is empty (for some reason, e.g., is tap-spreadsheets-anywhere when no files are found), the SQL query will have an empty column definition (e.g., ()), and the SQL query will fail.

The query is executed here:

pipelinewise-target-snowflake/target_snowflake/db_sync.py

Lines 811 to 814 in aa6ac2e

	if len(found_tables) == 0:
	query = self.create_table_query()
	self.logger.info('Table %s does not exist. Creating...', table_name_with_schema)
	self.query(query)

And constructed here:

pipelinewise-target-snowflake/target_snowflake/db_sync.py

Lines 549 to 569 in aa6ac2e

	def create_table_query(self, is_temporary=False):
	"""Generate CREATE TABLE SQL"""
	stream_schema_message = self.stream_schema_message
	columns = [
	column_clause(
	name,
	schema
	)
	for (name, schema) in self.flatten_schema.items()
	]
	primary_key = []
	if len(stream_schema_message.get('key_properties', [])) > 0:
	pk_list = ', '.join(primary_column_names(stream_schema_message))
	primary_key = [f"PRIMARY KEY({pk_list})"]
	p_temp = 'TEMP ' if is_temporary else ''
	p_table_name = self.table_name(stream_schema_message['stream'], is_temporary)
	p_columns = ', '.join(columns + primary_key)
	p_extra = 'data_retention_time_in_days = 0 ' if is_temporary else 'data_retention_time_in_days = 1 '
	return f'CREATE {p_temp}TABLE IF NOT EXISTS {p_table_name} ({p_columns}) {p_extra}'

Expected behavior

I expect the target to ignore the creation of a table (and expect that no records for this schema will arrive). In the case of tap-spreadsheets-anywhere, if a schema is empty, it probably means no files were found, and no records will come afterward.

Some other taps might output an empty schema and still send records afterward. Not sure how much the target should be responsible for handling those errors.