Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Snowflake performance comparison with new insights and results #64

Merged
merged 3 commits into from
Jan 16, 2025
Merged

Update Snowflake performance comparison with new insights and results #64

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8c41c0b
Update Snowflake performance comparison with new insights and results
delfrrr Jan 8, 2025
7f7acfd
Update content/blog/bigquery-vs-snowflake-gis-sql-performance-compari…
delfrrr Jan 11, 2025
4fa54e8
Update content/blog/bigquery-vs-snowflake-gis-sql-performance-compari…
delfrrr Jan 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 23 additions & 26 deletions content/blog/bigquery-vs-snowflake-gis-sql-performance-comparison-1/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,50 +70,47 @@ WHERE subtype = 'road'
<td>$0.002515</td>
</tr>
<tr>
<td>Snowflake</td>
<td>[outdated] Snowflake</td>
<td>1m 17s</td>
<td>63.61GB</td>
<td>$0.0556</td>
</tr>
<tr>
<td>Snowflake with Search Optimization</td>
<td>3.9s</td>
<td>364MB</td>
<td>$0.002527</td>
</tr>
</tbody>
</table>

### Query plans

{{< img src="query-plans.png" >}}

## Key insights

- BigQuery reads only a small portion of the table, leveraging clustering and geospatial partitioning effectively.
- Snowflake reads ~80% of the table, with 89% of query time spent on Remote Disk I/O.
#### *Update: 2024-12-08*

In this test, BigQuery is much faster and cheaper than Snowflake.
After reaching out to the Snowflake dev team over LinkedIn, they confirmed that Snowflake Search Optimization was not applied in my query because of Enterprise Account limitations on a dataset publisher side.

## Feature Gap
This means that at the moment, no one can have search optimization for GIS queries on the official Overture Maps dataset in Snowflake.

### Snowflake Search Optimization
#### *Update 2025–01–08*

In theory, the Snowflake search optimization service should improve the performance of queries with predicates that use geospatial functions with GEOGRAPHY objects. This feature requires Enterprise Edition according to the documentation. It has also required to be applied to the column.
After reading this article, Snowflake team made Search Optimization available for all users on the Overture Maps dataset.

I, tried to verify if feature is configured on dataset:
Updates results added to the table.

```sql
DESCRIBE SEARCH OPTIMIZATION ON OVERTURE_MAPS__TRANSPORTATION.CARTO.SEGMENT;
-- empty result
```

However, when I leave only ST_CONTAINS in the filter, Snowflake scans only 60% of the table, and I keep seeing Search Optimization Access node in the query planner.
### Query plans

*Update 2024–12–08*: After reaching out to the Snowflake dev team over LinkedIn, they confirmed that Snowflake Search Optimization was not applied in my query because of Enterprise Account limitations on a dataset publisher side.
{{< img src="query-plans.png" >}}

This means that at the moment, no one can have search optimization for GIS queries on the official Overture Maps dataset in Snowflake.
## Key insights

### Bigger warehouse size
- BigQuery reads only a small portion of the table, leveraging clustering and geospatial partitioning effectively.
- Without Search Optimization Snowflake scans the whole table
- With Search Optimization, Snowflake is as efficient as BigQuery.

Will execution time become better with a bigger warehouse size on Snowflake? With 89% of the query time being Remote Disk I/O, it probably will not improve more than 10%.
In this test, after Snowflake Search Optimization was applied, the performance of both platforms was similar.

## What is next?
## What’s next?

While this difference between BigQuery and Snowflake seems astonishing, in the next post we will look at more realistic examples where geometry is not constant and comparison between databases is closer.
Check the next post where we compare platforms in more realistic examples where geometry is not constant.

This article will be updated with feedback and links to other tests.
→ [Compare and optimize BigQuery and Snowflake performance on GIS queries. Part 2](http://localhost:1313/blog/compare-and-optimize-bigquery-and-snowflake-performance-on-gis-queries.-part-2./)
Loading