Bump datafusion -> 21.0.0 and add dyn and dyn_hash functions to custo… (#1094)

* Bump datafusion -> 20.0.0 and add dyn and dyn_hash functions to custom parsers

* Bump to datafusion 21.0.0

* Add Rust -> Python logging bridge

* updates

* updates

* updates

* Remove optimizer causing issues with simplfying expressions

* Updates to FilterColumnsPostJoin

* Manually create OwnedTableReference instead of relying on OwnedTableReference::from(&str) so we can honor our own dialect

* Remove xfail from test_jdbc:test_jdbc_has_table

* uncomment pytest section that was mistakenly committed after a portion was commented out for testing

* Temporarily xfail test select with casing

* Uncomment xfail for test select with case

* Fix typo

* Re-enable CommonSubexprEliminate optimizer for a test

* Comment out CommonSubExprOptimizer

* bumpy to 21.1.0 bug fix release version and remove filter_column_post_join optimizer that is no longer needed

* Update string literal output in test to updated format

* Add remove projections optmizer rule. Add/reorder rules to match datafusion 21.0

* Fix column container mapping from the rel plan in cross joins

* Update optimizer cargo test to remove redundant projections

* Bump Rust version to 1.67 in read the docs

* Move rust version to conda environment

* Pin nightly build Rust version to 1.67

* Change to Rust version 1.65

* Address review concerns

---------

Co-authored-by: Ayush Dattagupta <[email protected]>
Co-authored-by: Charles Blackmon-Luca <[email protected]>

Author: 3 people

.readthedocs.yaml

@@ -4,7 +4,6 @@ build:
   os: ubuntu-20.04
   tools:
     python: "mambaforge-4.10"
-    rust: "1.64"
 
 sphinx:
   configuration: docs/source/conf.py

continuous_integration/recipe/meta.yaml

@@ -22,7 +22,7 @@ build:
 
 requirements:
   build:
-    - {{ compiler('rust') }}
+    - {{ compiler('rust') }} >=1.65.0
     - setuptools-rust >=1.5.2
   host:
     - pip