Skip to content

Commit 348d081

Browse files
goodwanghangoodwanghan
authored
Add fugue API (#396)
* Add fugue interfaceless util functions * update tests * fix test coverage * fix numpy brreaking change * update backends for utils functions * fix * update qpd * refactor code * Add test suite for express functions * add engine level utils * refactor code * add engine operations * update type annotations and docs * lint * top api docs * Refactor ibis, add fugue sql api * make duckdb columns encoded * improve test coverage * fix tests * refactor SQLEngine * fix ray tests and coverage * fix tests, add fugue.default.partitions * update docs * fix tests * fix test coverage * update docs * add all sql api functions * lint * add join functions * Make PartitionSpec more flexible
1 parent b8b7ace commit 348d081

File tree

108 files changed

+5300
-1890
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

108 files changed

+5300
-1890
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,7 @@ pythonenv*
120120

121121
# mkdocs documentation
122122
/site
123+
.virtual_documents
123124

124125
# mypy
125126
.mypy_cache

‎.pylintrc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
[MESSAGES CONTROL]
2-
disable = C0103,C0114,C0115,C0116,C0122,C0200,C0201,C0302,C0411,C0415,E0401,E0712,E1130,E5110,R0201,R0205,R0801,R0902,R0903,R0904,R0911,R0912,R0913,R0914,R0915,R1705,R1710,R1718,R1720,R1724,W0102,W0107,W0108,W0201,W0212,W0221,W0223,W0237,W0511,W0613,W0631,W0640,W0703,W0707,W1116
2+
disable = C0103,C0114,C0115,C0116,C0122,C0200,C0201,C0302,C0411,C0415,E0401,E0712,E1130,E5110,R0201,R0205,R0801,R0902,R0903,R0904,R0911,R0912,R0913,R0914,R0915,R1705,R1710,R1718,R1720,R1724,W0102,W0107,W0108,W0201,W0212,W0221,W0223,W0237,W0511,W0613,W0622,W0631,W0640,W0703,W0707,W1116
33
# TODO: R0205: inherits from object, can be safely removed

‎RELEASE.md

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,20 @@
11
# Release Notes
22

3-
## 0.7.4
4-
5-
- [340](https://github.com/fugue-project/fugue/issues/340) Migrate to plugin mode (DataFrames & Extensions)
3+
## 0.8.0
4+
5+
- [384](https://github.com/fugue-project/fugue/issues/384) Expanding Fugue API
6+
- [396](https://github.com/fugue-project/fugue/issues/396) Ray/Dask engines guess optimal default partitions
7+
- [403](https://github.com/fugue-project/fugue/issues/403) Deprecate register_raw_df_type
8+
- [392](https://github.com/fugue-project/fugue/issues/392) Aggregations on Spark dataframes fail intermittently
9+
- [398](https://github.com/fugue-project/fugue/issues/398) Rework API Docs and Favicon
10+
- [393](https://github.com/fugue-project/fugue/issues/393) ExecutionEngine as_context
11+
- [385](https://github.com/fugue-project/fugue/issues/385) Remove DataFrame metadata
12+
- [381](https://github.com/fugue-project/fugue/issues/381) Change SparkExecutionEngine to use pandas udf by default
13+
- [380](https://github.com/fugue-project/fugue/issues/380) Refactor ExecutionEngine (Separate out MapEngine)
14+
- [378](https://github.com/fugue-project/fugue/issues/378) Refactor DataFrame show
15+
- [377](https://github.com/fugue-project/fugue/issues/377) Create bag
16+
- [372](https://github.com/fugue-project/fugue/issues/372) Infer execution engine from input
17+
- [340](https://github.com/fugue-project/fugue/issues/340) Migrate to plugin mode
618
- [369](https://github.com/fugue-project/fugue/issues/369) Remove execution from FugueWorkflow context manager, remove engine from FugueWorkflow
719
- [373](https://github.com/fugue-project/fugue/issues/373) Fixed Spark engine rename slowness when there are a lot of columns
820

‎docs/api/fugue.dataframe.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ fugue.dataframe
2727
.. |FugueDataTypes| replace:: :doc:`Fugue Data Types <tutorial:tutorials/appendix/generate_types>`
2828

2929

30+
fugue.dataframe.api
31+
-------------------
32+
33+
.. automodule:: fugue.dataframe.api
34+
:members:
35+
:undoc-members:
36+
:show-inheritance:
37+
3038
fugue.dataframe.array\_dataframe
3139
--------------------------------
3240

‎docs/api/fugue.dataset.rst

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
fugue.dataset
2+
==============
3+
4+
.. |SchemaLikeObject| replace:: :ref:`Schema like object <tutorial:tutorials/advanced/x-like:schema>`
5+
.. |ParamsLikeObject| replace:: :ref:`Parameters like object <tutorial:tutorials/advanced/x-like:parameters>`
6+
.. |DataFrameLikeObject| replace:: :ref:`DataFrame like object <tutorial:tutorials/advanced/x-like:dataframe>`
7+
.. |DataFramesLikeObject| replace:: :ref:`DataFrames like object <tutorial:tutorials/advanced/x-like:dataframes>`
8+
.. |PartitionLikeObject| replace:: :ref:`Partition like object <tutorial:tutorials/advanced/x-like:partition>`
9+
.. |RPCHandlerLikeObject| replace:: :ref:`RPChandler like object <tutorial:tutorials/advanced/x-like:rpc>`
10+
11+
.. |ExecutionEngine| replace:: :class:`~fugue.execution.execution_engine.ExecutionEngine`
12+
.. |NativeExecutionEngine| replace:: :class:`~fugue.execution.native_execution_engine.NativeExecutionEngine`
13+
.. |FugueWorkflow| replace:: :class:`~fugue.workflow.workflow.FugueWorkflow`
14+
15+
.. |ReadJoin| replace:: Read Join tutorials on :ref:`workflow <tutorial:tutorials/advanced/dag:join>` and :ref:`engine <tutorial:tutorials/advanced/execution_engine:join>` for details
16+
.. |FugueConfig| replace:: :doc:`the Fugue Configuration Tutorial <tutorial:tutorials/advanced/useful_config>`
17+
.. |PartitionTutorial| replace:: :doc:`the Partition Tutorial <tutorial:tutorials/advanced/partition>`
18+
.. |FugueSQLTutorial| replace:: :doc:`the Fugue SQL Tutorial <tutorial:tutorials/fugue_sql/index>`
19+
.. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:tutorials/advanced/schema_dataframes:dataframe>`
20+
.. |ExecutionEngineTutorial| replace:: :doc:`the ExecutionEngine Tutorial <tutorial:tutorials/advanced/execution_engine>`
21+
.. |ZipComap| replace:: :ref:`Zip & Comap <tutorial:tutorials/advanced/execution_engine:zip & comap>`
22+
.. |LoadSave| replace:: :ref:`Load & Save <tutorial:tutorials/advanced/execution_engine:load & save>`
23+
.. |AutoPersist| replace:: :ref:`Auto Persist <tutorial:tutorials/advanced/useful_config:auto persist>`
24+
.. |TransformerTutorial| replace:: :doc:`the Transformer Tutorial <tutorial:tutorials/extensions/transformer>`
25+
.. |CoTransformer| replace:: :ref:`CoTransformer <tutorial:tutorials/advanced/dag:cotransformer>`
26+
.. |CoTransformerTutorial| replace:: :doc:`the CoTransformer Tutorial <tutorial:tutorials/extensions/cotransformer>`
27+
.. |FugueDataTypes| replace:: :doc:`Fugue Data Types <tutorial:tutorials/appendix/generate_types>`
28+
29+
30+
fugue.dataset.api
31+
-----------------
32+
33+
.. automodule:: fugue.dataset.api
34+
:members:
35+
:undoc-members:
36+
:show-inheritance:
37+
38+
fugue.dataset.dataset
39+
---------------------
40+
41+
.. automodule:: fugue.dataset.dataset
42+
:members:
43+
:undoc-members:
44+
:show-inheritance:
45+

‎docs/api/fugue.execution.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ fugue.execution
2727
.. |FugueDataTypes| replace:: :doc:`Fugue Data Types <tutorial:tutorials/appendix/generate_types>`
2828

2929

30+
fugue.execution.api
31+
-------------------
32+
33+
.. automodule:: fugue.execution.api
34+
:members:
35+
:undoc-members:
36+
:show-inheritance:
37+
3038
fugue.execution.execution\_engine
3139
---------------------------------
3240

‎docs/api/fugue.rst

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ fugue
88
fugue.collections
99
fugue.column
1010
fugue.dataframe
11+
fugue.dataset
1112
fugue.execution
1213
fugue.extensions
1314
fugue.rpc
@@ -40,18 +41,18 @@ fugue
4041
.. |FugueDataTypes| replace:: :doc:`Fugue Data Types <tutorial:tutorials/appendix/generate_types>`
4142

4243

43-
fugue.constants
44-
---------------
44+
fugue.api
45+
---------
4546

46-
.. automodule:: fugue.constants
47+
.. automodule:: fugue.api
4748
:members:
4849
:undoc-members:
4950
:show-inheritance:
5051

51-
fugue.dataset
52-
-------------
52+
fugue.constants
53+
---------------
5354

54-
.. automodule:: fugue.dataset
55+
.. automodule:: fugue.constants
5556
:members:
5657
:undoc-members:
5758
:show-inheritance:
@@ -64,10 +65,10 @@ fugue.exceptions
6465
:undoc-members:
6566
:show-inheritance:
6667

67-
fugue.interfaceless
68-
-------------------
68+
fugue.plugins
69+
-------------
6970

70-
.. automodule:: fugue.interfaceless
71+
.. automodule:: fugue.plugins
7172
:members:
7273
:undoc-members:
7374
:show-inheritance:

‎docs/api/fugue.sql.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ fugue.sql
2727
.. |FugueDataTypes| replace:: :doc:`Fugue Data Types <tutorial:tutorials/appendix/generate_types>`
2828

2929

30+
fugue.sql.api
31+
-------------
32+
33+
.. automodule:: fugue.sql.api
34+
:members:
35+
:undoc-members:
36+
:show-inheritance:
37+
3038
fugue.sql.workflow
3139
------------------
3240

‎docs/api/fugue.workflow.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ fugue.workflow
2727
.. |FugueDataTypes| replace:: :doc:`Fugue Data Types <tutorial:tutorials/appendix/generate_types>`
2828

2929

30+
fugue.workflow.api
31+
------------------
32+
33+
.. automodule:: fugue.workflow.api
34+
:members:
35+
:undoc-members:
36+
:show-inheritance:
37+
3038
fugue.workflow.input
3139
--------------------
3240

‎docs/index.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,5 +33,7 @@ For contributing, start with the `contributing guide <https://github.com/fugue-p
3333
:maxdepth: 3
3434
:hidden:
3535

36+
tutorials
37+
top_api
3638
api
3739

‎docs/top_api.rst

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
Top Level User API Reference
2+
============================
3+
4+
.. |SchemaLikeObject| replace:: :ref:`Schema like object <tutorial:tutorials/advanced/x-like:schema>`
5+
.. |ParamsLikeObject| replace:: :ref:`Parameters like object <tutorial:tutorials/advanced/x-like:parameters>`
6+
.. |DataFrameLikeObject| replace:: :ref:`DataFrame like object <tutorial:tutorials/advanced/x-like:dataframe>`
7+
.. |DataFramesLikeObject| replace:: :ref:`DataFrames like object <tutorial:tutorials/advanced/x-like:dataframes>`
8+
.. |PartitionLikeObject| replace:: :ref:`Partition like object <tutorial:tutorials/advanced/x-like:partition>`
9+
.. |RPCHandlerLikeObject| replace:: :ref:`RPChandler like object <tutorial:tutorials/advanced/x-like:rpc>`
10+
11+
.. |ExecutionEngine| replace:: :class:`~fugue.execution.execution_engine.ExecutionEngine`
12+
.. |NativeExecutionEngine| replace:: :class:`~fugue.execution.native_execution_engine.NativeExecutionEngine`
13+
.. |FugueWorkflow| replace:: :class:`~fugue.workflow.workflow.FugueWorkflow`
14+
15+
.. |ReadJoin| replace:: Read Join tutorials on :ref:`workflow <tutorial:tutorials/advanced/dag:join>` and :ref:`engine <tutorial:tutorials/advanced/execution_engine:join>` for details
16+
.. |FugueConfig| replace:: :doc:`the Fugue Configuration Tutorial <tutorial:tutorials/advanced/useful_config>`
17+
.. |PartitionTutorial| replace:: :doc:`the Partition Tutorial <tutorial:tutorials/advanced/partition>`
18+
.. |FugueSQLTutorial| replace:: :doc:`the Fugue SQL Tutorial <tutorial:tutorials/fugue_sql/index>`
19+
.. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:tutorials/advanced/schema_dataframes:dataframe>`
20+
.. |ExecutionEngineTutorial| replace:: :doc:`the ExecutionEngine Tutorial <tutorial:tutorials/advanced/execution_engine>`
21+
.. |ZipComap| replace:: :ref:`Zip & Comap <tutorial:tutorials/advanced/execution_engine:zip & comap>`
22+
.. |LoadSave| replace:: :ref:`Load & Save <tutorial:tutorials/advanced/execution_engine:load & save>`
23+
.. |AutoPersist| replace:: :ref:`Auto Persist <tutorial:tutorials/advanced/useful_config:auto persist>`
24+
.. |TransformerTutorial| replace:: :doc:`the Transformer Tutorial <tutorial:tutorials/extensions/transformer>`
25+
.. |CoTransformer| replace:: :ref:`CoTransformer <tutorial:tutorials/advanced/dag:cotransformer>`
26+
.. |CoTransformerTutorial| replace:: :doc:`the CoTransformer Tutorial <tutorial:tutorials/extensions/cotransformer>`
27+
.. |FugueDataTypes| replace:: :doc:`Fugue Data Types <tutorial:tutorials/appendix/generate_types>`
28+
29+
IO
30+
~~
31+
32+
.. autofunction:: fugue.api.as_fugue_dataset
33+
34+
.. autofunction:: fugue.api.as_fugue_df
35+
.. autofunction:: fugue.api.load
36+
.. autofunction:: fugue.api.save
37+
38+
39+
40+
Information
41+
~~~~~~~~~~~
42+
43+
.. autofunction:: fugue.api.count
44+
.. autofunction:: fugue.api.is_bounded
45+
.. autofunction:: fugue.api.is_empty
46+
.. autofunction:: fugue.api.is_local
47+
.. autofunction:: fugue.api.show
48+
49+
.. autofunction:: fugue.api.get_column_names
50+
.. autofunction:: fugue.api.get_num_partitions
51+
.. autofunction:: fugue.api.get_schema
52+
.. autofunction:: fugue.api.is_df
53+
.. autofunction:: fugue.api.peek_array
54+
.. autofunction:: fugue.api.peek_dict
55+
56+
57+
Transformation
58+
~~~~~~~~~~~~~~
59+
60+
.. autofunction:: fugue.api.transform
61+
.. autofunction:: fugue.api.out_transform
62+
63+
.. autofunction:: fugue.api.alter_columns
64+
.. autofunction:: fugue.api.drop_columns
65+
.. autofunction:: fugue.api.head
66+
.. autofunction:: fugue.api.normalize_column_names
67+
.. autofunction:: fugue.api.rename
68+
.. autofunction:: fugue.api.select_columns
69+
70+
.. autofunction:: fugue.api.distinct
71+
.. autofunction:: fugue.api.dropna
72+
.. autofunction:: fugue.api.fillna
73+
.. autofunction:: fugue.api.sample
74+
.. autofunction:: fugue.api.take
75+
76+
SQL
77+
~~~
78+
79+
.. autofunction:: fugue.api.fugue_sql
80+
.. autofunction:: fugue.api.fugue_sql_flow
81+
.. autofunction:: fugue.api.raw_sql
82+
83+
.. autofunction:: fugue.api.join
84+
.. autofunction:: fugue.api.semi_join
85+
.. autofunction:: fugue.api.anti_join
86+
.. autofunction:: fugue.api.inner_join
87+
.. autofunction:: fugue.api.left_outer_join
88+
.. autofunction:: fugue.api.right_outer_join
89+
.. autofunction:: fugue.api.full_outer_join
90+
.. autofunction:: fugue.api.cross_join
91+
92+
.. autofunction:: fugue.api.union
93+
.. autofunction:: fugue.api.intersect
94+
.. autofunction:: fugue.api.subtract
95+
96+
.. autofunction:: fugue.api.assign
97+
.. autofunction:: fugue.api.select
98+
.. autofunction:: fugue.api.filter
99+
.. autofunction:: fugue.api.aggregate
100+
101+
Conversion
102+
~~~~~~~~~~
103+
104+
.. autofunction:: fugue.api.as_local
105+
.. autofunction:: fugue.api.as_local_bounded
106+
.. autofunction:: fugue.api.as_array
107+
.. autofunction:: fugue.api.as_array_iterable
108+
.. autofunction:: fugue.api.as_arrow
109+
.. autofunction:: fugue.api.as_dict_iterable
110+
.. autofunction:: fugue.api.as_pandas
111+
.. autofunction:: fugue.api.get_native_as_df
112+
113+
ExecutionEngine
114+
~~~~~~~~~~~~~~~
115+
116+
.. autofunction:: fugue.api.engine_context
117+
.. autofunction:: fugue.api.set_global_engine
118+
.. autofunction:: fugue.api.clear_global_engine
119+
.. autofunction:: fugue.api.get_current_engine
120+
.. autofunction:: get_current_parallelism
121+
122+
123+
Big Data Operations
124+
~~~~~~~~~~~~~~~~~~~
125+
.. autofunction:: fugue.api.broadcast
126+
.. autofunction:: fugue.api.persist
127+
.. autofunction:: fugue.api.repartition
128+
129+
130+
Development
131+
~~~~~~~~~~~
132+
133+
.. autofunction:: fugue.api.run_engine_function
134+
135+
136+
137+
138+

0 commit comments

Comments
 (0)