Skip to content

Commit ca215ca

Browse files
whynowywhynowy
committed
Add python client
Signed-off-by: Derek Wang <whynowy@gmail.com>
1 parent fe530bb commit ca215ca

File tree

153 files changed

+24212
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

153 files changed

+24212
-1
lines changed

‎.gitignore‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22

33
/java/.openapi-generator/
44
/java/api/
5-
/client/.gitignore
65
/java/.openapi-generator-ignore
76
/java/build.*
87
/java/gradle*
@@ -11,3 +10,9 @@
1110
/java/target/
1211
/java/git_push.sh
1312
/java/.travis.yml
13+
14+
/python/.openapi-generator/
15+
/python/.openapi-generator-ignore
16+
/python/test
17+
/python/git_push.sh
18+
/python/dist/

‎python/.gitignore‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
env/
12+
build/
13+
develop-eggs/
14+
dist/
15+
downloads/
16+
eggs/
17+
.eggs/
18+
lib/
19+
lib64/
20+
parts/
21+
sdist/
22+
var/
23+
*.egg-info/
24+
.installed.cfg
25+
*.egg
26+
27+
# PyInstaller
28+
# Usually these files are written by a python script from a template
29+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
30+
*.manifest
31+
*.spec
32+
33+
# Installer logs
34+
pip-log.txt
35+
pip-delete-this-directory.txt
36+
37+
# Unit test / coverage reports
38+
htmlcov/
39+
.tox/
40+
.coverage
41+
.coverage.*
42+
.cache
43+
nosetests.xml
44+
coverage.xml
45+
*,cover
46+
.hypothesis/
47+
venv/
48+
.venv/
49+
.python-version
50+
.pytest_cache
51+
52+
# Translations
53+
*.mo
54+
*.pot
55+
56+
# Django stuff:
57+
*.log
58+
59+
# Sphinx documentation
60+
docs/_build/
61+
62+
# PyBuilder
63+
target/
64+
65+
#Ipython Notebook
66+
.ipynb_checkpoints

‎python/.gitlab-ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# ref: https://docs.gitlab.com/ee/ci/README.html
2+
3+
stages:
4+
- test
5+
6+
.tests:
7+
stage: test
8+
script:
9+
- pip install -r requirements.txt
10+
- pip install -r test-requirements.txt
11+
- pytest --cov=numaflow
12+
13+
test-3.6:
14+
extends: .tests
15+
image: python:3.6-alpine
16+
test-3.7:
17+
extends: .tests
18+
image: python:3.7-alpine
19+
test-3.8:
20+
extends: .tests
21+
image: python:3.8-alpine
22+
test-3.9:
23+
extends: .tests
24+
image: python:3.9-alpine

‎python/.travis.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# ref: https://docs.travis-ci.com/user/languages/python
2+
language: python
3+
python:
4+
- "3.6"
5+
- "3.7"
6+
- "3.8"
7+
- "3.9"
8+
# command to install dependencies
9+
install:
10+
- "pip install -r requirements.txt"
11+
- "pip install -r test-requirements.txt"
12+
# command to run tests
13+
script: pytest --cov=numaflow

‎python/Makefile‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
VERSION ?= main
2+
3+
SDK_VERSION := $(shell if [[ "$(VERSION)" =~ ^v[0-9]+\.[0-9]+\.[0-9]+.* ]]; then echo ${VERSION} | cut -c2-; elif [[ "$(VERSION)" =~ ^main$$ ]]; then echo 0.0.0-pre; else echo $(VERSION); fi)
4+
5+
GENERATOR_VERSION := v5.4.0
6+
7+
DOCKER = docker run --rm -v `pwd -P`:/base --workdir /base
8+
9+
publish: generate
10+
echo TODO
11+
12+
generate:
13+
rm -Rf ./docs ./test ./numaflow/models/ ./numaflow/model/
14+
mkdir -p ./dist
15+
cat ../swagger.json | ../hack/swaggerfilter.py io.numaproj.numaflow | \
16+
sed 's/io.k8s.api.core.//' | \
17+
sed 's/io.k8s.apimachinery.pkg.apis.meta.//' | \
18+
sed 's/io.k8s.apimachinery.pkg.api.//' | \
19+
sed 's/io.numaproj.numaflow.v1alpha1.//' \
20+
> ./dist/swagger.json
21+
$(DOCKER) openapitools/openapi-generator-cli:$(GENERATOR_VERSION) \
22+
generate \
23+
-i /base/dist/swagger.json \
24+
-g python \
25+
-o /base \
26+
--additional-properties packageVersion=${SDK_VERSION} \
27+
--additional-properties packageName="numaflow" \
28+
--additional-properties projectName="numaflow" \
29+
--additional-properties hideGenerationTimestamp=true \
30+
--remove-operation-id-prefix \
31+
--model-name-prefix '' \
32+
--model-name-suffix '' \
33+
--artifact-id numaflow-python-client \
34+
--global-property modelTests=false \
35+
--global-property packageName=numaflow \
36+
--import-mappings V1ResourceRequirements="from kubernetes.client import V1ResourceRequirements" \
37+
--import-mappings V1SecretKeySelector="from kubernetes.client import V1SecretKeySelector" \
38+
--generate-alias-as-model
39+
echo "kubernetes >= 22.6.0" >> requirements.txt
40+
41+
install:
42+
pip3 install ./

‎python/README.md‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
# numaflow
2+
No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator)
3+
4+
This Python package is automatically generated by the [OpenAPI Generator](https://openapi-generator.tech) project:
5+
6+
- API version: v0.7.0
7+
- Package version: 0.7.0
8+
- Build package: org.openapitools.codegen.languages.PythonClientCodegen
9+
10+
## Requirements.
11+
12+
Python >=3.6
13+
14+
## Installation & Usage
15+
### pip install
16+
17+
If the python package is hosted on a repository, you can install directly using:
18+
19+
```sh
20+
pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git
21+
```
22+
(you may need to run `pip` with root permission: `sudo pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git`)
23+
24+
Then import the package:
25+
```python
26+
import numaflow
27+
```
28+
29+
### Setuptools
30+
31+
Install via [Setuptools](http://pypi.python.org/pypi/setuptools).
32+
33+
```sh
34+
python setup.py install --user
35+
```
36+
(or `sudo python setup.py install` to install the package for all users)
37+
38+
Then import the package:
39+
```python
40+
import numaflow
41+
```
42+
43+
## Getting Started
44+
45+
Please follow the [installation procedure](#installation--usage) and then run the following:
46+
47+
```python
48+
49+
import time
50+
import numaflow
51+
from pprint import pprint
52+
```
53+
54+
## Documentation for API Endpoints
55+
56+
All URIs are relative to *http://localhost*
57+
58+
Class | Method | HTTP request | Description
59+
------------ | ------------- | ------------- | -------------
60+
61+
62+
## Documentation For Models
63+
64+
- [AbstractPodTemplate](docs/AbstractPodTemplate.md)
65+
- [AbstractVertex](docs/AbstractVertex.md)
66+
- [Authorization](docs/Authorization.md)
67+
- [BasicAuth](docs/BasicAuth.md)
68+
- [Buffer](docs/Buffer.md)
69+
- [BufferServiceConfig](docs/BufferServiceConfig.md)
70+
- [Container](docs/Container.md)
71+
- [ContainerBuilder](docs/ContainerBuilder.md)
72+
- [ContainerTemplate](docs/ContainerTemplate.md)
73+
- [DaemonTemplate](docs/DaemonTemplate.md)
74+
- [Edge](docs/Edge.md)
75+
- [EdgeLimits](docs/EdgeLimits.md)
76+
- [FixedWindow](docs/FixedWindow.md)
77+
- [ForwardConditions](docs/ForwardConditions.md)
78+
- [Function](docs/Function.md)
79+
- [GeneratorSource](docs/GeneratorSource.md)
80+
- [GetContainerReq](docs/GetContainerReq.md)
81+
- [GetDaemonDeploymentReq](docs/GetDaemonDeploymentReq.md)
82+
- [GetJetStreamServiceSpecReq](docs/GetJetStreamServiceSpecReq.md)
83+
- [GetJetStreamStatefulSetSpecReq](docs/GetJetStreamStatefulSetSpecReq.md)
84+
- [GetRedisServiceSpecReq](docs/GetRedisServiceSpecReq.md)
85+
- [GetRedisStatefulSetSpecReq](docs/GetRedisStatefulSetSpecReq.md)
86+
- [GetVertexPodSpecReq](docs/GetVertexPodSpecReq.md)
87+
- [GroupBy](docs/GroupBy.md)
88+
- [HTTPSource](docs/HTTPSource.md)
89+
- [InterStepBufferService](docs/InterStepBufferService.md)
90+
- [InterStepBufferServiceList](docs/InterStepBufferServiceList.md)
91+
- [InterStepBufferServiceSpec](docs/InterStepBufferServiceSpec.md)
92+
- [InterStepBufferServiceStatus](docs/InterStepBufferServiceStatus.md)
93+
- [JetStreamBufferService](docs/JetStreamBufferService.md)
94+
- [JetStreamConfig](docs/JetStreamConfig.md)
95+
- [JobTemplate](docs/JobTemplate.md)
96+
- [KafkaSink](docs/KafkaSink.md)
97+
- [KafkaSource](docs/KafkaSource.md)
98+
- [Lifecycle](docs/Lifecycle.md)
99+
- [Metadata](docs/Metadata.md)
100+
- [NativeRedis](docs/NativeRedis.md)
101+
- [NatsAuth](docs/NatsAuth.md)
102+
- [NatsSource](docs/NatsSource.md)
103+
- [PBQStorage](docs/PBQStorage.md)
104+
- [PersistenceStrategy](docs/PersistenceStrategy.md)
105+
- [Pipeline](docs/Pipeline.md)
106+
- [PipelineLimits](docs/PipelineLimits.md)
107+
- [PipelineList](docs/PipelineList.md)
108+
- [PipelineSpec](docs/PipelineSpec.md)
109+
- [PipelineStatus](docs/PipelineStatus.md)
110+
- [RedisBufferService](docs/RedisBufferService.md)
111+
- [RedisConfig](docs/RedisConfig.md)
112+
- [RedisSettings](docs/RedisSettings.md)
113+
- [Scale](docs/Scale.md)
114+
- [Sink](docs/Sink.md)
115+
- [SlidingWindow](docs/SlidingWindow.md)
116+
- [Source](docs/Source.md)
117+
- [Status](docs/Status.md)
118+
- [TLS](docs/TLS.md)
119+
- [Templates](docs/Templates.md)
120+
- [UDF](docs/UDF.md)
121+
- [UDSink](docs/UDSink.md)
122+
- [Vertex](docs/Vertex.md)
123+
- [VertexInstance](docs/VertexInstance.md)
124+
- [VertexLimits](docs/VertexLimits.md)
125+
- [VertexList](docs/VertexList.md)
126+
- [VertexSpec](docs/VertexSpec.md)
127+
- [VertexStatus](docs/VertexStatus.md)
128+
- [Watermark](docs/Watermark.md)
129+
- [Window](docs/Window.md)
130+
131+
132+
## Documentation For Authorization
133+
134+
All endpoints do not require authorization.
135+
136+
## Author
137+
138+
139+
140+
## Notes for Large OpenAPI documents
141+
If the OpenAPI document is large, imports in numaflow.apis and numaflow.models may fail with a
142+
RecursionError indicating the maximum recursion limit has been exceeded. In that case, there are a couple of solutions:
143+
144+
Solution 1:
145+
Use specific imports for apis and models like:
146+
- `from numaflow.api.default_api import DefaultApi`
147+
- `from numaflow.model.pet import Pet`
148+
149+
Solution 2:
150+
Before importing the package, adjust the maximum recursion limit as shown below:
151+
```
152+
import sys
153+
sys.setrecursionlimit(1500)
154+
import numaflow
155+
from numaflow.apis import *
156+
from numaflow.models import *
157+
```
158+

‎python/docs/AbstractPodTemplate.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# AbstractPodTemplate
2+
3+
AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on.
4+
5+
## Properties
6+
Name | Type | Description | Notes
7+
------------ | ------------- | ------------- | -------------
8+
**affinity** | [**V1Affinity**](V1Affinity.md) | | [optional]
9+
**image_pull_secrets** | [**[V1LocalObjectReference]**](V1LocalObjectReference.md) | ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod | [optional]
10+
**metadata** | [**Metadata**](Metadata.md) | | [optional]
11+
**node_selector** | **{str: (str,)}** | NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node&#39;s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ | [optional]
12+
**priority** | **int** | The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ | [optional]
13+
**priority_class_name** | **str** | If specified, indicates the Redis pod&#39;s priority. \&quot;system-node-critical\&quot; and \&quot;system-cluster-critical\&quot; are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ | [optional]
14+
**security_context** | [**V1PodSecurityContext**](V1PodSecurityContext.md) | | [optional]
15+
**service_account_name** | **str** | ServiceAccountName applied to the pod | [optional]
16+
**tolerations** | [**[V1Toleration]**](V1Toleration.md) | If specified, the pod&#39;s tolerations. | [optional]
17+
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
18+
19+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
20+
21+

‎python/docs/AbstractVertex.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# AbstractVertex
2+
3+
4+
## Properties
5+
Name | Type | Description | Notes
6+
------------ | ------------- | ------------- | -------------
7+
**name** | **str** | |
8+
**affinity** | [**V1Affinity**](V1Affinity.md) | | [optional]
9+
**container_template** | [**ContainerTemplate**](ContainerTemplate.md) | | [optional]
10+
**image_pull_secrets** | [**[V1LocalObjectReference]**](V1LocalObjectReference.md) | ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod | [optional]
11+
**init_container_template** | [**ContainerTemplate**](ContainerTemplate.md) | | [optional]
12+
**init_containers** | [**[V1Container]**](V1Container.md) | List of init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ | [optional]
13+
**limits** | [**VertexLimits**](VertexLimits.md) | | [optional]
14+
**metadata** | [**Metadata**](Metadata.md) | | [optional]
15+
**node_selector** | **{str: (str,)}** | NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node&#39;s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ | [optional]
16+
**priority** | **int** | The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ | [optional]
17+
**priority_class_name** | **str** | If specified, indicates the Redis pod&#39;s priority. \&quot;system-node-critical\&quot; and \&quot;system-cluster-critical\&quot; are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ | [optional]
18+
**scale** | [**Scale**](Scale.md) | | [optional]
19+
**security_context** | [**V1PodSecurityContext**](V1PodSecurityContext.md) | | [optional]
20+
**service_account_name** | **str** | ServiceAccountName applied to the pod | [optional]
21+
**sidecars** | [**[V1Container]**](V1Container.md) | List of sidecar containers belonging to the pod. | [optional]
22+
**sink** | [**Sink**](Sink.md) | | [optional]
23+
**source** | [**Source**](Source.md) | | [optional]
24+
**tolerations** | [**[V1Toleration]**](V1Toleration.md) | If specified, the pod&#39;s tolerations. | [optional]
25+
**udf** | [**UDF**](UDF.md) | | [optional]
26+
**volumes** | [**[V1Volume]**](V1Volume.md) | | [optional]
27+
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
28+
29+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
30+
31+

‎python/docs/Authorization.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Authorization
2+
3+
4+
## Properties
5+
Name | Type | Description | Notes
6+
------------ | ------------- | ------------- | -------------
7+
**token** | [**V1SecretKeySelector**](V1SecretKeySelector.md) | | [optional]
8+
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
9+
10+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
11+
12+

0 commit comments

Comments
 (0)