Feat: add airbyte app (#71)

Signed-off-by: xingcan-ltc <[email protected]>

Author: xingcan-hu

catalog/airbyte/README.md

@@ -0,0 +1,18 @@
+### 1. 引言
+Airbyte 是一个开源的数据集成平台，提供 Airbyte 开源版和 Airbyte Cloud 解决方案。它使用户能够从多种数据源同步数据到目的地，便于数据的整合，以供分析和商业智能使用。
+
+### 2. 产品特性
+#### 2.1 广泛的连接器目录
+拥有全面的连接器列表，适用于各种数据源（数据库、API、SaaS 应用程序等）。300 多个预构建的连接器是业界最大的，并且每年都在翻倍增长。
+
+#### 2.2 易于使用
+用户友好的界面使用户能够轻松设计和维护数据管道，可通过 UI、API 和 Terraform 访问。平台提供无忧的配置选项和简化的数据连接器设置。
+
+#### 2.3 完全托管和可扩展
+完全托管并针对云的可扩展性进行了优化，能够适应特定任务不断增长的数据需求。能够高效处理大量数据，并在不遇到问题或失败的情况下缓解数据偏斜。
+
+#### 2.4 开源社区
+Airbyte 开源社区为用户提供了一个透明且易于访问的平台，鼓励知识交流、故障排除，并为全球数据集成爱好者创建了一个支持性的生态系统。
+
+#### 2.5 监控和管理
+平台配备了内置的监控工具，允许用户通过电子邮件、Web Hooks 以及各种目标系统无缝跟踪数据管道性能。

catalog/airbyte/apps/airbyte.app/README.md

@@ -0,0 +1,74 @@
+### 1. 简介
+Airbyte 是一个开源的数据移动基础设施，用于构建提取和加载（EL）数据管道。它被设计为多功能、可扩展且易于使用。
+
+### 2. 核心概念
+**源**
+源是一个API、文件、数据库或数据仓库，您希望从中摄取数据。
+
+**目的地**
+目的地是一个数据仓库、数据湖、数据库或分析工具，您希望将摄取的数据加载到此处。
+
+**连接器**
+连接器是Airbyte的一个组件，用于从源拉取数据或将数据推送到目的地。
+
+**连接**
+连接是一个自动化的数据管道，用于将数据从源复制到目的地。
+
+更多详情，请参考官方文档：https://docs.airbyte.com/using-airbyte/core-concepts/
+
+### 2. 快速开始
+
+安装Airbyte后，您可以打开Airbyte入口（例如 http://airbyte-kdp-data.kdp-e2e.io）。然后输入任何电子邮件和组织名称。之后，您可以创建一个连接。
+
+#### 2.1 添加源
+
+添加一个“Faker”源。
+
+点击“Sources”，搜索“faker”然后选择“Faker”，点击“Set up source”。
+Airbyte将测试源并显示连接状态。（Airbyte将启动一个Pod来测试连接，这可能需要几分钟时间）
+
+更多详情，请参考官方文档：https://docs.airbyte.com/using-airbyte/getting-started/add-a-source
+
+#### 2.2 添加目的地
+
+添加一个“S3”目的地。在添加目的地之前，您需要创建一个Minio桶：
+```bash
+kubectl exec -it airbyte-minio-0 -n kdp-data  -- bash
+mc alias set local http://localhost:9000 minio minio123 
+mc mb local/tmp
+```
+
+点击“Destinations”，搜索“S3”然后选择它，输入以下字段：
+
+- S3 Key ID: `minio`
+- S3 Access Key: `minio123`
+- S3 Bucket Name: `tmp`
+- S3 Bucket Path: `/`
+- S3 Bucket Region: 选择任何区域
+
+可选字段：
+
+- S3 Endpoint: http://airbyte-minio-svc:9000
+
+点击“Set up destination”。
+
+2.3 创建连接
+点击“Connections”，点击“Create connection”，选择您刚刚创建的源和目的地。
+
+- Define source: `faker`
+- Define destination: `S3`
+- Select the stream: click `Next` button
+- Conifgure connection: click `Finish & Sync` button
+
+
+同步任务将被触发，您可以在UI中看到同步状态。（下载image和启动Pod可能需要几分钟时间。）
+
+如果同步成功，您可以在Minio桶中看到数据。
+
+```bash
+kubectl exec -it airbyte-minio-0 -n kdp-data  -- bash
+# list the bucket
+mc ls -r local/tmp
+# you can delete the bucket if you want
+mc rm -r local/tmp
+```

catalog/airbyte/apps/airbyte.app/app.yaml

@@ -0,0 +1,14 @@
+apiVersion: bdc.kdp.io/v1alpha1
+kind: Application
+metadata:
+  annotations:
+    app.core.bdos/catalog: airbyte
+  labels:
+    app: airbyte
+    app.core.bdos/type: system
+spec:
+  name: airbyte
+  type: airbyte
+  properties:
+    server: 
+      replicaCount: 1

catalog/airbyte/apps/airbyte.app/i18n/en/README.md

@@ -0,0 +1,78 @@
+### 1. Introduction
+Airbyte is an open-source data movement infrastructure for building extract and load (EL) data pipelines. It is designed for versatility, scalability, and ease-of-use.
+
+
+### 2. Core Concepts
+**Source**
+A source is an API, file, database, or data warehouse that you want to ingest data from.
+
+**Destination**
+A destination is a data warehouse, data lake, database, or an analytics tool where you want to load your ingested data.
+
+**Connector**
+An Airbyte component which pulls data from a source or pushes data to a destination.
+
+**Connection**
+A connection is an automated data pipeline that replicates data from a source to a destination.
+
+For more details, refer to the official documentation: https://docs.airbyte.com/using-airbyte/core-concepts/
+
+
+### 2. Quick Start
+
+After installing Airbyte, you can open the Airbyte ingress (like http://airbyte-kdp-data.kdp-e2e.io). Then input any email and organization name. After that, you can create a connection.
+
+#### 2.1 Add a Source
+
+Add a "Faker" source.
+
+Click "Sources", search for "faker" then select "Faker", click "Set up source". 
+Airbyte will test the source and show the connection status. (Airbyte will launch a Pod to test the connection that mqy take a few minutes)
+
+For more details, refer to the official documentation: https://docs.airbyte.com/using-airbyte/getting-started/add-a-source
+
+#### 2.2 Add a Destination
+
+Add a "S3" destination. Before adding a destination, you need to create a Minio bucket:
+```bash
+kubectl exec -it airbyte-minio-0 -n kdp-data  -- bash
+mc alias set local http://localhost:9000 minio minio123 
+mc mb local/tmp
+``` 
+
+Click "Destinations", search for "S3" then select it, input the following fields:
+
+- S3 Key ID: `minio`
+- S3 Access Key: `minio123`
+- S3 Bucket Name: `tmp`
+- S3 Bucket Path: `/`
+- S3 Bucket Region: select any region
+  
+Optional fields:
+- S3 Endpoint: `http://airbyte-minio-svc:9000`
+
+Click "Set up destination".
+
+#### 2.3 Create a Connection
+Click "Connections", click "Create connection", select the source and destination you just created. 
+- Define source: `faker`
+- Define destination: `S3`
+- Select the stream: click `Next` button
+- Conifgure connection: click `Finish & Sync` button
+
+Sync task will be triggered，you can see the sync status in the UI. (It will take a few minutes to download images and to launch pods.)
+
+
+If sync is successful, you can see the data in the Minio bucket.
+
+```bash
+kubectl exec -it airbyte-minio-0 -n kdp-data  -- bash
+# list the bucket
+mc ls -r local/tmp
+# you can delete the bucket if you want
+mc rm -r local/tmp
+```
+
+
+
+

catalog/airbyte/apps/airbyte.app/metadata.yaml

@@ -0,0 +1,8 @@
+version: 0.59.1
+alias: Airbyte
+description: Airbyte 是一个开源的数据移动基础设施，专注于构建提取和加载（EL）数据管道。它被设计为多功能、可扩展且易于使用。 https://artifacthub.io/packages/helm/airbyte/airbyte/0.67.17
+isGlobal: false
+i18n:
+  en:
+    description: >-
+      Airbyte is an open-source data movement infrastructure for building extract and load (EL) data pipelines. It is designed for versatility, scalability, and ease-of-use. https://artifacthub.io/packages/helm/airbyte/airbyte/0.67.17

catalog/airbyte/i18n/en/README.md

@@ -0,0 +1,20 @@
+### 1. Introduction
+Airbyte is an open-source data integration platform that offers both Airbyte Open Source and Airbyte Cloud solutions. It empowers users to synchronize data from various sources to destinations, facilitating the consolidation of data for analysis and business intelligence.
+
+### 2. Product Features
+#### 2.1 Extensive Connector Catalog
+Comprehensive list of connectors for various data sources (databases, APIs, SaaS applications, etc). 300+ pre-built connectors is the largest in the industry and is doubling every year for various data sources.
+
+#### 2.2 Ease of Use
+A user-friendly interface empowers users to design and manage data pipelines effortlessly, accessible via UI, API, and Terraform. The platform offers hassle-free configuration options and simplified setup for data connectors.
+
+#### 2.3 Fully Managed and Scalable
+Fully managed and optimized for cloud scalability, adept at accommodating the increasing data demands of specific tasks. Capable of efficiently handling large data volumes and mitigating data skewness without encountering issues or failures.
+
+#### 2.4 Open Source Community
+The Airbyte open-source community provides users with a transparent and accessible platform, encouraging knowledge exchange, troubleshooting, and the creation of a supportive ecosystem for data integration enthusiasts worldwide.
+
+#### 2.5 Monitoring and Management
+The platform comes equipped with built-in monitoring tools, allowing users to track data pipeline performance seamlessly via email, web hooks, and various target systems.
+
+

catalog/airbyte/metadata.yaml

@@ -0,0 +1,8 @@
+name: Airbyte
+category: 开发工具
+description: Airbyte 是一个开源的数据移动基础设施，专注于构建提取和加载（EL）数据管道。它被设计为多功能、可扩展且易于使用。 https://artifacthub.io/packages/helm/airbyte/airbyte/0.67.17
+i18n:
+  en:
+    category: devtools
+    description: >-
+      Airbyte is an open-source data movement infrastructure for building extract and load (EL) data pipelines. It is designed for versatility, scalability, and ease-of-use. https://artifacthub.io/packages/helm/airbyte/airbyte/0.67.17

catalog/airbyte/x-definitions/app-airbyte.cue

@@ -0,0 +1,140 @@
+"airbyte": {
+	annotations: {}
+	labels: {}
+	attributes: {
+		apiResource: {
+			definition: {
+				apiVersion: "bdc.kdp.io/v1alpha1"
+				kind:       "Application"
+				type:       "airbyte"
+			}
+		}
+	}
+	description: "airbyte"
+	type:        "xdefinition"
+}
+
+template: {
+	output: {
+		apiVersion: "core.oam.dev/v1beta1"
+		kind:       "Application"
+		metadata: {
+			name:      context["name"]
+			namespace: context["namespace"]
+		}
+		spec: {
+			components: [
+				{
+					name: context["name"]
+					properties: {
+						chart:           "airbyte"
+						releaseName:     context["name"]
+						repoType:        "oci"
+						targetNamespace: context["namespace"]
+						url:             context["helm_repo_url"]
+						values: {
+							webapp: {
+								replicaCount: 1
+							}
+							server: {
+								replicaCount: 1
+							}
+							worker: {
+								replicaCount: parameter.worker.replicaCount
+							}
+							"airbyte-api-server": {
+								replicaCount: 1
+							}
+							postgresql: {
+								enabled: true
+							}
+							minio: {
+								storage: {
+									volumeClaimValue: parameter.minio.storage.size
+								}
+							}
+							postgresql: {
+								enabled: true
+							}
+						}
+						version: "0.67.17"
+					}
+					traits: [
+						{
+							properties: {
+								rules: [
+									{
+										host: context["name"] + "-" + context["namespace"] + "." + context["ingress.root_domain"]
+										paths: [
+											{
+												path:        "/"
+												serviceName: "airbyte-airbyte-webapp-svc"
+												servicePort: 80
+											},
+										]
+									},
+
+								]
+								tls: [
+									{
+										hosts: [
+											context["name"] + "-" + context["namespace"] + "." + context["ingress.root_domain"],
+										]
+										tlsSecretName: context["ingress.tls_secret_name"]
+									},
+								]
+							}
+							type: "bdos-ingress"
+						},
+
+					]
+					type: "helm"
+				},
+
+			]
+
+		}
+	}
+
+	parameter: {
+		// +ui:description=Minio存储配置
+		// +ui:order=1
+		minio: {
+			// +ui:order=1
+			storage: {
+				// +ui:description=配置存储大小
+				// +pattern=^([1-9]\d*)(Ti|Gi|Mi)$
+				// +err:options={"pattern":"请输入正确格式，如1024Mi, 1Gi, 1Ti"}
+				size: *"500Mi" | string
+			}
+		}
+
+		// +ui:description=Worker配置
+		// +ui:order=2
+		worker: {
+			// +minimum=1
+			// +ui:description=副本数
+			replicaCount: *1 | int
+		}
+
+		// +ui:description=Webapp配置
+		// +ui:order=3
+		webapp: {
+			// +minimum=1
+			// +ui:description=副本数
+			replicaCount: *1 | int
+		}
+
+		// +ui:description=Server配置
+		// +ui:order=4
+		server: {
+			// +minimum=1
+			// +ui:description=副本数
+			replicaCount: *1 | int
+		}
+
+		// +ui:description= 配置环境变量，如 MAX_SYNC_WORKERS=5
+		// +ui:order=5
+		env_vars?: {...}
+	}
+}