Packages:
Resource Types:
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, MonoVertexSpec, NativeRedis, SideInputsManagerTemplate, VertexTemplate)
AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on.
|
Field |
Description |
|---|---|
|
|
(Optional)
Metadata sets the pods’s metadata, i.e. annotations and labels |
|
|
(Optional)
NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ |
|
|
(Optional)
If specified, the pod’s tolerations. |
|
|
(Optional)
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field. |
|
|
(Optional)
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod |
|
|
(Optional)
If specified, indicates the Redis pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
|
|
(Optional)
The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
|
|
(Optional)
The pod’s scheduling constraints More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ |
|
|
(Optional)
ServiceAccountName applied to the pod |
|
|
(Optional)
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class |
|
|
(Optional)
AutomountServiceAccountToken indicates whether a service account token should be automatically mounted. |
|
|
(Optional)
Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’. |
|
|
(Optional)
Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. |
|
|
(Optional)
ResourceClaims defines which ResourceClaims must be allocated and reserved before the Pod is allowed to start. The resources will be made available to those containers which consume them by name. |
(Appears on: Sink)
|
Field |
Description |
|---|---|
|
|
(Optional)
Log sink is used to write the data to the log. |
|
|
(Optional)
Kafka sink is used to write the data to the Kafka. |
|
|
(Optional)
Blackhole sink is used to write the data to the blackhole sink, which is a sink that discards all the data written to it. |
|
|
(Optional)
UDSink sink is used to write the data to the user-defined sink. |
(Appears on: PipelineSpec, VertexSpec)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Container template for the main numa container. |
|
|
(Optional)
Container template for all the vertex pod init containers spawned by numaflow, excluding the ones specified by the user. |
|
|
(Members of (Optional) |
|
|
(Optional) |
|
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings |
|
|
(Optional)
Settings for autoscaling |
|
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
|
|
(Optional)
List of customized sidecar containers belonging to the pod. |
|
|
(Optional)
Number of partitions of the vertex owned buffers. It applies to udf and sink vertices only. |
|
|
(Optional)
Names of the side inputs used in this vertex. |
|
|
(Optional)
Container template for the side inputs watcher container. |
|
|
(Optional)
The strategy to use to replace existing pods with new ones. |
|
|
(Optional)
Names of the serving store used in this vertex. |
(Appears on: Window)
AccumulatorWindow describes a special kind of SessionWindow (similar to Global Window) where output should always have monotonically increasing WM but it can be manipulated through event-time by reordering the messages. NOTE: Quite powerful, should not be abused; it can cause stalling of pipelines and leaks.
|
Field |
Description |
|---|---|
|
|
Timeout is the duration of inactivity after which the state of the accumulator is removed. |
(Appears on: HTTPSource, ServingSource)
|
Field |
Description |
|---|---|
|
(Optional)
A secret selector which contains bearer token To use this, the client needs to add “Authorization: Bearer ” in the header |
(Appears on: RetryStrategy)
Backoff defines parameters used to systematically configure the retry strategy.
|
Field |
Description |
|---|---|
|
|
(Optional)
Interval sets the delay to wait before retry, after a failure occurs. |
|
|
(Optional)
Steps defines the number of times to try writing to a sink including retries |
(Appears on: NatsAuth)
BasicAuth represents the basic authentication approach which contains a user name and a password.
|
Field |
Description |
|---|---|
|
(Optional)
Secret for auth user |
|
|
|
(Optional)
Secret for auth password |
(Appears on: AbstractSink)
Blackhole is a sink to emulate /dev/null
(Appears on: Edge)
(Appears on: InterStepBufferServiceStatus)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Appears on: VertexSpec)
CombinedEdge is a combination of Edge and some other properties such as vertex type, partitions, limits. It’s used to decorate the fromEdges and toEdges of the generated Vertex objects, so that in the vertex pod, it knows the properties of the connected vertices, for example, how many partitioned buffers I should write to, what is the write buffer length, etc.
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
From vertex type. |
|
|
(Optional)
The number of partitions of the from vertex, if not provided, the default value is set to “1”. |
|
|
(Optional) |
|
|
To vertex type. |
|
|
(Optional)
The number of partitions of the to vertex, if not provided, the default value is set to “1”. |
|
|
(Optional) |
ConditionType is a valid value of Condition.Type
(Appears on: ServingStore, SideInput, UDF, UDSink, UDSource, UDTransformer)
Container is used to define the container properties for user-defined functions, sinks, etc.
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
(Optional) |
|
|
(Optional) |
|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
(Optional) |
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, MonoVertexSpec, NativeRedis, SideInputsManagerTemplate, VertexTemplate)
ContainerTemplate defines customized spec for a container
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
(Optional) |
|
|
(Optional) |
|
|
|
(Optional) |
|
|
(Optional) |
(Appears on: MonoVertexSpec, Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Replicas is the number of desired replicas of the Deployment. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller |
|
|
(Optional) |
|
|
(Optional) |
(Appears on: CombinedEdge, PipelineSpec)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
(Optional)
Conditional forwarding, only allowed when “From” is a Source or UDF. |
|
|
(Optional)
OnFull specifies the behaviour for the write actions when the inter step buffer is full. There are currently two options, retryUntilSuccess and discardLatest. if not provided, the default value is set to “retryUntilSuccess” |
(Appears on: Window)
FixedWindow describes a fixed window
|
Field |
Description |
|---|---|
|
|
Length is the duration of the fixed window. |
|
|
(Optional)
Streaming should be set to true if the reduce udf is streaming. |
(Appears on: Edge)
|
Field |
Description |
|---|---|
|
|
Tags used to specify tags for conditional forwarding |
(Appears on: UDF)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Optional) |
|
|
(Optional) |
(Appears on: SASL)
GSSAPI represents a SASL GSSAPI config
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
UsernameSecret refers to the secret that contains the username |
|
|
valid inputs - KRB5_USER_AUTH, KRB5_KEYTAB_AUTH |
|
|
(Optional)
PasswordSecret refers to the secret that contains the password |
|
|
(Optional)
KeytabSecret refers to the secret that contains the keytab |
|
|
(Optional)
KerberosConfigSecret refers to the secret that contains the kerberos config |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Size of each generated message |
|
|
(Optional)
KeyCount is the number of unique keys in the payload |
|
|
(Optional)
Value is an optional uint64 value to be written in to the payload |
|
|
(Optional)
Jitter is the jitter for the message generation, used to simulate out of order messages for example if the jitter is 10s, then the message’s event time will be delayed by a random time between 0 and 10s which will result in the message being out of order by 0 to 10s |
|
|
(Optional)
ValueBlob is an optional string which is the base64 encoding of direct payload to send. This is useful for attaching a GeneratorSource to a true pipeline to test load behavior with true messages without requiring additional work to generate messages through the external source if present, the Value and MsgSize fields will be ignored. |
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Appears on: UDF)
GroupBy indicates it is a reducer UDF
|
Field |
Description |
|---|---|
|
|
Window describes the windowing strategy. |
|
|
(Optional) |
|
|
(Optional)
AllowedLateness allows late data to be included for the Reduce operation as long as the late data is not later than (Watermark - AllowedLateness). |
|
|
Storage is used to define the PBQ storage for a reduce vertex. |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
Whether to create a ClusterIP Service |
(Appears on: InterStepBufferServiceStatus)
(Appears on: GetDaemonDeploymentReq, GetSideInputDeploymentReq, GetVertexPodSpecReq, InterStepBufferServiceStatus)
(Appears on: Watermark)
|
Field |
Description |
|---|---|
|
|
Threshold is the duration after which a source is marked as Idle due to
lack of data. Ex: If watermark found to be idle after the Threshold
duration then the watermark is progressed by |
|
|
(Optional)
StepInterval is the duration between the subsequent increment of the
watermark as long the source remains Idle. The default value is 0s which
means that once we detect idle source, we will be incrementing the
watermark by |
|
|
IncrementBy is the duration to be added to the current watermark to progress the watermark when source is idling. |
|
Field |
Description |
||||
|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
|
||||
|
|
|||||
|
|
(Optional) |
(Appears on: InterStepBufferService)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Appears on: InterStepBufferService)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Appears on: InterStepBufferServiceSpec)
|
Field |
Description |
|---|---|
|
|
JetStream version, such as “2.7.1” |
|
|
JetStream StatefulSet size |
|
|
(Optional)
ContainerTemplate contains customized spec for NATS container |
|
|
(Optional)
ReloaderContainerTemplate contains customized spec for config reloader container |
|
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
|
|
(Optional) |
|
|
(Members of (Optional) |
|
|
(Optional)
Nats/JetStream configuration, if not specified, global settings in numaflow-controller-config will be used. See https://docs.nats.io/running-a-nats-service/configuration#limits and https://docs.nats.io/running-a-nats-service/configuration#jetstream. For limits, only “max_payload” is supported for configuration, defaults to 1048576 (1MB), not recommended to use values over 8388608 (8MB) but max_payload can be set up to 67108864 (64MB). For jetstream, only “max_memory_store” and “max_file_store” are supported for configuration, do not set “store_dir” as it has been hardcoded. |
|
|
(Optional)
Optional arguments to start nats-server. For example, “-D” to enable debugging output, “-DV” to enable debugging and tracing. Check https://docs.nats.io/ for all the available arguments. |
|
|
(Optional)
Optional configuration for the streams, consumers and buckets to be created in this JetStream service, if specified, it will be merged with the default configuration in numaflow-controller-config. It accepts a YAML format configuration, it may include 4 sections, “stream”, “consumer”, “otBucket” and “procBucket”. Available fields under “stream” include “retention” (e.g. interest, limits, workerQueue), “maxMsgs”, “maxAge” (e.g. 72h), “replicas” (1, 3, 5), “duplicates” (e.g. 5m). Available fields under “consumer” include “ackWait” (e.g. 60s) Available fields under “otBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). Available fields under “procBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). |
|
|
(Optional)
Whether encrypt the data at rest, defaults to false Enabling encryption might impact the performance, see https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest for the detail Toggling the value will impact encrypting/decrypting existing messages. |
|
|
(Optional)
Whether enable TLS, defaults to false Enabling TLS might impact the performance |
(Appears on: BufferServiceConfig)
|
Field |
Description |
|---|---|
|
|
JetStream (NATS) URL |
|
|
|
|
|
(Optional) |
|
|
TLS enabled or not |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
|
|
Stream represents the name of the stream. |
|
|
(Optional)
TLS configuration for the nats client. |
|
|
(Optional)
Auth information |
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional) |
|
|
(Optional)
ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won’t be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. Numaflow defaults to 30 |
|
|
(Optional)
Specifies the number of retries before marking this job failed. More info: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy Numaflow defaults to 20 |
(Appears on: GSSAPI)
KRB5AuthType describes the kerberos auth type
(Appears on: AbstractSink)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
(Optional)
SetKey sets the Kafka key to the keys passed in the Message. When the key is null (default), the record is sent randomly to one of the available partitions of the topic. If a key exists, Kafka hashes the key, and the result is used to map the message to a specific partition. This ensures that messages with the same key end up in the same partition. |
|
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
|
|
(Optional) |
|
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
|
|
(Optional) |
|
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
|
|
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DeletionGracePeriodSeconds used to delete pipeline gracefully |
|
|
(Optional)
DesiredPhase used to bring the pipeline from current phase to desired phase |
|
|
(Optional)
PauseGracePeriodSeconds used to pause pipeline gracefully |
|
|
(Optional)
DeleteGracePeriodSeconds used to delete pipeline gracefully Deprecated: Use DeletionGracePeriodSeconds instead |
(Appears on: AbstractSink)
(Appears on: TagConditions)
(Appears on: AbstractPodTemplate)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
Field |
Description |
||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
|
||||||||||||||||||||||||||
|
|
|
||||||||||||||||||||||||||
|
|
(Optional) |
(Appears on: MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DesiredPhase used to bring the MonoVertex from current phase to desired phase |
(Appears on: MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size from the source. |
|
|
(Optional)
Read timeout duration from the source. |
(Appears on: MonoVertexLifecycle, MonoVertexStatus)
(Appears on: MonoVertex)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
|
|
|
|
|
|
(Members of (Optional) |
|
|
(Optional)
Container template for the main numa container. |
|
|
(Optional) |
|
|
(Optional)
Limits define the limitations such as read batch size for the mono vertex. |
|
|
(Optional)
Settings for autoscaling |
|
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
|
|
(Optional)
List of customized sidecar containers belonging to the pod. |
|
|
(Optional)
Template for the daemon service deployment. |
|
|
(Optional)
The strategy to use to replace existing pods with new ones. |
|
|
(Optional)
Lifecycle defines the Lifecycle properties of a MonoVertex |
(Appears on: MonoVertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional) |
|
|
(Optional)
Total number of non-terminated pods targeted by this MonoVertex (their labels match the selector). |
|
|
(Optional)
The number of desired replicas. |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Time of last scaling operation. |
|
|
(Optional)
The generation observed by the MonoVertex controller. |
|
|
(Optional)
The number of pods targeted by this MonoVertex with a Ready Condition. |
|
|
The number of Pods created by the controller from the MonoVertex version indicated by updateHash. |
|
|
The number of ready Pods created by the controller from the MonoVertex version indicated by updateHash. |
|
|
If not empty, indicates the current version of the MonoVertex used to generate Pods. |
|
|
If not empty, indicates the updated version of the MonoVertex used to generate Pods. |
(Appears on: RedisBufferService)
|
Field |
Description |
|---|---|
|
|
Redis version, such as “6.0.16” |
|
|
Redis StatefulSet size |
|
|
(Optional)
RedisContainerTemplate contains customized spec for Redis container |
|
|
(Optional)
SentinelContainerTemplate contains customized spec for Redis container |
|
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Members of (Optional) |
|
|
(Optional)
Redis configuration, if not specified, global settings in numaflow-controller-config will be used. |
(Appears on: JetStreamConfig, JetStreamSource, NatsSource)
NatsAuth defines how to authenticate the nats access
|
Field |
Description |
|---|---|
|
|
(Optional)
Basic auth which contains a username and a password |
|
(Optional)
Token auth |
|
|
(Optional)
NKey auth |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
|
|
Subject holds the name of the subject onto which messages are published. |
|
|
Queue is used for queue subscription. |
|
|
(Optional)
TLS configuration for the nats client. |
|
|
(Optional)
Auth information |
(Appears on: PBQStorage)
NoStore means there will be no persistence storage and there will be data loss during pod restarts. Use this option only if you do not care about correctness (e.g., approx statistics pipeline like sampling rate, etc.).
(Appears on: RetryStrategy)
(Appears on: GroupBy)
PBQStorage defines the persistence configuration for a vertex.
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
(Optional) |
|
|
|
(Optional) |
(Appears on: JetStreamBufferService, NativeRedis, PBQStorage)
PersistenceStrategy defines the strategy of persistence
|
Field |
Description |
|---|---|
|
|
(Optional)
Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 |
|
(Optional)
Available access modes such as ReadWriteOnce, ReadWriteMany https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes |
|
|
|
Volume size, e.g. 50Gi |
|
Field |
Description |
||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
|
||||||||||||||||||
|
|
|
||||||||||||||||||
|
|
(Optional) |
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size for all the vertices in the pipeline, can be overridden by the vertex’s limit settings. |
|
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. Only applies to UDF and Source vertices as only they do buffer write. It can be overridden by the settings in vertex limits. |
|
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. Only applies to UDF and Source vertices as only they do buffer write. It will be overridden by the settings in vertex limits. |
|
|
(Optional)
Read timeout for all the vertices in the pipeline, can be overridden by the vertex’s limit settings |
(Appears on: Lifecycle, PipelineStatus)
(Appears on: GetVertexPodSpecReq, Pipeline)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
|
|
|
Edges define the relationships between vertices |
|
|
(Optional)
Lifecycle define the Lifecycle properties |
|
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex’s settings |
|
|
(Optional)
Watermark enables watermark progression across the entire pipeline. |
|
|
(Optional)
Templates are used to customize additional kubernetes resources required for the Pipeline |
|
|
(Optional)
SideInputs defines the Side Inputs of a pipeline. |
|
|
(Optional)
ServingStore defines the Serving Store for this pipeline. |
(Appears on: Pipeline)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
The generation observed by the Pipeline controller. |
|
|
(Optional)
Field to indicate if a pipeline drain successfully occurred, only meaningful when the pipeline is paused. True means it has been successfully drained. |
(Appears on: Container, ContainerTemplate)
Probe is used to customize the configuration for Readiness and Liveness probes.
|
Field |
Description |
|---|---|
|
|
(Optional)
Number of seconds after the container has started before liveness probes are initiated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes |
|
|
(Optional)
Number of seconds after which the probe times out. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes |
|
|
(Optional)
How often (in seconds) to perform the probe. |
|
|
(Optional)
Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1. |
|
|
(Optional)
Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. Minimum value is 1. |
(Appears on: PulsarSource)
PulsarAuth defines how to authenticate with Pulsar
|
Field |
Description |
|---|---|
|
(Optional)
JWT Token auth |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Maximum number of messages that are in not yet acked state. Once this limit is crossed, futher read requests will return empty list. |
|
|
(Optional)
Auth information |
(Appears on: InterStepBufferServiceSpec)
|
Field |
Description |
|---|---|
|
|
Native brings up a native Redis service |
|
|
External holds an External Redis config |
(Appears on: BufferServiceConfig, RedisBufferService)
|
Field |
Description |
|---|---|
|
|
(Optional)
Redis URL |
|
|
(Optional)
Sentinel URL, will be ignored if Redis URL is provided |
|
|
(Optional)
Only required when Sentinel is used |
|
|
(Optional)
Redis user |
|
|
(Optional)
Redis password secret selector |
|
|
(Optional)
Sentinel password secret selector |
(Appears on: NativeRedis)
|
Field |
Description |
|---|---|
|
|
(Optional)
Redis settings shared by both master and slaves, will override the global settings from controller config |
|
|
(Optional)
Special settings for Redis master node, will override the global settings from controller config |
|
|
(Optional)
Special settings for Redis replica nodes, will override the global settings from controller config |
|
|
(Optional)
Sentinel settings, will override the global settings from controller config |
(Appears on: Sink)
RetryStrategy struct encapsulates the settings for retrying operations in the event of failures. It includes a BackOff strategy to manage the timing of retries and defines the action to take upon failure.
|
Field |
Description |
|---|---|
|
|
(Optional)
BackOff specifies the parameters for the backoff strategy, controlling how delays between retries should increase. |
|
|
(Optional)
OnFailure specifies the action to take when the specified retry strategy fails. The possible values are: 1. “retry”: start another round of retrying the operation, 2. “fallback”: re-route the operation to a fallback sink and 3. “drop”: drop the operation and perform no further action. The default action is to retry. |
(Appears on: UpdateStrategy)
RollingUpdateStrategy is used to communicate parameter for RollingUpdateStrategyType.
|
Field |
Description |
|---|---|
|
|
(Optional)
The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. Defaults to 25%. Example: when this is set to 30%, the old pods can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old pods can be scaled down further, followed by scaling up the new pods, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods. |
(Appears on: KafkaSink, KafkaSource)
|
Field |
Description |
|---|---|
|
|
SASL mechanism to use |
|
|
(Optional)
GSSAPI contains the kerberos config |
|
|
(Optional)
SASLPlain contains the sasl plain config |
|
|
(Optional)
SASLSCRAMSHA256 contains the sasl plain config |
|
|
(Optional)
SASLSCRAMSHA512 contains the sasl plain config |
|
|
(Optional)
OAuth contains the oauth config |
(Appears on: SASL)
|
Field |
Description |
|---|---|
|
|
ClientID refers to the secret that contains the client id |
|
|
ClientSecret refers to the secret that contains the client secret |
|
|
TokenEndpoint refers to the token endpoint |
(Appears on: SASL)
|
Field |
Description |
|---|---|
|
|
UserSecret refers to the secret that contains the user |
|
|
(Optional)
PasswordSecret refers to the secret that contains the password |
|
|
(Appears on: SASL)
SASLType describes the SASL type
(Appears on: AbstractVertex, MonoVertexSpec)
Scale defines the parameters for autoscaling.
|
Field |
Description |
|---|---|
|
|
(Optional)
Whether to disable autoscaling. Set to “true” when using Kubernetes HPA or any other 3rd party autoscaling strategies. |
|
|
(Optional)
Minimum replicas. |
|
|
(Optional)
Maximum replicas. |
|
|
(Optional)
Lookback seconds to calculate the average pending messages and processing rate. |
|
|
(Optional)
After scaling down the source vertex to 0, sleep how many seconds before scaling the source vertex back up to peek. |
|
|
(Optional)
TargetProcessingSeconds is used to tune the aggressiveness of autoscaling for source vertices, it measures how fast you want the vertex to process all the pending messages. Typically increasing the value, which leads to lower processing rate, thus less replicas. It’s only effective for source vertices. |
|
|
(Optional)
TargetBufferAvailability is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read. |
|
|
(Optional)
DeprecatedReplicasPerScale defines the number of maximum replicas that can be changed in a single scale up or down operation. The is use to prevent from too aggressive scaling operations Deprecated: Use ReplicasPerScaleUp and ReplicasPerScaleDown instead |
|
|
(Optional)
ScaleUpCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling up. It defaults to the CooldownSeconds if not set. |
|
|
(Optional)
ScaleDownCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling down. It defaults to the CooldownSeconds if not set. |
|
|
(Optional)
ReplicasPerScaleUp defines the number of maximum replicas that can be changed in a single scaled up operation. The is use to prevent from too aggressive scaling up operations |
|
|
(Optional)
ReplicasPerScaleDown defines the number of maximum replicas that can be changed in a single scaled down operation. The is use to prevent from too aggressive scaling down operations |
(Appears on: Source)
ServingSource is the HTTP endpoint for Numaflow.
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
Whether to create a ClusterIP Service |
|
|
The header key from which the message id will be extracted |
|
|
(Optional)
Request timeout in seconds. Default value is 120 seconds. |
(Appears on: PipelineSpec)
ServingStore defines information of a Serving Store used in a pipeline
|
Field |
Description |
|---|---|
|
|
|
|
|
(Appears on: Window)
SessionWindow describes a session window
|
Field |
Description |
|---|---|
|
|
Timeout is the duration of inactivity after which a session window closes. |
(Appears on: PipelineSpec)
SideInput defines information of a Side Input
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
(Optional) |
|
|
(Appears on: SideInput)
|
Field |
Description |
|---|---|
|
|
The schedule to trigger the retrievement of the side input data. It supports cron format, for example, “0 30 * * * *”. Or interval based format, such as “@hourly”, “@every 1h30m”, etc. |
|
|
(Optional) |
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Template for the side inputs manager numa container |
|
|
(Optional)
Template for the side inputs manager init container |
(Appears on: AbstractVertex, MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional)
Fallback sink can be imagined as DLQ for primary Sink. The writes to Fallback sink will only be initiated if the ud-sink response field sets it. |
|
|
(Optional)
RetryStrategy struct encapsulates the settings for retrying operations in the event of failures. |
(Appears on: Window)
SlidingWindow describes a sliding window
|
Field |
Description |
|---|---|
|
|
Length is the duration of the sliding window. |
|
Slide is the slide parameter that controls the frequency at which the sliding window is created. |
|
|
|
(Optional)
Streaming should be set to true if the reduce udf is streaming. |
(Appears on: AbstractVertex, MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
(Appears on: InterStepBufferServiceStatus, MonoVertexStatus, PipelineStatus, VertexStatus)
Status is a common structure which can be used for Status field.
|
Field |
Description |
|---|---|
|
|
(Optional)
Conditions are the latest available observations of a resource’s current state. |
(Appears on: JetStreamSource, KafkaSink, KafkaSource, NatsSource)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
CACertSecret refers to the secret that contains the CA cert |
|
|
(Optional)
CertSecret refers to the secret that contains the cert |
|
|
(Optional)
KeySecret refers to the secret that contains the key |
(Appears on: ForwardConditions)
|
Field |
Description |
|---|---|
|
|
(Optional)
Operator specifies the type of operation that should be used for conditional forwarding value could be “and”, “or”, “not” |
|
|
Values tag values for conditional forwarding |
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DaemonTemplate is used to customize the Daemon Deployment. |
|
|
(Optional)
JobTemplate is used to customize Jobs. |
|
|
(Optional)
SideInputsManagerTemplate is used to customize the Side Inputs Manager. |
|
|
(Optional)
VertexTemplate is used to customize the vertices of the pipeline. |
(Appears on: UDTransformer)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Optional) |
|
|
(Optional) |
(Appears on: AbstractVertex)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
(Appears on: AbstractSink)
|
Field |
Description |
|---|---|
|
|
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
(Appears on: AbstractVertex, MonoVertexSpec)
UpdateStrategy indicates the strategy that the controller will use to perform updates for Vertex or MonoVertex.
|
Field |
Description |
|---|---|
|
|
(Optional)
Type indicates the type of the StatefulSetUpdateStrategy. Default is RollingUpdate. |
|
|
(Optional)
RollingUpdate is used to communicate parameters when Type is RollingUpdateStrategy. |
(Appears on: UpdateStrategy)
UpdateStrategyType is a string enumeration type that enumerates all possible update strategies.
(Appears on: VertexInstance)
|
Field |
Description |
||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
|
||||||||||||||||
|
|
|
||||||||||||||||
|
|
(Optional) |
VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index.
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
(Appears on: VertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DesiredPhase used to bring the vertex from current phase to desired phase |
(Appears on: AbstractVertex, CombinedEdge)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size from the source or buffer. It overrides the settings from pipeline limits. |
|
|
(Optional)
Read timeout duration from the source or buffer It overrides the settings from pipeline limits. |
|
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. It overrides the settings from pipeline limits. |
|
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. It overrides the settings from pipeline limits. |
(Appears on: VertexLifecycle, VertexStatus)
(Appears on: Vertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Watermark indicates watermark progression in the vertex, it’s populated from the pipeline watermark settings. |
|
|
(Optional)
Lifecycle defines the Lifecycle properties of a vertex |
(Appears on: Vertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional) |
|
|
(Optional)
Total number of non-terminated pods targeted by this Vertex (their labels match the selector). |
|
|
(Optional)
The number of desired replicas. |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Time of last scaling operation. |
|
|
(Optional)
The generation observed by the Vertex controller. |
|
|
(Optional)
The number of pods targeted by this Vertex with a Ready Condition. |
|
|
The number of Pods created by the controller from the Vertex version indicated by updateHash. |
|
|
The number of ready Pods created by the controller from the Vertex version indicated by updateHash. |
|
|
If not empty, indicates the current version of the Vertex used to generate Pods. |
|
|
If not empty, indicates the updated version of the Vertex used to generate Pods. |
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Template for the vertex numa container |
|
|
(Optional)
Template for the vertex init container |
(Appears on: CombinedEdge)
(Appears on: PipelineSpec, VertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Disabled toggles the watermark propagation, defaults to false. |
|
|
(Optional)
Maximum delay allowed for watermark calculation, defaults to “0s”, which means no delay. |
|
|
(Optional)
IdleSource defines the idle watermark properties, it could be configured in case source is idling. |
(Appears on: GroupBy)
Window describes windowing strategy
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
Generated with gen-crd-api-reference-docs.