diff --git a/content/programming-guides/cbor.md b/content/programming-guides/cbor.md new file mode 100644 index 00000000..4171f1ac --- /dev/null +++ b/content/programming-guides/cbor.md @@ -0,0 +1,206 @@ ++++ +title = "ProtoCBOR Format" +weight = 62 +description = "Covers how to use the Protobuf to CBOR conversion utilities." +type = "docs" ++++ + +Protobuf supports a canonical encoding in CBOR, making it easier to share data +with systems that do not support the standard protobuf binary wire format and +have processing constraints that prefer more concise encoding. + +ProtoCBOR Format is not as efficient as protobuf wire format. +Like ProtoJSON, the converter uses more CPU to encode and decode messages and encoded messages usually consume more space. +CBOR is a schemaless type-length-value (TLV) encoding, so it incurs more interpretive overhead than the standard wire format. + +The encoding is described on a type-by-type basis in the table later in this topic. + +When parsing CBOR-encoded data into a protocol buffer, if a value is missing or if its value is `null`, it will be interpreted as the corresponding [default value](/programming-guides/editions#default). + +When generating CBOR-encoded output from a protocol buffer, if a protobuf field has the default value and if the field doesn't support field presence, it will be omitted from the output by default. +An implementation may provide options to include fields with default values in the output. + +ProtoCBOR does support field presence. +A message type field in any edition of protobuf supports field presence and if set will appear in the output. +Proto3 implicit-presence scalar fields will only appear in the CBOR output if they are not set to the default value for that type. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProtobufCBORCBOR example (diagnostic notation)Notes
messagemap{/fooBar/ 0: v, /g/ 1: null, ...}Generates CBOR maps. Message field numbers are represented as CBOR `int` and become CBOR map keys. + null is an accepted value for all field types and treated as the default value of the corresponding field type. +
enumint5The enum value as specified in proto is used and sent as a CBOR-encoded int. +
map<K,V>map{ * K => V}All keys are converted to their CBOR encodings.
repeated Vdefinite-length array[v, ...]null is accepted as the empty list [].
booltrue, falsetrue, false
stringstring"Hello World!"
bytesbstrh'deadc0de'CBOR value will be the data encoded as a byte string. +
int32, int64, uint32, uint64int1, -10, 0CBOR value will use its variable length encodings for integers, which account for the first 60 entries of the initial byte jump table. + Unsigned integers do not use the latter 32 entries of the initial 60 entries of the initial byte jump table.
fixed32tagged little endian1, 10, 0CBOR value is #6.70(bytes .size 4) encoded as little-endian.
fixed64tagged little endian1, 10, 0CBOR value is #6.71(bytes .size 8) encoded as little-endian.
sfixed32tagged little endian1, -10, 0CBOR value is #6.78(bytes .size 4) encoded as little-endian.
sfixed64tagged little endian1, -10, 0CBOR value is #6.79(bytes .size 8) encoded as little-endian.
float, doublefloat, double1.1, -10.0, 0, NaN, InfinityCBOR value will be the smallest lossless encoding of the floating point number, choosing between simple value, half-precision, single-precision, or double-precision. +
AnyGeneric value CBOR27([TypeURL, value])The URL of a binary `google.protobuf.Type` paired with the deserialized CBOR representation. + Check google.protobuf.Any for details.
Timestamp#6.0(string)0("1972-01-01T10:00:20.021Z")Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. + Offsets other than "Z" are also accepted. +
Duration#6.1002(map)1002({1: 17020, -9: 900000})A duration in RFC9581 format, limited to durations representable in `google.protobuf.Duration`.
Structmap{ int => any }Any CBOR map with integer keys and respectively constrained values. See struct.proto.
Wrapper typesvarious types2, "2", "foo", true, "true", null, 0, ...Wrappers use the same representation in CBOR as the wrapped primitive + type, except that null is allowed and preserved during data + conversion and transfer. +
FieldMaskstring"f.fooBar,h"See field_mask.proto. TODO
ListValuearray[foo, bar, ...]Definite-length array.
ValuevalueA CBOR value that is JSON-like, but structs have integer keys. Check + google.protobuf.Value + for details. +
NullValuenullCBOR null
Emptymap{}An empty CBOR map, but represented differently as an Any value.
+ +### CBOR Options {#json-options} + +A conformant protobuf CBOR implementation may provide the following options: + +* **Ignore unknown fields**: The protobuf CBOR parser should reject unknown + fields by default but may provide an option to ignore unknown fields in + parsing. + +* **Emit validation code**: For registered CBOR tags that have further + validation requirements, the protobuf CBOR message should have validation + methods emitted. + diff --git a/content/reference/protobuf/google.protobuf.md b/content/reference/protobuf/google.protobuf.md index 8fa3be75..28cba29a 100644 --- a/content/reference/protobuf/google.protobuf.md +++ b/content/reference/protobuf/google.protobuf.md @@ -119,6 +119,75 @@ custom JSON in addition to the `@type` field. Example (for message +#### CBOR {#cbor} + +The CBOR representation of an `Any` value uses the generic serialized value tag #6.27 with the type identifier and regular representation of the deserialized, embedded message. +A type identifier is either a URL to a binary representation of the protocol buffer type, or an integer representing a well-known type that doesn't have a different specialized encoding in CBOR. +If the value has a special representation with major type 6, or is a value type with presence, the value itself is used without the #6.27 wrapper. + +```proto +package google.profile; +message Person { + string first_name = 1; + string last_name = 2; +} +``` + +```cbor-diag +27(["type.googleapis.com/google.profile.Person", {1: , 2: }]) +``` + +The rules of the type URL are the same as ProtoJSON. + +The following table provides the type identifiers for other well-known types. + +| Identifier | Type | +-- +| 0x0 | Api | +| 0x1 | Empty | +| 0x2 | Enum | +| 0x3 | EnumValue | +| 0x4 | Field | +| 0x5 | Field.Cardinality | +| 0x6 | Field.Kind | +| 0x7 | FieldMask | +| 0x8 | Method | +| 0x9 | Mixin | +| 0xa | Option | +| 0xb | SourceContext | +| 0xc | Syntax | +| 0xd | Type | + +The Field.Cardinality and Field.Kind enumerations are assigned the following values and representations in the Any type. + +| Cardinality enum | Literal value | CBOR Any encoding | +| `CARDINALITY_UNKNOWN` | 0 | 0xd5_1a82_0500 | +| `CARDINALITY_OPTIONAL` | 1 | 0xd5_1a82_0501 | +| `CARDINALITY_REQUIRED` | 2 | 0xd5_1a82_0502 | +| `CARDINALITY_REPEATED` | 3 | 0xd5_1a82_0503 | + +| Kind enum | Literal value | CBOR Any encoding | +| `TYPE_UNKNOWN` | 0x00 | 0xd5_1a82_0600 | +| `TYPE_DOUBLE` | 0x01 | 0xd5_1a82_0601 | +| `TYPE_FLOAT` | 0x02 | 0xd5_1a82_0602 | +| `TYPE_INT64` | 0x03 | 0xd5_1a82_0603 | +| `TYPE_UINT64` | 0x04 | 0xd5_1a82_0604 | +| `TYPE_INT32` | 0x05 | 0xd5_1a82_0605 | +| `TYPE_UINT32` | 0x06 | 0xd5_1a82_0606 | +| `TYPE_FIXED64` | 0x08 | 0xd5_1a82_0608 | +| `TYPE_FIXED32` | 0x09 | 0xd5_1a82_0609 | +| `TYPE_BOOL` | 0x0a | 0xd5_1a82_060a | +| `TYPE_STRING` | 0x0b | 0xd5_1a82_060b | +| `TYPE_GROUP` | 0x0c | 0xd5_1a82_060c | +| `TYPE_MESSAGE` | 0x0d | 0xd5_1a82_060d | +| `TYPE_BYTES` | 0x0e | 0xd5_1a82_060e | +| `TYPE_UINT32` | 0x0f | 0xd5_1a82_060f | +| `TYPE_ENUM` | 0x10 | 0xd5_1a82_0610 | +| `TYPE_SFIXED32` | 0x11 | 0xd5_1a82_0611 | +| `TYPE_SFIXED64` | 0x12 | 0xd5_1a82_0612 | +| `TYPE_SINT32` | 0x13 | 0xd5_1a82_0613 | +| `TYPE_SINT64` | 0x14 | 0xd5_1a82_0614 | + ## Api {#api} Api is a light-weight descriptor for a protocol buffer service. @@ -127,6 +196,7 @@ Api is a light-weight descriptor for a protocol buffer service. Field name + CBOR map key Type Description @@ -134,6 +204,7 @@ Api is a light-weight descriptor for a protocol buffer service. name + 1 string The fully qualified name of this api, including package name followed by @@ -142,6 +213,7 @@ Api is a light-weight descriptor for a protocol buffer service. methods + 2 Method @@ -149,6 +221,7 @@ Api is a light-weight descriptor for a protocol buffer service. options + 3 Option @@ -156,6 +229,7 @@ Api is a light-weight descriptor for a protocol buffer service. version + 4 string

@@ -186,6 +260,7 @@ Api is a light-weight descriptor for a protocol buffer service. source_context + 5 mixins + 6 Mixin syntax + 7 Syntax @@ -225,6 +302,7 @@ Api is a light-weight descriptor for a protocol buffer service. Wrapper message for `bool`. The JSON representation for `BoolValue` is JSON `true` and `false`. +The CBOR representation for `BoolValue` is CBOR `true` and `false`. @@ -248,6 +326,7 @@ The JSON representation for `BoolValue` is JSON `true` and `false`. Wrapper message for `bytes`. The JSON representation for `BytesValue` is JSON string. +The CBOR representation for `BytesValue` is a CBOR byte string.
@@ -271,6 +350,7 @@ The JSON representation for `BytesValue` is JSON string. Wrapper message for `double`. The JSON representation for `DoubleValue` is JSON number. +The CBOR representation for `DoubleValue` is a CBOR major type 7 value.
@@ -373,6 +453,33 @@ expressed as fractional seconds.
+The CBOR representation for `Duration` is in the RFC9581 format of a #6.1002-tagged map limited to the representation capacity of a `google.protobuf.Duration`. +Note that the nanoseconds are `uint` and not `int`, so negative durations are represented with different second and nanosecond values. +A duration below -263s + 0.999999999s is illegal. +A duration above 263s - 0.999999999s is illegal. + + + + + + + + + + + + + + + + + + + + + +
KeyValue TypeDescription
&(seconds: 1)intTime in seconds.
&(nanoseconds: -9)uintFractional time in 1e-9s added to the given seconds. Must be between 0 and 999,999,999.
+ ## Empty {#empty} A generic empty message that you can re-use to avoid defining duplicated empty @@ -387,6 +494,8 @@ service Foo { The JSON representation for `Empty` is empty JSON object `{}`. +The CBOR representation for `Empty` is an empty CBOR map `{}`. + ## Enum {#enum} Enum type definition @@ -395,6 +504,7 @@ Enum type definition Field name + CBOR map key Type Description @@ -402,11 +512,13 @@ Enum type definition name + 1 string Enum type name. enumvalue + 2 EnumValue @@ -414,6 +526,7 @@ Enum type definition options + 3 Option @@ -421,6 +534,7 @@ Enum type definition source_context + 4 SourceContext @@ -428,11 +542,18 @@ Enum type definition syntax + 5 Syntax The source syntax. + + edition + 6 + >string + The syntax edition. + @@ -444,6 +565,7 @@ Enum value definition. Field name + CBOR map key Type Description @@ -451,16 +573,19 @@ Enum value definition. name + 1 string Enum value name. number + 2 int32 Enum value number. options + 3 Option @@ -822,6 +947,8 @@ Wrapper message for `float`. The JSON representation for `FloatValue` is JSON number. +The CBOR representation for `FloatValue` is a CBOR major type 7 value. + @@ -845,6 +972,8 @@ Wrapper message for `int32`. The JSON representation for `Int32Value` is JSON number. +The CBOR representation for `Int32Value` is a CBOR major type 0 or 1 value. +
@@ -868,6 +997,8 @@ Wrapper message for `int64`. The JSON representation for `Int64Value` is JSON string. +The CBOR representation for `Int32Value` is a CBOR major type 0 or 1 value. +
@@ -891,6 +1022,8 @@ The JSON representation for `Int64Value` is JSON string. The JSON representation for `ListValue` is JSON array. +The CBOR representation for `ListValue` is a CBOR definite-length array. +
@@ -918,6 +1051,7 @@ Method represents a method of an api. + @@ -925,31 +1059,37 @@ Method represents a method of an api. + + + + + + @@ -957,6 +1097,7 @@ Method represents a method of an api. + @@ -1056,6 +1197,7 @@ service Storage { + @@ -1063,11 +1205,13 @@ service Storage { + +
Field nameCBOR map key Type Description
name1 string The simple name of this method.
request_type_url2 string A URL of the input message type.
request_streaming3 bool If true, the request is streamed.
response_type_url4 string The URL of the output message type.
response_streaming5 bool If true, the response is streamed.
options6 Option
syntax7 Syntax
Field nameCBOR map key Type Description
name1 string The fully qualified name of the API which is included.
root2 string If non-empty specifies a path under which inherited HTTP paths are @@ -1084,6 +1228,8 @@ service Storage { The JSON representation for `NullValue` is JSON `null`. +The CBOR representation for `NullValue` is CBOR `null`. + @@ -1108,6 +1254,7 @@ enumeration, etc. + @@ -1115,6 +1262,7 @@ enumeration, etc. + + @@ -1159,12 +1308,16 @@ like the file in which it is defined.
Field nameCBOR map key Type Description
name1 string The option's name. For example, "java_package". @@ -1122,6 +1270,7 @@ enumeration, etc.
value2 Any
+The CBOR representation for `SourceContext` with `file_name` _f_ uses the Any representation, `27([0xb, `_f_`])`. + ## StringValue {#string-value} Wrapper message for `string`. The JSON representation for `StringValue` is JSON string. +The CBOR representation for `StringValue` is a CBOR UTF-8 string (major type 3). + @@ -1192,6 +1345,8 @@ together with the proto support for the language. The JSON representation for `Struct` is JSON object. +The CBOR representation for `Struct` is an int-keyed map. +
@@ -1389,6 +1544,8 @@ Wrapper message for `uint32`. The JSON representation for `UInt32Value` is JSON number. +The CBOR representation for `UInt32Value` is a major type 0 value. +
@@ -1412,6 +1569,8 @@ Wrapper message for `uint64`. The JSON representation for `UInt64Value` is JSON string. +The CBOR representation for `UInt64Value` is a major type 0 value. +
@@ -1438,6 +1597,14 @@ indicates an error. The JSON representation for `Value` is JSON value. +The CBOR representation for `Value` validates against the following CDDL. + +```cddl +value /= null / int / float / tstr / bool +value /= { * int => value } +value /= [ * value ] +``` +