-
Notifications
You must be signed in to change notification settings - Fork 58
[BLOG]Introducing Enhanced JSON Capabilities in Valkey #265
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 6 commits
755ec53
3b88b8c
1942312
80f0d42
79d39cf
4348151
dc5802a
1948351
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
--- | ||
title: Roshan Khatri | ||
extra: | ||
photo: '/assets/media/authors/roshkhatri.jpeg' | ||
github: roshkhatri | ||
--- | ||
|
||
Roshan Khatri is a software engineer at AWS, passionate about building resilient, high-performance systems and simplifying infrastructure through automation. | ||
|
||
Outside of work, he enjoys travelling exploring new countries, diving into aviation tech, has deep love for aviation, and occasionally getting way too excited about benchmarking graphs. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,196 @@ | ||
+++ | ||
title = "Introducing Enhanced JSON Capabilities in Valkey" | ||
description = "Learn more about the new Valkey JSON module with some real-world examples and application." | ||
sort_by = "2025-06-03 01:01:01" | ||
authors= [ "roshkhatri"] | ||
+++ | ||
|
||
JSON is a ubiquitous universal format for semi-structured data, and modern developers expect to work with it natively across their stack, including in-memory stores like Valkey. Until now, using JSON in Valkey meant serializing entire objects into strings or flattening them into Hashes with limited nesting. These workarounds added complexity and made updates harder than they should be. | ||
|
||
That changes with the general availability of native JSON support in Valkey. You can now store, query, and update JSON documents directly, without manual parsing or transformation. This brings a cleaner model to working with semi-structured data and makes your code easier to write and maintain. | ||
|
||
In this post, I’ll walk through installing the Valkey JSON module and demonstrate how to use it for some common workloads. | ||
|
||
## Core Capabilities and Performance | ||
|
||
Valkey’s JSON support goes beyond basic storage and retrieval. It introduces powerful querying and filtering capabilities that streamline complex data operations. Tasks that once required extensive development time and multiple network requests can now be executed through a single JSON command, significantly increasing developer productivity and reducing operational costs. | ||
|
||
Valkey JSON supports six data types—**null, boolean, number, string, object, and array**—allowing developers to represent complex, nested data structures without the constraints of traditional string serialization. Unlike other composite types, JSON objects and arrays in Valkey can contain any combination of the six value types, enabling deeply nested and recursive data models to be stored natively. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can you give an example of how Valkey JSON can store something recursively? |
||
|
||
Internally, Valkey JSON utilizes an optimized **binary tree-like format**, which enables rapid traversal and manipulation of substructures without requiring full document parsing. This structure not only minimizes memory usage but also ensures that operations on specific paths remain efficient. Path-based commands like `JSON.GET`, `JSON.SET`, and `JSON.DEL` allow targeted interactions with specific elements, supporting multiple paths within a single operation. Additionally, Valkey JSON integrates with Valkey’s Access Control Lists (ACLs), introducing a `@json` command category to enforce granular permissions alongside existing data types. | ||
|
||
## Getting Started with Valkey JSON | ||
|
||
Let’s walk through a real-world example and see how simple, yet powerful Valkey JSON is to use. | ||
|
||
### Installation and Setup | ||
|
||
Valkey JSON is a module and at time of writing, you'll need to build the module locally. | ||
However, it is also available in the [valkey-extensions](https://hub.docker.com/r/valkey/valkey-extensions), a docker image with all the valkey modules built-in. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note: Might change ... valkey-extensions |
||
|
||
#### 1. Build Valkey JSON locally | ||
|
||
``` | ||
git clone https://github.com/valkey-io/valkey-json.git | ||
cd valkey-json | ||
./build.sh | ||
``` | ||
|
||
This will produce `libjson.so` inside the `bin/` or `build/` | ||
|
||
#### 2. Start Valkey with the JSON module | ||
|
||
``` | ||
valkey-server --loadmodule /path/to/libjson.so | ||
``` | ||
|
||
Let's connect to the valkey-server with `valkey-cli` client for the following commands. | ||
|
||
We will not verify the module is loaded: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems weird. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Now? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. haha |
||
|
||
``` | ||
127.0.0.1:6379> MODULE LIST | ||
1) 1) "name" | ||
2) "json" | ||
3) "ver" | ||
4) (integer) 10000 | ||
5) "path" | ||
6) "/usr/lib/valkey/libjson.so" | ||
7) "args" | ||
8) (empty array) | ||
``` | ||
|
||
You’re now ready to work with JSON in Valkey! | ||
|
||
### Basic JSON Operations | ||
|
||
Let’s take a real world example where you want to store a **list of users** in a single key like this: | ||
|
||
``` | ||
127.0.0.1:6379> JSON.SET users $ '[{"name":"Alice","email":"[email protected]","city":"Seattle","state":"WA"},{"name":"Bob","email":"[email protected]","city":"Bellevue","state":"WA"},{"name":"Charlie","email":"[email protected]","city":"Austin","state":"TX"}]' | ||
OK | ||
``` | ||
This command stores a JSON array of user objects at the root path `$` under the key `users`. You can visualize it like this: | ||
``` | ||
[ | ||
{ | ||
"name": "Alice", | ||
"email": "[email protected]", | ||
"city": "Seattle", | ||
"state": "WA" | ||
}, | ||
{ | ||
"name": "Bob", | ||
"email": "[email protected]", | ||
"city": "Bellevue", | ||
"state": "WA" | ||
}, | ||
{ | ||
"name": "Charlie", | ||
"email": "[email protected]", | ||
"city": "Austin", | ||
"state": "TX" | ||
} | ||
] | ||
``` | ||
|
||
Next, let’s say you want to fetch all the users in this list. | ||
roshkhatri marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
``` | ||
127.0.0.1:6379>JSON.GET users | ||
"[{\"name\":\"Alice\",\"email\":\"[email protected]\",\"city\":\"Seattle\",\"state\":\"WA\"}, | ||
{\"name\":\"Bob\",\"email\":\"[email protected]\",\"city\":\"Bellevue\",\"state\":\"WA\"}, | ||
{\"name\":\"Charlie\",\"email\":\"[email protected]\",\"city\":\"Austin\",\"state\":\"TX\"}]" | ||
``` | ||
|
||
Now we will retrieve only the cities associated with each user: | ||
|
||
``` | ||
127.0.0.1:6379>JSON.GET users $[*].city | ||
"[\"Seattle\",\"Bellevue\",\"Austin\"]" | ||
``` | ||
|
||
## Advanced JSON Queries | ||
|
||
Valkey JSON supports sophisticated [JSONPath](https://goessner.net/articles/JsonPath/) expressions | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. On this and the following lines you 'sophisticated' and 'powerful' - I think you tone down the use of adjectives and it will sound more authentic (here and generally) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. you only need to link JSONPath once, btw |
||
Let us look at some powerful filtering using [JSONPath](https://goessner.net/articles/JsonPath/) expressions for in database querying. | ||
|
||
#### Retrieve all users from Washington State | ||
|
||
``` | ||
127.0.0.1:6379>JSON.GET users '$[?(@.state=="WA")]' | ||
"[{\"name\":\"Alice\",\"email\":\"[email protected]\",\"city\":\"Seattle\",\"state\":\"WA\"}, | ||
{\"name\":\"Bob\",\"email\":\"[email protected]\",\"city\":\"Bellevue\",\"state\":\"WA\"}]" | ||
``` | ||
|
||
This helps you find all users from the `state` of Washington - `"WA"`. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems repetitive, maybe explain the syntax that filters instead of restating the heading. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. may be something like this:
|
||
|
||
### Multi-Path Queries | ||
|
||
Below we retrieve the names of users from WA and those from other states: | ||
|
||
``` | ||
127.0.0.1:6379> JSON.GET users '$[?(@.state=="WA")].name' | ||
"[\"Charlie\"]" | ||
127.0.0.1:6379> JSON.GET users '$[?(@.state!="WA")].name' | ||
"[\"Alice\",\"Bob\"]" | ||
``` | ||
|
||
This would return an array of names matching the condition | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. repetitive |
||
|
||
#### We can also get the emails of users from Seattle or Austin | ||
|
||
``` | ||
127.0.0.1:6379> JSON.GET users '$[?(@.city=="Seattle" || @.city=="Austin")].email' | ||
"[\"[email protected]\",\"[email protected]\"]" | ||
``` | ||
|
||
This makes querying and filtering JSON documents straightforward. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not needed and 'straightforward' is subjective. |
||
|
||
### Key Use Cases for with Valkey JSON | ||
|
||
Valkey JSON is both powerful and straightforward to implement, making it an ideal solution for a variety of scenarios. With native JSON support, developers can manage structured, nested data without complex serialization or parsing logic, enabling rapid, targeted updates and high-performance queries. Here are three impactful use cases that showcase how Valkey JSON can drive value: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 'Valkey JSON is both powerful and straightforward to implement, making it an ideal solution for a variety of scenarios.' <- not needed. just get rid of it. |
||
|
||
### Per-User Event Counters for Ad or Notification Delivery | ||
|
||
Valkey JSON works well in high-throughput systems that require tracking per-user counters for ad impressions, push notifications, or message deliveries. For example, an ad platform may store a JSON document per user with nested metadata for each campaign — including impression counts, last delivery timestamps, and click history. Instead of serializing and deserializing large blobs, Valkey JSON enables in-place updates using `JSON.NUMINCRBY` or `JSON.SET` on specific paths (e.g., `$.ad_campaigns.ad_123.count`). This reduces network I/O and latency while ensuring atomicity. Microservices can also retrieve only the required subfields using JSONPath queries, like `$.ad_campaigns.ad_123.lastSeen`, allowing for efficient real-time decisioning. Compared to alternatives like managing multiple hash keys or plain strings, this approach is both cleaner and faster — making it well-suited for ad tech and notification delivery platforms operating at millions of ops/sec under tight latency constraints. | ||
|
||
|
||
### Shared Reference Metadata Store for Microservices | ||
|
||
Across games, e-commerce platforms, or internal developer tools, multiple microservices often need fast access to consistent, structured reference data. This can include things like product attributes, game character metadata, tax codes, or ID mappings — which are naturally stored as JSON documents. Valkey JSON provides an ideal solution for centralizing this reference data in-memory. Teams can store large JSON blobs (hundreds of KB or more) using `JSON.SET`, and services can retrieve targeted subfields using path expressions like `$.items[?(@.rarity=="epic")]` or `$.idToName["1234"]`. Updates happen in bulk during patch releases or deployment cycles, but reads are constant and latency-sensitive. By keeping this metadata in Valkey, services avoid making remote API calls or parsing local files, achieving very low lookup latency even under load. This pattern greatly simplifies infrastructure, improves cache coherency, and is especially powerful in cloud-native environments where rapid bootstrapping and shared context matter. | ||
|
||
|
||
### Identity Graph and Profile Storage at Scale | ||
|
||
For organization operating large-scale identity platforms — such as those in fintech, healthtech, or fraud detection — managing complex user or entity profiles is a core requirement. These profiles often include deeply nested data like names, contact info, document verification, scores, and historical activity. Valkey JSON allows each profile to be stored as a single JSON document and updated atomically as new data arrives, without needing to rewrite the entire object. Queries like `$.email`, `$.history[-1]`, or `$.risk.score` can be executed efficiently with sub-millisecond latency. This architecture supports hundreds of thousands of concurrent reads and writes per second and can scale to multi-terabyte datasets using a mix of memory and persistent storage. For workloads that demand both schema flexibility and ultra-low latency, Valkey JSON offers a compelling alternative to rigid relational databases or slower document stores. | ||
|
||
|
||
## Integration with the Valkey Ecosystem | ||
|
||
Ensuring smooth integration across data stores and libraries is vital for minimizing migration friction and maximizing developer productivity. Valkey JSON is fully compatible with **Valkey 8.0 and above**, making it accessible to users on the latest Valkey versions without requiring additional configuration. | ||
|
||
It is also compliant with **RFC7159 and ECMA-404**, adhering to widely accepted JSON standards. This alignment enables consistent JSON handling across Valkey modules and supports integration with **Valkey-Search**, and querying across JSON datasets for advanced analytics and data exploration. Additionally, Valkey JSON integrates with **Valkey’s Access Control Lists (ACLs)**, providing fine-grained permissions for JSON commands alongside other data types like strings and hashes. | ||
|
||
For users migrating from Redis, Valkey JSON is designed as a **drop-in replacement for RedisJSON v2**, maintaining API and RDB compatibility. This ensures a smooth transition for RedisJSON users while leveraging Valkey’s optimized performance and scalability. | ||
|
||
## Conclusion | ||
|
||
The introduction of the Valkey JSON module is a significant milestone for developers building modern, data-driven applications. By bringing native JSON support into Valkey’s high-performance in-memory engine, it bridges the gap between structured document modeling and real-time responsiveness. Whether you’re updating millions of ad counters per second, serving game metadata to distributed microservices, or powering identity search across terabytes of profiles, ValkeyJSON simplifies your architecture while boosting speed and scalability. With Valkey 8.0 and beyond, developers gain a powerful, intuitive, and production-grade toolset to manipulate JSON data at scale — all with the familiar simplicity and reliability of Valkey. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This feels sales-y again. I don't think this paragraph adds much, I would drop it or replace it with something that tell a more before/after value. |
||
|
||
Explore Valkey JSON in more depth and start building with it today! | ||
|
||
Please refer to the | ||
|
||
* [Valkey Official Documentation](https://valkey.io/topics/valkey-json/) to learn more. | ||
* [Valkey JSON GitHub Repository](https://github.com/valkey-io/valkey-json) if you have issues or for any feature requests. | ||
* [Valkey Extensions](https://hub.docker.com/r/valkey/valkey-extensions) to start build with Valkey JSON. | ||
|
||
Thank you to all those who helped develop the module: | ||
|
||
* Roshan Khatri ([roshkhatri](https://github.com/roshkhatri)) | ||
* Joe Hu ([joehu21](https://github.com/joehu21)) | ||
* Nikhil Manglore ([Nikhil-Manglore](https://github.com/Nikhil-Manglore)) | ||
* Cameron Zack ([zackcam](https://github.com/zackcam)) | ||
|
||
Happy coding |
roshkhatri marked this conversation as resolved.
Show resolved
Hide resolved
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Real-world is used serveral times in this blog post - consider changing some of these to alternate terms.