docs: refactor virtual column docs (#2074)

* docs: refactor virtual column docs

* fix broken link

* fix

* fix

---------

Co-authored-by: z <[email protected]>

Author: b41sh

docs/cn/sql-reference/10-sql-commands/00-ddl/07-virtual-column/alter-virtual-column.md

@@ -7,33 +7,49 @@ import EEFeature from '@site/src/components/EEFeature';
 
 <EEFeature featureName='VIRTUAL COLUMN'/>
 
-A virtual column is a construct formed by extracting nested fields within [Variant](/sql/sql-reference/data-types/variant) data and storing that data in separate storage files. Consider using virtual columns when you regularly query specific nested fields within Variant data to realize the following benefits:
+# Virtual Columns in Databend: Accelerating Queries on Semi-Structured Data
 
-- **Accelerated Query Processing**: Virtual columns streamline the querying process by eliminating the need to traverse the entire nested structure to locate the desired data. Direct data retrieval from virtual columns parallels the process of accessing regular columns, resulting in a significant acceleration of query execution.
+Virtual columns in Databend provide a powerful and automatic way to significantly accelerate queries on semi-structured data, particularly data stored in the [Variant](/sql/sql-reference/data-types/variant) data type. This feature dynamically optimizes data access, leading to faster query execution and reduced resource consumption.
 
-- **Reduced Memory Usage**: Variant data often includes numerous internal fields, and reading all of them can lead to substantial memory consumption. By transitioning to reading virtual columns, there is a notable reduction in memory usage, mitigating the risk of potential memory overflows.
+## Overview
+
+When working with nested data structures within `VARIANT` columns, accessing specific data points can be a performance bottleneck. Databend's virtual columns address this by automatically identifying and optimizing nested fields. Instead of repeatedly traversing the entire nested structure, virtual columns enable direct data retrieval, similar to accessing regular columns.
+
+Databend automatically detects nested fields within `VARIANT` columns during data ingestion. If a field meets a certain threshold for presence, it's materialized as a virtual column in the background, ensuring that data is readily available for optimized querying. This process is entirely automatic, requiring no manual configuration or intervention.
 
 ![Alt text](/img/sql/virtual-column.png)
 
-## Managing Virtual Columns
+## Performance Benefits
+
+*   **Significant Query Acceleration:** Virtual columns dramatically reduce query execution time by enabling direct access to nested fields. This eliminates the overhead of traversing complex JSON structures for each query.
+*   **Reduced Resource Consumption:** By materializing only the necessary nested fields, virtual columns minimize memory consumption during query processing. This leads to more efficient resource utilization and improved overall system performance.
+*   **Automatic Optimization:** Databend automatically identifies and materializes fields as virtual columns. The query optimizer then automatically rewrites queries to utilize these virtual columns when accessing data within the `VARIANT` column.
+*   **Transparent Operation:** The creation and management of virtual columns are entirely transparent to the user. Queries are automatically optimized without requiring any changes to the query syntax or data loading process. The query optimizer handles the rewriting of queries to leverage virtual columns.
+
+## How it Works
 
-Databend provides a variety of commands to manage virtual columns. For details, see [VIRTUAL COLUMN](/sql/sql-commands/ddl/virtual-column/).
+1.  **Data Ingestion:** When data containing `VARIANT` columns is ingested, Databend analyzes the structure of the JSON data.
+2.  **Field Presence Check:** Databend checks if a nested field meets a certain threshold for presence.
+3.  **Virtual Column Materialization:** If the field presence threshold is met, the system automatically materializes the field as a virtual column in the background.
+4.  **Query Optimization:** When a query accesses a nested field within a `VARIANT` column, the query optimizer automatically rewrites the query to use the corresponding virtual column for faster data retrieval.
+
+## Important Considerations
+
+*   **Overhead:** While virtual columns generally improve query performance, they do introduce some storage and maintenance overhead. Databend automatically balances the benefits of virtual columns against this overhead to ensure optimal performance.
+*   **Experimental Feature:** Virtual columns are currently an experimental feature. They are disabled by default. To enable virtual columns, you must set the `enable_experimental_virtual_column` setting to `1`:
+*   **Automatic Refresh:** Virtual columns will be refreshed automatically after inserting data. If you don't want to generate virtual column data automatically, you can set `enable_refresh_virtual_column_after_write` to `0` to disable the generation of virtual columns. Asynchronous refresh can be done by using the refresh virtual column command. For details, see [REFRESH VIRTUAL COLUMN](/sql/sql-commands/ddl/virtual-column/refresh-virtual-column).
+*   **Show Virtual columns:** You can view information about virtual columns through the [SHOW VIRTUAL COLUMNS](/sql/sql-commands/ddl/virtual-column/show-virtual-columns) command, and you can view information about virtual column metas through the [FUSE_VIRTUAL_COLUMN](/sql/sql-functions/system-functions/fuse_virtual_column) system function.
 
 ## Usage Examples
 
 This example demonstrates the practical use of virtual columns and their impact on query execution:
 
 ```sql
+SET enable_experimental_virtual_column=1;
+
 -- Create a table named 'test' with columns 'id' and 'val' of type Variant.
 CREATE TABLE test(id int, val variant);
 
--- Create virtual columns for specific elements in the 'val' column.
-CREATE VIRTUAL COLUMN (
-  val ['name'],                 -- Extract the 'name' field.
-  val ['tags'] [0],             -- Extract the first element in the 'tags' array.
-  val ['pricings'] [0] ['type'] -- Extract the 'type' field from the first pricing in the 'pricings' array.
-) FOR test;
-
 -- Insert sample records into the 'test' table with Variant data.
 INSERT INTO
   test
@@ -65,8 +81,7 @@ INSERT INTO test SELECT * FROM test;
 INSERT INTO test SELECT * FROM test;
 INSERT INTO test SELECT * FROM test;
 
--- Refresh the virtual columns
-REFRESH VIRTUAL COLUMN FOR test;
+-- Show the virtual columns
 
 -- Explain the query execution plan for selecting specific fields from the table.
 EXPLAIN
@@ -79,16 +94,16 @@ FROM
 
 -[ EXPLAIN ]-----------------------------------
 Exchange
-├── output columns: [test.val['name'] (#2), test.val['tags'][0] (#3), test.val['pricings'][0]['type'] (#4)]
+├── output columns: [test.val['name'] (#3), test.val['pricings'][0]['type'] (#5), test.val['tags'][0] (#8)]
 ├── exchange type: Merge
 └── TableScan
-    ├── table: default.book_db.test
-    ├── output columns: [val['name'] (#2), val['tags'][0] (#3), val['pricings'][0]['type'] (#4)]
+    ├── table: default.default.test
+    ├── output columns: [val['name'] (#3), val['pricings'][0]['type'] (#5), val['tags'][0] (#8)]
     ├── read rows: 160
-    ├── read size: 4.96 KiB
-    ├── partitions total: 16
-    ├── partitions scanned: 16
-    ├── pruning stats: [segments: <range pruning: 6 to 6>, blocks: <range pruning: 16 to 16>]
+    ├── read size: 1.69 KiB
+    ├── partitions total: 6
+    ├── partitions scanned: 6
+    ├── pruning stats: [segments: <range pruning: 6 to 6>, blocks: <range pruning: 6 to 6>]
     ├── push downs: [filters: [], limit: NONE]
     ├── virtual columns: [val['name'], val['pricings'][0]['type'], val['tags'][0]]
     └── estimated rows: 160.00
@@ -108,23 +123,28 @@ Exchange
     ├── table: default.book_db.test
     ├── output columns: [val['name'] (#2)]
     ├── read rows: 160
-    ├── read size: 1.70 KiB
+    ├── read size: < 1 KiB
     ├── partitions total: 16
     ├── partitions scanned: 16
     ├── pruning stats: [segments: <range pruning: 6 to 6>, blocks: <range pruning: 16 to 16>]
     ├── push downs: [filters: [], limit: NONE]
     ├── virtual columns: [val['name']]
     └── estimated rows: 160.00
 
--- Display all the virtual columns defined in the system.
-SHOW VIRTUAL COLUMNS;
-
-┌─────────────────────────────────────────────────────────────────────────────┐
-│ database │  table │                     virtual_columns                     │
-├──────────┼────────┼─────────────────────────────────────────────────────────┤
-│ default  │ test   │ val['name'], val['pricings'][0]['type'], val['tags'][0] │
-└─────────────────────────────────────────────────────────────────────────────┘
-
--- Drop the virtual columns associated with the 'test' table.
-DROP VIRTUAL COLUMN FOR test;
+-- Display all the auto generated virtual columns.
+SHOW VIRTUAL COLUMNS WHERE table='test';
+
+╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ database │  table │ source_column │ virtual_column_id │    virtual_column_name   │ virtual_column_type │
+│  String  │ String │     String    │       UInt32      │          String          │        String       │
+├──────────┼────────┼───────────────┼───────────────────┼──────────────────────────┼─────────────────────┤
+│ default  │ test   │ val           │        3000000000 │ ['id']                   │ UInt64              │
+│ default  │ test   │ val           │        3000000001 │ ['name']                 │ String              │
+│ default  │ test   │ val           │        3000000002 │ ['pricings'][0]['price'] │ String              │
+│ default  │ test   │ val           │        3000000003 │ ['pricings'][0]['type']  │ String              │
+│ default  │ test   │ val           │        3000000004 │ ['pricings'][1]['price'] │ String              │
+│ default  │ test   │ val           │        3000000005 │ ['pricings'][1]['type']  │ String              │
+│ default  │ test   │ val           │        3000000006 │ ['tags'][0]              │ String              │
+│ default  │ test   │ val           │        3000000007 │ ['tags'][1]              │ String              │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```

docs/cn/sql-reference/10-sql-commands/00-ddl/07-virtual-column/create-virtual-column.md

@@ -11,11 +11,15 @@ Contains information about the created virtual columns in the system.
 See also: [SHOW VIRTUAL COLUMNS](../../10-sql-commands/00-ddl/07-virtual-column/show-virtual-columns.md)
 
 ```sql
+SET enable_experimental_virtual_column=1;
+
 SELECT * FROM system.virtual_columns;
 
-┌───────────────────────────────────────────────────────────────────────────────────────────────┐
-│ database │  table │ virtual_columns │         created_on         │         updated_on         │
-├──────────┼────────┼─────────────────┼────────────────────────────┼────────────────────────────┤
-│ default  │ test   │ val['name']     │ 2023-12-25 21:24:26.127790 │ 2023-12-25 21:24:38.455268 │
-└───────────────────────────────────────────────────────────────────────────────────────────────┘
+╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ database │  table │ source_column │ virtual_column_id │ virtual_column_name │ virtual_column_type │
+│  String  │ String │     String    │       UInt32      │        String       │        String       │
+├──────────┼────────┼───────────────┼───────────────────┼─────────────────────┼─────────────────────┤
+│ default  │ test   │ val           │        3000000000 │ ['id']              │ UInt64              │
+│ default  │ test   │ val           │        3000000001 │ ['name']            │ String              │
+╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```