Bytes to bson

Author: aclark4life

bson/_rbson/README.md

@@ -183,20 +183,94 @@ When these missing features were added to achieve 100% compatibility, the true p
    - Edge case handling for all 88 tests
 3. **The Fundamental Issue**: Both implementations suffer from the same architectural limitation (Python → Bson enum → bytes), but it only becomes a significant bottleneck when you implement all the features required for production use.
 
+## Direct Byte-Writing Performance Results
+
+### Implementation: `_dict_to_bson_direct()`
+
+A new implementation has been added that writes BSON bytes directly from Python objects without converting to `Bson` enum types first. This eliminates the intermediate conversion layer.
+
+**Architecture Comparison:**
+```
+Regular:  Python objects → Rust Bson enum → BSON bytes
+Direct:   Python objects → BSON bytes (no intermediate types)
+```
+
+### Benchmark Results
+
+Comprehensive benchmarks on realistic document types show **consistent 2x speedup**:
+
+| Document Type | Regular (ops/sec) | Direct (ops/sec) | Speedup |
+|--------------|-------------------|------------------|---------|
+| User Profile | 99,970 | 208,658 | **2.09x** |
+| E-commerce Order | 93,578 | 165,636 | **1.77x** |
+| IoT Sensor Data | 136,824 | 312,058 | **2.28x** |
+| Blog Post | 65,782 | 134,154 | **2.04x** |
+
+**Average Speedup: 2.04x** (range: 1.77x - 2.28x)
+
+### Performance by Document Composition
+
+| Document Type | Regular (ops/sec) | Direct (ops/sec) | Speedup |
+|--------------|-------------------|------------------|---------|
+| Simple types (int, str, float, bool, None) | 177,588 | 800,670 | **4.51x** |
+| Mixed types | 223,856 | 342,305 | **1.53x** |
+| Nested documents | 130,884 | 287,758 | **2.20x** |
+| BSON-specific types only | 342,059 | 304,844 | 0.89x |
+
+### Key Findings
+
+1. **Massive speedup for simple types**: 4.51x faster for documents with Python native types
+2. **Consistent 2x improvement for real-world documents**: All realistic mixed-type documents show 1.77x - 2.28x speedup
+3. **Slight slowdown for pure BSON types**: Documents with only BSON-specific types (ObjectId, Binary, etc.) are 10% slower due to extra Python attribute lookups
+4. **100% correctness**: All outputs verified to be byte-identical to the regular implementation
+
+### Why Direct Byte-Writing is Faster
+
+1. **Eliminates heap allocations**: No need to create intermediate `Bson` enum values
+2. **Reduces function call overhead**: Writes bytes immediately instead of going through `python_to_bson()` → `write_bson_value()`
+3. **Better for common types**: Python's native types (int, str, float, bool) can be written directly without any conversion
+
+### Implementation Details
+
+The direct approach is implemented in these functions:
+- `_dict_to_bson_direct()` - Public API function
+- `write_document_bytes_direct()` - Writes document structure directly
+- `write_element_direct()` - Writes individual elements without Bson conversion
+- `write_bson_type_direct()` - Handles BSON-specific types directly
+
+### Usage
+
+```python
+from bson import _rbson
+from bson.codec_options import DEFAULT_CODEC_OPTIONS
+
+# Use direct byte-writing approach
+doc = {"name": "John", "age": 30, "score": 95.5}
+bson_bytes = _rbson._dict_to_bson_direct(doc, False, DEFAULT_CODEC_OPTIONS)
+```
+
+### Benchmarking
+
+Run the benchmarks yourself:
+```bash
+python benchmark_direct_bson.py        # Quick comparison
+python benchmark_bson_types.py         # Individual type analysis
+python benchmark_comprehensive.py      # Detailed statistics
+```
+
 ## Steps to Achieve Performance Parity with C Extensions
 
-Based on the analysis in PR #2695, here are the steps needed to match C extension performance:
+Based on the analysis in PR #2695 and the direct byte-writing results, here are the steps needed to match C extension performance:
 
-### 1. Eliminate Intermediate Bson Enum (High Impact)
+### 1. ✅ Eliminate Intermediate Bson Enum (High Impact) - COMPLETED
 **Current**: Python → Bson → bytes
 **Target**: Python → bytes (direct)
 
-Implement direct BSON byte writing from Python objects without converting to `Bson` enum first. This would require:
-- Custom serialization logic for each Python type
-- Manual BSON format handling (type bytes, length prefixes, etc.)
-- Bypassing the `bson` crate's serialization layer
+**Status**: ✅ **Implemented as `_dict_to_bson_direct()`**
+
+**Actual Impact**: **2.04x average speedup** on realistic documents (range: 1.77x - 2.28x)
 
-**Estimated Impact**: 3-4x performance improvement
+This brings the Rust extension from ~0.21x (5x slower than C) to **~0.43x (2.3x slower than C)** - a significant improvement!
 
 ### 2. Optimize Python API Calls (Medium Impact)
 - Reduce `getattr()` calls by caching attribute lookups
@@ -220,15 +294,14 @@ Implement direct BSON byte writing from Python objects without converting to `Bs
 
 **Estimated Impact**: 1.05-1.1x performance improvement
 
-### Combined Potential
-If all optimizations are implemented successfully:
-- Current: 0.21x (5x slower)
-- Target: 0.21x × 3.5 × 1.3 × 1.2 × 1.05 = **~1.13x** (13% faster than C)
+### Combined Potential (Updated with Direct Byte-Writing Results)
+With direct byte-writing implemented:
+- **Before**: 0.21x (5x slower than C)
+- **After direct byte-writing**: 0.43x (2.3x slower than C) ✅
+- **With all optimizations**: 0.43x × 1.3 × 1.2 × 1.05 = **~0.71x** (1.4x slower than C)
+- **Optimistic target**: Could potentially reach **~0.9x - 1.0x** (parity with C)
 
-However, achieving this would require:
-- Significant engineering effort (weeks to months)
-- Bypassing the `bson` crate (losing its benefits)
-- Complex low-level code (harder to maintain)
+The direct byte-writing approach has already delivered the largest performance gain (2x). Additional optimizations could close the remaining gap to C extension performance.
 
 ## Building