⚡ Bolt: Accelerate UTF-8 validation with simdutf8#23
⚡ Bolt: Accelerate UTF-8 validation with simdutf8#23google-labs-jules[bot] wants to merge 1 commit into
Conversation
This commit introduces a performance optimization to the UTF-8 validation logic in `arrow-data`. The `validate_utf8` function is now accelerated using the `simdutf8` crate. A fast-path check is performed on the entire byte buffer. If the buffer is valid UTF-8, a faster, unchecked string conversion is used for subsequent offset validation. If the `simdutf8` check fails, the implementation falls back to the standard `std::str::from_utf8` on a per-string basis. This ensures that detailed, informative error messages are preserved for invalid data, which is critical for downstream crates that rely on specific error content. Benchmark results on `validate_utf8_array_data` show a ~12.5% performance improvement for valid UTF-8 data. - `simdutf8` is added as an optional, default-on dependency. - The `validate_utf8` function is conditionally compiled based on the `simd` feature flag.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
This PR introduces a performance optimization to the UTF-8 validation logic in
arrow-databy leveraging thesimdutf8crate for a fast-path check. For valid UTF-8 data, this change results in a ~12.5% performance improvement on thevalidate_utf8_array_databenchmark. The implementation includes a fallback to the standard library's UTF-8 validation to ensure that detailed error messages are preserved in the case of invalid data.PR created automatically by Jules for task 11497423432925373325 started by @Dandandan