Byte matching #96
Byte matching #96
Conversation
…ytes to compare with bytes_lit
.DS_Store
Outdated
There was a problem hiding this comment.
can you add all .DS_Store files to .gitignore?
Cargo.toml
Outdated
| @@ -9,7 +9,7 @@ members = [ | |||
| "examples/protocols", | |||
| "examples/basic", | |||
| "examples/basic_file", | |||
| "examples/log_ssh", | |||
| "examples/log_ssh", "examples/byte_match", | |||
There was a problem hiding this comment.
nit - can you add this to a newline for consistent formatting?
There was a problem hiding this comment.
Actually -- I think you can take out the example. It's important for functionality testing this feature, and you should save it in a branch in your repo, but I don't think we need to actually check it into the core repository.
One nice alternative would be to modify one of the other programs to add a subscription that looks for NULL bytes so that there is an example of the syntax somewhere. You could modify log_ssh to add an additional subscription matching on content or add something to filter_stats maybe.
There was a problem hiding this comment.
Sounds good! I modified one of the existing subscriptions in log_ssh to use byte matching to only log SSH handshakes whose protocol version field is equal to 2.0.
core/src/filter/ast.rs
Outdated
| @@ -837,6 +838,7 @@ impl fmt::Display for Value { | |||
| Value::Ipv4(net) => write!(f, "{}", net), | |||
| Value::Ipv6(net) => write!(f, "{}", net), | |||
| Value::Text(val) => write!(f, "{}", val), | |||
| Value::Byte(val) => write!(f, "{:?}", val), | |||
There was a problem hiding this comment.
Can you format this to print in a consistent format with the input? With the || and in hex.
For hex, might need an iter().for_each to format each byte individually in the Vec -- I don't remember if there's an easier way.
core/src/filter/parser.rs
Outdated
| let bytes_vec = bytes_as_str | ||
| .replace("|", "") | ||
| .split_whitespace() | ||
| .map(|s| u8::from_str_radix(s, 16).expect("Not a valid hex byte")) |
There was a problem hiding this comment.
Can you print in this error message the input that caused it? There could be a ton of subscriptions, so you should signal to the user which pattern caused the failure.
There was a problem hiding this comment.
Modified to include more info on what input caused the failure. As an example, an input like 2G should lead to a failure since it's not a valid hex byte. The failure message looks like:
There was a problem hiding this comment.
Edit the documentation for Field types in filtergen/lib.rs (will show up here) to add the raw byte vector type.
.gitignore
Outdated
| @@ -6,3 +6,4 @@ Cargo.lock | |||
| *.csv | |||
| *.json | |||
| *.jsonl | |||
| .DS_Store | |||
There was a problem hiding this comment.
@diana-qing I think this needs to be *.DS_Store so that it also catches files in subdirectories
|
one .gitignore comment, but this LGTM to merge otherwise. your functionality testing looks good. I ran the SSH example on the tap (without logging) and saw no performance red flags. |
This PR implements support for matching protocol fields on raw bytes. In a filter, bytes are enclosed in pipes (
|) and represented as hex bytes.