clamav_to_yara.py creates lots of invalid jumps e.g. [4-4]

[GoogleCodeExporter]

I've tried converting clamAV signature files to YARA but the converted yara 
file contain lots of invalid jumps like [4-4]. I've tried this on Windows 7 SP1 
x64 and Ubuntu 11.10 x64. The python version is 2.7. Yara version tried 1.5 and 
1.6. 

Original issue reported on code.google.com by [email protected] on 17 Nov 2011 at 8:41

[GoogleCodeExporter]

The problem is not that clamav_to_yara.py is introducing the invalid jumps but 
rather that the program fails to recognize several signature forms

In this case, the form is an anchored signature described in Creating 
signatures for ClamAV http://www.clamav.net/doc/latest/signatures.pdf (p8) as 

HEXSIG[x-y]aa or aa[x-y]HEXSIG 
Match aa anchored to a hex-signature, see https://wwws.clamav.net/ 
bugzilla/show_bug.cgi?id=776for discussion and examples.

The script has no transform for the [x-y] constuct that appears in quite a few 
clamav signatures and simply ignores this, allowing the [x-x] cases through to 
the yara output where they cause an error.
My solution is to add a transform for this case.  I am attaching my modified 
version.  The patch is immediately after the initialization of the rule list.  
It must precede the introduction of any yara jumps as they will trigger the 
transform involved.  This should be harmless, but why take a chance.  Note: I 
am not a python programmer and there may be better or other ways to do this.

In addition, there are several other cases the program does not handle:

rules starting with 5? run afoul of a yara "no initial wildcard" rule.  My 
workaround is to turn the rule into 16 alternatives starting with 50 ... 5F.  
The yara (50|51|...|5F) form provokes the same error.  yara issue 34 addresses 
this.

Also, the script rejects as empty some rules in the W32_Virut_ia, etc. family.  
These rules start with a 

(hex | hex )

construct and are not empty.  I have not looked closely, but this is probably 
another transform omission. Unfortunately, the obvious translation of these 
rules will run afoul of the same problem noted above and fixing it is equally 
complicated at the clamav_to_yara level.

Original comment by [email protected] on 24 Jan 2012 at 11:28

Attachments:

• clamav_to_yara.py

[GoogleCodeExporter]

Thank you for the patches! I'll get it committed to the trunk shortly. 

Original comment by [email protected] on 2 Feb 2012 at 3:56

[GoogleCodeExporter]

No, they are dealing with the case of matching ranges.

They aren't matching the "range" {\d-}, though.  

The problem is on line 110, when processing a case of a range with a start 
value, but no end value.  They are substituting a phantom second value, where 
none exists.  The substitution statement fails, and the malformed range gets 
passed through untouched.

I patched the file to simply substitute the value "1" in, giving a range of 
"0-1".  This is incorrect, but I know nothing about yara or ClamAV.  The 
offending rule won't match anything, but it was broken to begin with, so my 
patch leads to at least an output that won't crash yara.

Please fix this, thanks.

Original comment by [email protected] on 22 Feb 2012 at 8:50

Attachments:

• clamav_to_yara.py

[GoogleCodeExporter]

Had less issues with these others versions of the script, but still had issues. 
 Side note - created ~21MB YARA rule file which took too long for YARA or 
volatility/malfind to process... split it up into 512 chunks and processed 
through an array - much better.

Original comment by [email protected] on 22 Mar 2012 at 12:50

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hey guys,

Fixed all of the issues I had (for now) with latest main.ndb from ClamAV.

Here is the patch:
- Fix invalid [4-4] ranges
- Fix 5?5?... + (8a... errors (through regexp construct ;))
- Fix silent error about this signature (using a hack):
EOL.0.94.2:0:*:This ClamAV version has reached End of Life! Please upgrade to 
version 0.95 or later. For more information see  www.clamav.net/eol-clamav-094 
and www.clamav.net/download:0:38

I just ignore it (should be converted to proper string though).
The hack is about detecting '/' presence.

Cheers,

Original comment by [email protected] on 29 Oct 2012 at 6:23

Attachments:

• fixes.patch