Convert HashVocab Values to upper case when parsing #291

Activity

1. Danko90

   changed the title [-]Term SSDEEP only in uppercase and Namespace CISCP not present[/-] [+]Term SSDEEP only in uppercase[/+] on Apr 26, 2017
2. gtback commented on Apr 26, 2017

   

   gtback

   on Apr 26, 2017

   Contributor

   More actions

   Thanks, @Danko90.

   The CybOX XML schemas define SSDEEP as being in upper-case only. If you're getting XML with the mixed case SSDeep, you should bring that up with the producer of the data.

   There might be something we do to work around this in python-cybox. For example, since all of the values in HashNameEnum-1.0 are upper-case, we can call upper() during parsing. I would not recommend adding a separate vocabulary value, since then "SSDeep" and "SSDEEP" will be treated as different values, which is probably not what you want.

   Can you paste the exact error you're getting (without adding the TERM_SSDEEP_LOWER = 'SSDeep' line)? That will help us track down the best way to solve this.
3. Danko90 commented on Apr 27, 2017

   

   Danko90

   on Apr 27, 2017

   Last edited by Danko90

   Author

   More actions

   Hi @gtback,
   Thanks for replying!
   Yes, below the traceback:

   During handling of the above exception, another exception occurred:
   Traceback (most recent call last):
     File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 156, in manage_stix
       stix_package = STIXPackage.from_xml(stix)
     File "/usr/local/lib/python3.4/dist-packages/stix/core/stix_package.py", line 249, in from_xml
       return entity_parser.parse_xml(xml_file, encoding=encoding)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/parser.py", line 187, in parse_xml
       entity = self.get_entity_class(xml_root_node.tag).from_obj(entity_obj)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 361, in from_obj
       val = transformer.from_obj(val)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 359, in from_obj
       val = [transformer.from_obj(x) for x in val]
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 359, in <listcomp>
       val = [transformer.from_obj(x) for x in val]
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 361, in from_obj
       val = transformer.from_obj(val)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 361, in from_obj
       val = transformer.from_obj(val)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 184, in from_obj
       return klass.from_obj(cls_obj)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 361, in from_obj
       val = transformer.from_obj(val)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 359, in from_obj
       val = [transformer.from_obj(x) for x in val]
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 359, in <listcomp>
       val = [transformer.from_obj(x) for x in val]
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 361, in from_obj
       val = transformer.from_obj(val)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 184, in from_obj
       return klass.from_obj(cls_obj)
     File "/usr/local/lib/python3.4/dist-packages/cybox/common/vocabs.py", line 142, in from_obj
       obj = super(VocabString, cls).from_obj(cls_obj)
     File "/usr/local/lib/python3.4/dist-packages/cybox/common/attribute_groups.py", line 109, in from_obj
       obj = super(PatternFieldGroup, cls).from_obj(cls_obj)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 363, in from_obj
       field.__set__(entity, val)
     File "/usr/local/lib/python3.4/dist-packages/mixbox/fields.py", line 213, in __set__
       self.preset_hook(instance, value)
     File "/usr/local/lib/python3.4/dist-packages/cybox/common/vocabs.py", line 29, in validate_value
       raise ValueError(error)
   ValueError: Value for vocab <class 'cybox.common.vocabs.HashName'> must be one of ('MD5', 'SHA1', 'SHA384', 'SHA256', 'SHA224', 'MD6', 'SSDEEP', 'SHA512'). Received 'SSDeep'
   
4. gtback commented on Apr 27, 2017

   

   gtback

   on Apr 27, 2017

   Contributor

   More actions

   Thanks @Danko90. The error you're getting is what I expected. I'll have to look around a bit to see the best place to fix it, since we don't want to blindly upper-case all vocabularies, just the HashName ones.
5. Danko90 commented on May 2, 2017

   

   Danko90

   on May 2, 2017

   Author

   More actions

   Thanks @gtback ! I'll be waiting for your updates!
6. gtback

   changed the title [-]Term SSDEEP only in uppercase[/-] [+]Convert HashVocab Values to upper case when parsing[/+] on May 2, 2017
7. gtback commented on May 2, 2017

   

   gtback

   on May 2, 2017

   Contributor

   More actions

   @Danko90, in addition to fixing this in python-cybox, I would encourage you to tell the producer of the data that they are producing invalid CybOX. The fix for it here is really just a workaround to "do the right thing" in a case where we're pretty sure what the producer meant to do.
8. Danko90 commented on May 3, 2017

   

   Danko90

   on May 3, 2017

   Author

   More actions

   @gtback, Thank you so much.
   I have already advised the producer and been waiting for them answer.
   I'll let you know, thanks again!