Skip to content

Make idref usage easier in python-stix #98

Closed

Description

@bworrell

Right now using idrefs in python-stix is somewhat clumsy. When working with related objects, it's often desirable to reference objects, rather than embedding (or inlining) them. However, doing so in python-stix often requires duplicate objects to be created when working with Related* classes:

Source: http://stixproject.github.io/idioms/indicator/c2-indicator/
 7 stix_package = STIXPackage()
 8 ttp = TTP(title="C2 Behavior")
 9    
10 indicator = Indicator(title="IP Address for known C2 Channel")
11 indicator.add_indicator_type("IP Watchlist")
12   
13 addr = Address(address_value="10.0.0.0", category=Address.CAT_IPV4)
14 indicator.add_observable(addr)
15 indicator.add_indicated_ttp(TTP(idref=ttp.id_))
16     
17 stix_package.add_indicator(indicator)
18 stix_package.add_ttp(ttp)

On line 15 we need to create a separate TTP instance and set its idref property to the id_ of the actual TTP definition. It is possible to do the following:

15 indicator.add_indicated_ttp(ttp)

However, this will embed (or inline) the TTP definition, rather than reference it.

Proposed Solution
Add a by_reference flag for methods that establish relationships...

15 indicator.add_indicated_ttp(ttp, by_reference=True)

While, theoretically, this makes life easier for developers, it raises some issues:

  1. If I iterate over the indicator.indicated_ttps, do I get back the fleshed out TTP definition or the skeleton TTP with the idref ?(indicator.indicated_ttps is actually a list of RelatedTTP instances..but you get what I'm saying)
  2. Does this automatically insert the TTP definition into the STIXPackage.ttps list? (I don't think it should as referenced TTP definitions may exist elsewhere)
  3. How do we handle the placement of the TTP definition (where does it actually go and at what point during runtime does it get there?)
  4. How do we ensure a well-formed, schema-valid document will be produced (avoiding id collisions/duplicate constructs)?

It may be best to still require developers to place the TTP definition where it needs to go rather than try to figure out where it should go automagically:

15 indicator.add_indicated_ttp(ttp, by_reference=True)
...
18 stix_package.add_ttp(ttp)

Activity

  1. gtback commented on Apr 4, 2014

    @gtback
    Contributor

    To respond to your points:

    1. This is a difficult problem. I tried to work on something similar in python-cybox with caches, and it works ok, but is a bit clumsy. When parsing XML, it depends entirely on the order of the parsing to know whether the "actual" item is in the cache when the "reference" is parsed. The "dereference" operation is a bit clumsy as well, and I haven't thought through it entirely yet. It's also possible to have circular references between items (if each is related to the other), which is another wrinkle in the parsing process.
    2. I would lean towards "no".
    3. For now, I think it makes the most sense for the author to explictly control where it goes, and only include references anywhere else. It's more than typical content authors should need to concern themselves with, but I can't think of a better way.
    4. I've been thinking about a global "validate" function that gets called before items are exported, which could check this and other things that the validator currently does, and some it doesn't (i.e. making sure required elements are there, since the bindings don't enforce them). This could potentially be turned off in the to_xml() function (to_xml(validate=False)). Just a thought.

    An alternative implementation (which I think I would prefer) is to add a "get_reference()" function to the core constructs, which could make the actual line:

    indicator.add_indicated_ttp(ttp.get_reference())

    You could also write a global utilty function that works on any object with id_ and idref properties:

    def make_ref(obj):
        new_obj = obj.__class__()
        new_obj.idref = obj.id_
        new_obj.id_ = None
        return new_obj
    
    indicator.add_indicated_ttp(make_ref(ttp))
  2. wbolster commented on Nov 28, 2014

    @wbolster

    Semantically there doesn't seem to be any difference between referencing an internal object and embedding it directly at that point in the tree, or am I mistaken?

  3. gtback commented on Dec 1, 2014

    @gtback
    Contributor

    @wbolster: In general, there is no semantic difference. There are a couple nuances, though, discussed here.

  4. bworrell commented on Apr 2, 2015

    @bworrell
    ContributorAuthor

    We have begun working on a Cacheable mixin class that handles this, but our initial commits only resolve idref objects. This really needs to handle the coupling of idref and timestamp, so you can easily resolve an object with a given id at a defined timestamp.

    Other considerations

    • Should we allow a user to pass in only an idref to resolve? What's the behavior when more than one object has that id with different timestamp values? Do we return a sequence of objects?
  5. bworrell commented on Jun 10, 2015

    @bworrell
    ContributorAuthor

    An approach to making this easier has been implemented in the mixbox branch of python-stix.

    Classes that can be referenced via idref have filter() and get() classmethods, which allow fast lookup of instances by id_ and other properties (timestamp) if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions