Skip to content

The hashed values stored in the annotations indices are skewed by -1 #3072

Closed

Description

@sfilipi

Look at the hash extension sample and compare the hashed values with the values stored in the annotations of the "CategoryHashed" column.

Notice how the indices in the annotations are skewed by -1 from the values in the dataview.

// Category CategoryHashed Age AgeHashed
// MLB 36206 18 127
// NFL 19015 14 62
// NFL 19015 15 43
// MLB 36206 18 127
// MLS 6013 14 62

versus the annotations values:

// Output Data
//
// The original value of the 6012 category is MLS
// The original value of the 19014 category is NFL
// The original value of the 36205 category is MLB

Activity

  1. TomFinley commented on Mar 22, 2019

    @TomFinley
    Contributor

    Working as intended I believe, though perhaps some of the infrastructure surrounding this could be improved. Remember: we have key-values presented as indices into an enumerated set (so 0 must be the first valid value), but we have simultaneously declared that the default value for keys (which must be 0), should represent the missing value, rather than any particular element.

    So 0 is and must be the "missing" key. The alternative is that it become some arbitrary member of whatever set is being enumerated, which is an utterly ridiculous proposition. We are also constrained by the fact that we are operating in an environment with 0 indexing. These two constraints force this compromise, which is, despite misunderstandings like this, the correct compromise.

    Perhaps we could repurpose this example and documentation so as to properly explain to people what keys are.

  2. TomFinley commented on Mar 22, 2019

    @TomFinley
    Contributor

    So... remove bug, maybe rephrase as documentation? The sample must at least be improved.

  3. sfilipi commented on Apr 4, 2019

    @sfilipi
    MemberAuthor

    @TomFinley adding a bit more to this, as i think this appearance of skewed keys and mapping 0 to the missing value might be harder to solve through documentation. As i was working on the samples for the KeyToValue, KeyToVector seeing the values loaded being skewed is confusing.. Would it be better to change the mapping of the missing value to be something like MaxInt? cc @Ivanidzo4ka

  4. added
    enhancementNew feature or request
    and removed
    bugSomething isn't working
    on Apr 4, 2019
  5. sfilipi commented on Apr 6, 2019

    @sfilipi
    MemberAuthor

    Closing after discussing alternatives like mapping the missing value to maxInt, or switching the rawType of KeyTypes to int?.
    Mapping to something other than the default value is not possible, since uninitialized variables will initialize to that.
    Trying to use Nullable uints showed a perf hit, during testing.

    For more on KeyTypes see: https://github.com/dotnet/machinelearning/blob/master/docs/code/IDataViewTypeSystem.md#key-types
    https://github.com/dotnet/machinelearning/blob/master/docs/code/KeyValues.md

  6. ghost locked as resolved and limited conversation to collaborators on Mar 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions