Skip to content

IDataView Type System needs to be extensible to support Image scenarios properly #3723

Closed

Description

@glebuk

The Problem

Currently it is impossible make prediction from images if they do not come from file. PredictionEngine<IT,OT> and IEnumerable to IDV is impossible if T contains images/

The Root Cause

The fundamental issue is that in several places in our system, our types are hardcoded and limited to a small set. Actual IDataView type system IS open, so it should be possible to support any type. However, some of our machinery is not so extensible. For example, if we need to add full support of images, we would need to update:

  • converters between IEnumerable <-> IDataView,
  • PredictionEngine<>
  • API Utils,
  • Schema utils
  • etc.

What we need is a dependency-injection mechanism to allow component developers to inject definitions of types supported by our IDV machinery. Such mechanism should be flexible and extensible. For an example of hard-coded implementation, check #3263. It is illustrative but incorrect as it is not flexible. Note @TomFinley's comment on different options on how to do this properly. We need to define a mechanism that is extensible, so that a dev can add new transforms to work on new data types, such as sound and enjoy all benefits of ML.NET

Definition of Done:

  • Mechanism to register additional IDV types so that they are supported by PredictionEngine and IEnumerable converters.
  • Add such registration for images and date types
  • Ensure that the registration (at least for images) occurs automatically whenever appropriate component is used (for example, if image resizing transform is used, the image type is registered automatically )
  • Ensure that prediction engine, conversion to and from IEnumerable work correctly with tests.

The following issues will also be solved by this issue:

#3369, #3460, #3582, #2121, #2495, #3582, #3274
In addition, this work is needed to properly support ONNX and TF scenarios for structured data such as image, speech, video, or Audio.

Activity

  1. CESARDELATORRE commented on May 15, 2019

    @CESARDELATORRE
    Contributor

    Agree. I've been looking for in-memory images support since #2121. I think this is "a must" feature (pre-requisite) in ML.NET (support for in-memory images) in order to be able to release the TensorFlow and ONNX packages as v1.0 (Currently 0.12).
    Note that this proposed implementation would be centralized (not specific within the TensorFlow or ONNX packages), though, so it'll need a new version for ML.NET itself, too. Minor version?

  2. self-assigned this
    on May 16, 2019
  3. wschin commented on May 16, 2019

    @wschin
    Contributor

    It should be a minor version because we won't break BC. Once I had a solution locally but got removed when doing git reset... Let me see if I can reconstruct it.

  4. ghost locked as resolved and limited conversation to collaborators on Mar 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions