Skip to content

Create an ML.Net sample showing image classification using images already loaded in memory, rather than from disk #5980

Closed

Description

@Webreaper

I'm trying to put together an ML.Net image classification app. However, the samples I can find all assume that the images being classified are a folder of disk-based image files. This doesn't really help when processing already-in-memory images (for example, streaming video frames, etc).

I've tried adapting the example here to use bytes from a Bitmap pre-loaded from disk, but I don't seem to be getting the setup of the tensor data right, which means that I just get NaN results from the scoring.

It would be really useful if the ML.Net team could put together a sample where the input was a binary image - so for example, in the ScoreTensorFlowModel example above, if the GetTensorData method loaded a couple of JPEGs from disk, and then passed back the TensorData[] structure as an array of binary data from the images.

It seems, while googling, that a lot of people would find such an example extremely helpful - as there's a lot of people trying (and failing) to image-process video streams and other in-memory images using ML.Net and TF.

Thanks!

Activity

  1. yangpei1010110 commented on Dec 20, 2021

    @yangpei1010110

    agree, I need detect a Mat from opencvsharp model in real time, but I don't know how to do it.

  2. michaelgsharp commented on Jan 10, 2022

    @michaelgsharp
    Contributor

    @JakeRadMSFT @luisquintanilla don't we already have this sample somewhere? Is it in one of your notebooks jake?

  3. luisquintanilla commented on Jan 10, 2022

    @luisquintanilla
    Contributor

    We don't have an E2E sample that I'm aware of, but we do have the one in the reference docs

    @jwood803 is also working on a doc to provide guidance for image scenarios.

    dotnet/docs#26537

  4. JakeRadMSFT commented on Jan 10, 2022

    @JakeRadMSFT
    Member

    Yeah I have a notebook ... let me track it down.

  5. JakeRadMSFT commented on Jan 10, 2022

    @JakeRadMSFT
    Member

    ImageClassificationBytesAndFile.txt

    Attached is a Notebook (you can rename the extension to ipynb from txt) and then open it in VS with this extension:
    https://marketplace.visualstudio.com/items?itemName=MLNET.notebook

    The file should be compatible with any other IPYNB notebook editor (VS Code, Jupyter, etc ...)

    I'll also include Code Snippets here ... let me know if you have any questions:

    using Microsoft.ML.Data;
    
    public class ModelInput
    {
        [ColumnName(@"Label"), LoadColumn(0)]
        public string Label { get; set; }
    
        [ColumnName(@"ImageSource"), LoadColumn(1)]
        public string ImageSource { get; set; }
    
    }
    using Microsoft.ML.Data;
    
    public class ModelInputBytes
    {
        [ColumnName(@"Label")]
        public string Label { get; set; }
    
        [ColumnName(@"Features")]
        public byte[] ImageBytes { get; set; }
    
    }
    using Microsoft.ML.Data;
    
    public class ModelOutput
    {
        [ColumnName("PredictedLabel")]
        public string Prediction { get; set; }
    
        public float[] Score { get; set; }
    }
    using Microsoft.ML;
    
    public class ConsumeClass
    {
        public static ModelOutput Predict(ModelInput input, string MLNetModelPath)
        {
            MLContext mlContext = new MLContext();
    
            // Load model & create prediction engine
            ITransformer mlModel = mlContext.Model.Load(MLNetModelPath, out var modelInputSchema);
    
            ITransformer dataPreProcessTransform = LoadImageFromFileTransformer(input, mlContext);
    
            var predEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(dataPreProcessTransform.Append(mlModel));
    
            ModelOutput result = predEngine.Predict(input);
            return result;
        }
    
        public static ITransformer LoadImageFromFileTransformer(ModelInput input, MLContext mlContext)
        {
            var dataPreProcess = mlContext.Transforms.Conversion.MapValueToKey(@"Label", @"Label")
                                    .Append(mlContext.Transforms.LoadRawImageBytes(@"ImageSource_featurized","", @"ImageSource"))
                                    .Append(mlContext.Transforms.CopyColumns(@"Features", @"ImageSource_featurized"));
    
            var dataView = mlContext.Data.LoadFromEnumerable(new[] { input });
            var dataPreProcessTransform = dataPreProcess.Fit(dataView);
            return dataPreProcessTransform;
        }
    
        public static ModelOutput PredictFromBytes(ModelInputBytes input, string MLNetModelPath)
        {
            MLContext mlContext = new MLContext();
    
            // Load model & create prediction engine
            ITransformer mlModel = mlContext.Model.Load(MLNetModelPath, out var modelInputSchema);
    
            var predEngine = mlContext.Model.CreatePredictionEngine<ModelInputBytes, ModelOutput>(mlModel);
    
            ModelOutput result = predEngine.Predict(input);
            return result;
        }
    }
    using Microsoft.ML;
    
    public class DataLoadClass
    {
        public static IDataView LoadImageFromFile(MLContext mlContext, string trainDataPath)
        {
            IDataView trainData = mlContext.Data.LoadFromTextFile<ModelInput>(
            path: trainDataPath,
            hasHeader: true,
            separatorChar: '\t',
            allowQuoting: true,
            allowSparse: false);
    
            return trainData;
        }
    }
    using Microsoft.ML;
    
    public class TrainClass
    {
        public static ITransformer RetrainPipeline(MLContext context, IDataView trainData)
        {
    
    
            var dataPreProcess = context.Transforms.Conversion.MapValueToKey(@"Label", @"Label")
                                    .Append(context.Transforms.LoadRawImageBytes(@"ImageSource_featurized", "", @"ImageSource"))
                                    .Append(context.Transforms.CopyColumns(@"Features", @"ImageSource_featurized"));
    
            var trainingPipeline = context.MulticlassClassification.Trainers.ImageClassification(labelColumnName: @"Label")
                                    .Append(context.Transforms.Conversion.MapKeyToValue(@"PredictedLabel", @"PredictedLabel"));
    
            var processedData = dataPreProcess.Fit(trainData).Transform(trainData);
    
            var model = trainingPipeline.Fit(processedData);
    
            return model;
        }
    }
    // See https://aka.ms/new-console-template for more information
    
    using Microsoft.ML;
    using SixLabors.ImageSharp;
    using SixLabors.ImageSharp.PixelFormats;
    using SixLabors.ImageSharp.Processing;
    using System;
    using System.IO;
    
    var mlContext = new MLContext();
    var imageData = DataLoadClass.LoadImageFromFile(mlContext, "ImageTrainData.tsv");
    
    var rows = mlContext.Data.CreateEnumerable<ModelInput>(imageData, reuseRowObject: false)
                        .Take(5)
                        .ToList();
    
    display(rows);
    var pipeline = TrainClass.RetrainPipeline(mlContext, imageData);
    
    
    mlContext.Model.Save(pipeline, imageData.Schema, "model.mlnet");
    
    
    ModelInput modelInput = new ModelInput()
    {
        ImageSource = @"C:\dev\datasets\weather-small\Cloudy\cloudy1.jpg",
    };
    
    ModelOutput prediction = ConsumeClass.Predict(modelInput, "model.mlnet");
    
    Console.WriteLine("Prediction: " + prediction.Prediction);
    
    var image = Image.Load<SixLabors.ImageSharp.PixelFormats.Rgb24>(@"C:\dev\datasets\weather\Sunrise\sunrise103.jpg");
    image.Mutate(x => x.Resize(224, 224));
    byte[] imageBytes;
    
    using (MemoryStream memoryStream = new MemoryStream())
    {
        image.SaveAsJpeg(memoryStream);
        imageBytes = memoryStream.ToArray();
    }
    
    ModelInputBytes modelInputBytes = new ModelInputBytes()
    {
        ImageBytes = imageBytes,
    };
    
    var prediction2 = ConsumeClass.PredictFromBytes(modelInputBytes, "model.mlnet");
    Console.WriteLine("Prediction: " + prediction2.Prediction);
  6. vpenades commented on May 31, 2022

    @vpenades

    @JakeRadMSFT I'm having a similar problem, but in this case I need this for training.

    The key difference is that when you do inferences, you usually pass one image at a time, which is fine to do it from memory.

    But if you need to do training, things get more complicated, because you probably need thousands of input instances, and if each one has a bitmap, you can easily run out of memory just preparing the training data.

    my training case scenarios are: taking frames from a video, and downloading images on demand from a private server.

    As I propose in #6204 (close it if you feel it's redundant), I think what would be needed is some sort of callback to retrieve images on demand, or some interface that we would need to implement and pass to the loader, so the bitmaps would be loaded on demand from whatever source we need

  7. michaelgsharp commented on Jul 11, 2022

    @michaelgsharp
    Contributor

    Closing as the original question has already been answered and the question about the callback has an answer in #6204

  8. ghost locked as resolved and limited conversation to collaborators on Aug 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions