You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Create an ML.Net sample showing image classification using images already loaded in memory, rather than from disk #5980
I'm trying to put together an ML.Net image classification app. However, the samples I can find all assume that the images being classified are a folder of disk-based image files. This doesn't really help when processing already-in-memory images (for example, streaming video frames, etc).
I've tried adapting the example here to use bytes from a Bitmap pre-loaded from disk, but I don't seem to be getting the setup of the tensor data right, which means that I just get NaN results from the scoring.
It would be really useful if the ML.Net team could put together a sample where the input was a binary image - so for example, in the ScoreTensorFlowModel example above, if the GetTensorData method loaded a couple of JPEGs from disk, and then passed back the TensorData[] structure as an array of binary data from the images.
It seems, while googling, that a lot of people would find such an example extremely helpful - as there's a lot of people trying (and failing) to image-process video streams and other in-memory images using ML.Net and TF.
// See https://aka.ms/new-console-template for more informationusingMicrosoft.ML;usingSixLabors.ImageSharp;usingSixLabors.ImageSharp.PixelFormats;usingSixLabors.ImageSharp.Processing;usingSystem;usingSystem.IO;varmlContext=newMLContext();varimageData=DataLoadClass.LoadImageFromFile(mlContext,"ImageTrainData.tsv");varrows=mlContext.Data.CreateEnumerable<ModelInput>(imageData,reuseRowObject:false).Take(5).ToList();display(rows);
@JakeRadMSFT I'm having a similar problem, but in this case I need this for training.
The key difference is that when you do inferences, you usually pass one image at a time, which is fine to do it from memory.
But if you need to do training, things get more complicated, because you probably need thousands of input instances, and if each one has a bitmap, you can easily run out of memory just preparing the training data.
my training case scenarios are: taking frames from a video, and downloading images on demand from a private server.
As I propose in #6204 (close it if you feel it's redundant), I think what would be needed is some sort of callback to retrieve images on demand, or some interface that we would need to implement and pass to the loader, so the bitmaps would be loaded on demand from whatever source we need
Activity
yangpei1010110 commentedon Dec 20, 2021
agree, I need detect a Mat from opencvsharp model in real time, but I don't know how to do it.
michaelgsharp commentedon Jan 10, 2022
@JakeRadMSFT @luisquintanilla don't we already have this sample somewhere? Is it in one of your notebooks jake?
luisquintanilla commentedon Jan 10, 2022
We don't have an E2E sample that I'm aware of, but we do have the one in the reference docs
machinelearning/docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/MulticlassClassification/ImageClassification/ImageClassificationDefault.cs
Line 108 in 52ddbcd
@jwood803 is also working on a doc to provide guidance for image scenarios.
dotnet/docs#26537
JakeRadMSFT commentedon Jan 10, 2022
Yeah I have a notebook ... let me track it down.
JakeRadMSFT commentedon Jan 10, 2022
ImageClassificationBytesAndFile.txt
Attached is a Notebook (you can rename the extension to ipynb from txt) and then open it in VS with this extension:
https://marketplace.visualstudio.com/items?itemName=MLNET.notebook
The file should be compatible with any other IPYNB notebook editor (VS Code, Jupyter, etc ...)
I'll also include Code Snippets here ... let me know if you have any questions:
vpenades commentedon May 31, 2022
@JakeRadMSFT I'm having a similar problem, but in this case I need this for training.
The key difference is that when you do inferences, you usually pass one image at a time, which is fine to do it from memory.
But if you need to do training, things get more complicated, because you probably need thousands of input instances, and if each one has a bitmap, you can easily run out of memory just preparing the training data.
my training case scenarios are: taking frames from a video, and downloading images on demand from a private server.
As I propose in #6204 (close it if you feel it's redundant), I think what would be needed is some sort of callback to retrieve images on demand, or some interface that we would need to implement and pass to the loader, so the bitmaps would be loaded on demand from whatever source we need
michaelgsharp commentedon Jul 11, 2022
Closing as the original question has already been answered and the question about the callback has an answer in #6204