Scala: DataDesc IllegalArgumentException with simple example

[mariussoutier]

Description

Fitting using Scala and Module API throws an IAE.
Apparently the label shape of (50) doesn't correspond to expected the NCHW format.

Environment info (Required)

macOS 10.13.6
IntelliJ 2018.2.2
Scala 2.11.12
Java 1.8.0_121
MXNet 1.2.1

Error Message:

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: number of dimensions in shape :1 with shape: (50) should match the length of the layout: 4 with layout: NCHW
	at scala.Predef$.require(Predef.scala:224)
	at org.apache.mxnet.DataDesc.<init>(IO.scala:233)
	at org.apache.mxnet.DataDesc$$anonfun$ListMap2Descs$1.apply(IO.scala:256)
	at org.apache.mxnet.DataDesc$$anonfun$ListMap2Descs$1.apply(IO.scala:256)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at org.apache.mxnet.DataDesc$.ListMap2Descs(IO.scala:256)
	at org.apache.mxnet.module.BaseModule.fit(BaseModule.scala:399)

Minimum reproducible example

    val trainDataIter = IO.ImageRecordIter(Map(
      "data_name" -> dataName,
      "path_imgrec" -> this.getClass.getResource("/data/mydata.rec").getFile,
      "data_shape" -> "(3,128,128)",
      "batch_size" -> "50"
    ))

val mod = new Module(mlp)
    mod.fit(
      trainDataIter,
      Some(testDataIter),
      numEpoch = 10,
      fitParams =
        new FitParams()
          .setOptimizer(new SGD(0.1f, 0.9f, 0.0001f))
    )

Tried debugging this, but pretty difficult to find out what's going on with a stringly typed API.

println(trainDataIter.provideData) -> Map(data -> (50,3,128,128))
println(trainDataIter.provideLabel) -> Map(label -> (50))

[anirudhacharya]

@mxnet-label-bot [Scala]

[lanking520]

Hi @mariussoutier this problem actually addressed in 1.3 in this PR: #11844. Try with this one!

[mariussoutier]

@lanking520 Is there a published version of that I can use or do I have to build it myself?

[lanking520]

Unfortunately, you need to build from source with 1.3 or please wait for us to publish that. It will be out very soon.

[mariussoutier]

Ok I'll test again then. I feel like the Scala API could use some love :-)

[nswamy]

@mariussoutier we are working on a 1.3.0 pre-release, we should be able to have a RC maven package in staging soon.

[mariussoutier]

Nice, looking forward to it.

[nswamy]

@mariussoutier we released the package to Staging here if you would like to test it.
https://repository.apache.org/content/repositories/staging/org/apache/mxnet/

See the instructions on how to use packages in Staging repo here:
https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process#MXNet-ScalaReleaseProcess-UsingPackagesinStagingRepository

[mariussoutier]

@nswamy Just tried it out, Module API seems to work now. Thanks!

[mariussoutier]

Is there any difference between Module and FeedForward anyway?

[nswamy]

Feedforward as the name suggests works for FeedForward networks which are single input/single output but Module APIs will work for multi-input/multi-output FeedForward and CNNs, GANs, RNNs(probably requires the extended Bucketing Module which is in examples)

[mariussoutier]

Ok, makes sense. I've also used FeedForward for my CNN :-)

[mariussoutier]

One more issue, Module-based GPU training on Ubuntu fails during the second epoch with a SIGTERM. I don't know yet what's happening. FeedForward-based training in 1.2.1 works without a problem.

[nswamy]

could you please open a new issue with some reproducible code?