Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

[Vote] Softmax and Loss Convention #434

Description

@tqchen

According to discussions in #426
We agreed in all case, the multi-output and single output softmax can be combined into one-class, maybe overloaded by shape of label.

  • _Option 1_
    • SoftmaxOutput for output without gradient attached.
    • After attach a loss, XXXOutput will be able to backprop gradient of the loss, while forward behavior remains unchanged.
    • Softmax remains the same, with loss already attached.
    • May need to introduce attach_loss and maybe special loss class.
  • _Option 2_
    • SoftmaxOutput for output, with specific backward behavior of cross-entropy-loss
      • As an alternative task based naming instead of math based: MulticlassProbOutput to be clear to user.
    • Softmax behaves normally(only take input), and can prop gradient back from any output source(being able to compose as internal node)
    • CrossEntropyLoss can be composed with anything, including Softmax, to get loss in forward and gradient in backward.

Please edit this post to add more options .

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions