Label smoothing for SoftmaxOutput / cross-entropy loss

[fhieber]

Label smoothing is a nice technique to regularize models trained with cross-entropy error, for example described here, and applied for vision here, and for NLP/MT recently here.

This can be done in current MXNet when passed an explicit smoothed one-hot label distribution (see here) but this is probably wasteful w.r.t to memory if you have a large label space (e.g. large output vocabulary for LMs, translation models).
It also requires taking explicit care of padding symbols for variable-length models.

It would be nice to have such a flag in the SoftmaxOutput operator to do the smoothing internally without adding another node to the computation graph.

[zihaolucky]

follow the discussion.

[tqchen]

close as it is enabled