Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

NDArray API NN Optimizer (Multi-* update category) issues #15643

Description

@ChaiBapchya

Neural Network Optimizer updates such as
multi_mp_sgd_mom_update, multi_mp_sgd_update, multi_sgd_mom_update, multi_sgd_update

They are present in Symbol API doc but not in NDArray API doc.
However, upon checking definition of 1 of the operators

>>> help(mx.nd.multi_sgd_mom_update)
Help on function multi_sgd_mom_update:

multi_sgd_mom_update(*data, **kwargs)
    Momentum update function for Stochastic Gradient Descent (SGD) optimizer.
    
    Momentum update has better convergence rates on neural networks. Mathematically it looks
    like below:
    
    .. math::
    
      v_1 = \alpha * \nabla J(W_0)\\
      v_t = \gamma v_{t-1} - \alpha * \nabla J(W_{t-1})\\
      W_t = W_{t-1} + v_t
    
    It updates the weights using::
    
      v = momentum * v - learning_rate * gradient
      weight += v
    
    Where the parameter ``momentum`` is the decay rate of momentum estimates at each epoch.
    
    
    
    Defined in src/operator/optimizer_op.cc:L372
    
    Parameters
    ----------
    data : NDArray[]
        Weights, gradients and momentum
    lrs : tuple of <float>, required
        Learning rates.
    wds : tuple of <float>, required
        Weight decay augments the objective function with a regularization term that penalizes large weights. The penalty scales with the square of the magnitude of each weight.
    momentum : float, optional, default=0
        The decay rate of momentum estimates at each epoch.
    rescale_grad : float, optional, default=1
        Rescale gradient to grad = rescale_grad*grad.
    clip_gradient : float, optional, default=-1
        Clip gradient to the range of [-clip_gradient, clip_gradient] If clip_gradient <= 0, gradient clipping is turned off. grad = max(min(grad, clip_gradient), -clip_gradient).
    num_weights : int, optional, default='1'

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions