So we need preciso compute the gradient of CE Loss respect each CNN class risultato mediante \(s\)

So we need preciso compute the gradient of CE Loss respect each CNN class risultato mediante \(s\)

Defined the loss, now we’ll have esatto compute its gradient respect to the output neurons of the CNN per order onesto backpropagate it through the net and optimize the defined loss function tuning the net parameters. The loss terms coming from the negative classes are niente. However, the loss gradient respect those negative classes is not cancelled, since the Softmax of the positive class also depends on the negative classes scores.

The gradient expression will be the same for all \(C\) except for the ground truth class \(C_p\), because the conteggio of \(C_p\) (\(s_p\)) is in the nominator.

  • Caffe: SoftmaxWithLoss Layer. Is limited esatto multi-class classification.
  • Pytorch: CrossEntropyLoss. Is limited preciso multi-class classification.
  • TensorFlow: softmax_cross_entropy. Is limited preciso multi-class classification.

In this Facebook work they claim that, despite being counter-intuitive, Categorical Cross-Entropy loss, or Softmax loss worked better than Binary Cross-Entropy loss mediante their multi-label classification problem.

> Skip this part if you are not interested con Facebook or me using Softmax Loss for multi-label classification, which is not norma.

When Softmax loss is used is verso multi-label ambiente, the gradients get a bit more complex, since the loss contains an element for each positive class. Consider \(M\) are the positive classes of verso sample. The CE Loss with Softmax activations would be:

Where each \(s_p\) mediante \(M\) is the CNN score for each positive class. As con Facebook paper, I introduce verso scaling factor \(1/M\) esatto make the loss invariant onesto the number of positive classes, which ple.

As Caffe Softmax with Loss layer nor Multinomial Logistic Loss Layer accept multi-label targets, I implemented my own PyCaffe Softmax loss layer, following the specifications of the Facebook paper. Caffe python layers let’s us easily customize the operations done mediante the forward and backward passes of the layer:

Forward pass: Loss computation

We first compute Softmax activations for each class and paravent them in probs. Then we compute the loss for each image con the batch considering there might be more than one positive label. We use an scale_factor (\(M\)) and we also multiply losses by the labels, which can be binary or real numbers, so they can be used for instance puro introduce class balancing. The batch loss will be the mean loss of the elements con the batch. We then save the data_loss sicuro display it and the probs esatto use them sopra the backward pass.

Backward pass: Gradients computation

In the backward pass we need sicuro compute the gradients of each element of the batch respect onesto each one of the classes scores \(s\). As the gradient for all the classes \(C\) except positive classes \(M\) is equal preciso probs, we assign probs values to sbocco. For the positive classes mediante \(M\) we subtract 1 onesto the corresponding probs value and use scale_factor puro match the gradient expression. We compute the mean gradients of all the batch puro run the backpropagation.

Binary Cross-Entropy Loss

Also called Sigmoid Ciclocampestre-Entropy loss. It is a Sigmoid activation plus a Ciclocampestre-Entropy loss. Unlike Softmax loss it is independent for each vector component (class), meaning that the loss computed for every CNN output vector component is not affected by other component values. That’s why it is used for multi-label classification, were the insight of an element belonging onesto per insecable class should not influence the decision for another class. It’s called Binary Cross-Entropy Loss because it sets up down dating collegamento verso binary classification problem between \(C’ = 2\) classes for every class per \(C\), as explained above. So when using this Loss, the formulation of Ciclocampestre Entroypy Loss for binary problems is often used:

Αφήστε μια απάντηση

Η ηλ. διεύθυνση σας δεν δημοσιεύεται. Τα υποχρεωτικά πεδία σημειώνονται με *