Saturday, October 1, 2022

AI/ML Difference between Dense and Activation layer in Keras

I was wondering what was the difference between Activation Layer and Dense layer in Keras.

Since Activation Layer seems to be a fully connected layer, and Dense have a parameter to pass an activation function, what is the best practice ?

Let's imagine a fictionnal network like this : Input -> Dense -> Dropout -> Final Layer Final Layer should be : Dense(activation=softmax) or Activation(softmax) ? What is the cleanest and why ?

Using Dense(activation=softmax) is computationally equivalent to first add Dense and then add Activation(softmax). However there is one advantage of the second approach - you could retrieve the outputs of the last layer (before activation) out of such defined model. In the first approach - it's impossible.


References:

https://stackoverflow.com/questions/40866124/difference-between-dense-and-activation-layer-in-keras#:~:text=Using%20Dense(activation%3Dsoftmax),the%20first%20approach%20%2D%20it's%20impossible.



No comments:

Post a Comment