Tanh is quite similar to the Y=X function in the vicinity of the origin. When the value of the activation function is low, the matrix operation can be directly performed which makes the training process relatively easier. Both tanh and sigmoid activation functions are fired which makes the neural network heavier.