Training neural networks with local error signals
 Arild Nokland and Lars H Eidnes
 Idea is to use one+ supplementary neural networks to measure withinbatch matching loss between transformed hiddenlayer output and onehot label data to produce layerlocal learning signals (gradients) for improving local representation.
 Hence, no backprop. Error signals are all local, and interlayer dependencies are not explicitly accounted for (! I think).
 $L_{sim}$ : given a minibatch of hidden layer activations $H = (h_1, ..., h_n)$ and a onehot encoded label matrix $Y = (y_1, ..., y_n$ ,
 $L_{sim} =  S(NeuralNet(H))  S(Y)^2_F$ (don't know what F is..)
 $NeuralNet()$ is a convolutional neural net (trained how?) 3*3, stride 1, reduces output to 2.
 $S()$ is the cosine similarity matrix, or correlation matrix, of a minibatch.
 $L_{pred} = CrossEntropy(Y, W^T H)$ where W is a weight matrix, dim hidden_size * n_classes.
 Crossentropy is $H(Y, W^T H) = \Sigma_{i,j} Y_{i,j} log((W^T H)_{i,j}) + (1Y_{i,j}) log(1(W^T H)_{i,j})$
 Simbio loss: replace $NeuralNet()$ with averagepooling and standarddeviation op. Plus onehot target is replaced with a random transformation of the same target vector.
 Overall loss 99% $L_sim$ , 1% $L_pred$
 Despite the unequal weighting, both seem to improve test prediction on all examples.

 VGG like network, with dropout and cutout (blacking out square regions of input space), batch size 128.
 Tested on all the relevant datasets: MNIST, FashionMNIST, KuzushijiMNIST, CIFAR10, CIFAR100, STL10, SVHN.
 Pretty decent review of similarity matching measures at the beginning of the paper; not extensive but puts everything in context.
 See for example nonnegative matrix factorization using Hebbian and antiHebbian learning in and Chklovskii 2014.
 Emphasis put on biologically realistic learning, including the use of feedback alignment {1423}
 Yet: this was entirely supervised learning, as the labels were propagated back to each layer.
 More likely that biology is setup to maximize available labels (not a new concept).
