m8ta
You are not authenticated, login.
text: sort by
tags: modified
type: chronology
{1528}
hide / / print
ref: -2015 tags: olshausen redwood autoencoder VAE MNIST faces variation date: 11-27-2020 03:04 gmt revision:0 [head]

Discovering hidden factors of variation in deep networks

  • Well, they are not really that deep ...
  • Use a VAE to encode both a supervised signal (class labels) as well as unsupervised latents.
  • Penalize a combination of the MSE of reconstruction, logits of the classification error, and a special cross-covariance term to decorrelate the supervised and unsupervised latent vectors.
  • Cross-covariance penalty:
  • Tested on
    • MNIST -- discovered style / rotation of the characters
    • Toronto faces database -- seven expressions, many individuals; extracted eigen-emotions sorta.
    • Multi-PIE --many faces, many viewpoints ; was able to vary camera pose and illumination with the unsupervised latents.

{1449}
hide / / print
ref: -0 tags: sparse coding reference list olshausen field date: 03-11-2019 21:59 gmt revision:3 [2] [1] [0] [head]

This was compiled from searching papers which referenced Olhausen and Field 1996 PMID-8637596 Emergence of simple-cell receptive field properties by learning a sparse code for natural images.

{1448}
hide / / print
ref: -2004 tags: Olshausen sparse coding review date: 03-08-2019 07:02 gmt revision:0 [head]

PMID-15321069 Sparse coding of sensory inputs

  • Classic review, Olshausen and Field. 15 years old now!
  • Note the sparsity here is in neuronal activation, not synaptic activity (though one should follow the other).
  • References Lewicki's auditory studies, Efficient coding of natural sounds 2002; properties of early auditory neurons are well suited for producing a sparse independent code.
    • Studies have found near binary encoding of stimuli in rat auditory cortex -- e.g. one spike per noise.
  • Suggests that overcomplete representations (e.g. where there are more 'second layer' neurons than inputs or pixels) are useful for flattening manifolds in the input space, making feature extraction easier.
    • But then you have an under-determined problem, where presumably sparsity metrics step in to restrict the actual coding space. Authors mention that this could lead to degeneracy.
    • Example is the early visual cortex, where axons to higher layers exceed those from the LGN by a factor of 25. Which, they say, may be a compromise between over-representation and degeneracy.
  • Sparse coding is a necessity from an energy standpoint -- only one in 50 neurons can be active at any given time.
  • Sparsity increases when classical receptive field stimuli in V1 is expanded with a real-world-statistics surround. (Gallant 2002).