m8ta
You are not authenticated, login. |
|
{1545} | |||||
Self-organizaton in a perceptual network
One may critically challenge the infomax idea: we very much need to (and do) throw away spurious or irrelevant information in our sensory streams; what upper layers 'care about' when making decisions is certainly relevant to the lower layers. This credit-assignment is neatly solved by backprop, and there are a number 'biologically plausible' means of performing it, but both this and infomax are maybe avoiding the problem. What might the upper layers really care about? Likely 'care about' is an emergent property of the interacting local learning rules and network structure. Can you search directly in these domains, within biological limits, and motivated by statistical reality, to find unsupervised-learning networks? You'll still need a way to rank the networks, hence an objective 'care about' function. Sigh. Either way, I don't per se put a lot of weight in the infomax principle. It could be useful, but is only part of the story. Otherwise Linsker's discussion is accessible, lucid, and prescient. Lol. | |||||
{1543} |
ref: -2019
tags: backprop neural networks deep learning coordinate descent alternating minimization
date: 07-21-2021 03:07 gmt
revision:1
[0] [head]
|
||||
Beyond Backprop: Online Alternating Minimization with Auxiliary Variables
This is interesting in that the weight updates can be cone in parallel - perhaps more efficient - but you are still propagating errors backward, albeit via optimizing 'codes'. Given the vast infractructure devoted to auto-diff + backprop, I can't see this being adopted broadly. That said, the idea of alternating minimization (which is used eg for EM clustering) is powerful, and this paper does describe (though I didn't read it) how there are guarantees on the convexity of the alternating minimization. Likewise, the authors show how to improve the performance of the online / minibatch algorithm by keeping around memory variables, in the form of covariance matrices. | |||||
{1522} | |||||
Schema networks: zero-shot transfer with a generative causal model of intuitive physics
| |||||
{1517} | |||||
PMID-26621426 Causal Inference and Explaining Away in a Spiking Network
| |||||
{1516} | |||||
Inductive representation learning on large graphs
This is a well-put together paper, with some proofs of convergence etc -- but it still feels only lightly tested. As with many of these papers, could benefit from a positive control, where the generating function is known & you can see how well the algorithm discovers it. Otherwise, the structure / algorithm feels rather intuitive; surprising to me that it was not developed before the matrix factorization methods. Worth comparing this to word2vec embeddings, where local words are used to predict the current word & the resulting vector in the neck-down of the NN is the representation. | |||||
{1511} | |||||
| |||||
{1507} | |||||
| |||||
{1492} | |||||
PMID: Spiking neurons can discover predictive features by aggregate-label learning
Editorializing a bit: I said this was interesting, but why? The first part of the paper is another form of SGD, albeit in a spiking neural network, where the gradient is harder compute hence is done numerically. It's the aggregate part that is new -- pulling in repeated patterns through synaptic learning rules. Of course, to do this, the full trace of pre and post synaptic activity must be recorded (??) for estimating the STS (i think). An eligibility trace moves in the right direction as a biologically plausible approximation, but as always nothing matches the precision of SGD. Can the eligibility trace be amended with e.g. neuromodulators to push the performance near that of SGD? The next step of adding self supervised singular and multiple features is perhaps toward the way the brain organizes itself -- small local feedback loops. These features annotate repeated occurrences of stimuli, or tile a continuous feature space. Still, the fact that I haven't seen any follow-up work is suggestive... Editorializing further, there is a limited quantity of work that a single human can do. In this paper, it's a great deal of work, no doubt, and the author offers some good intuitions for the design decisions. Yet still, the total complexity that even a very determined individual can amass is limited, and likely far below the structural complexity of a mammalian brain. This implies that inference either must be distributed and compositional (the normal path of science), or the process of evaluating & constraining models must be significantly accelerated. This later option is appealing, as current progress in neuroscience seems highly technology limited -- old results become less meaningful when the next wave of measurement tools comes around, irrespective of how much work went into it. (Though: the impedtus for measuring a particular thing in biology is only discovered through these 'less meaningful' studies...). A third option, perhaps one which many theoretical neuroscientists believe in, is that there are some broader, physics-level organizing principles to the brain. Karl Friston's free energy principle is a good example of this. Perhaps at a meta level some organizing theory can be found, or likely a set of theories; but IMHO, you'll need at least one theory per brain area, at least, just the same as each area is morphologically, cytoarchitecturaly, and topologically distinct. (There may be only a few theories of the cortex, despite all the areas, which is why so many are eager to investigate it!) So what constitutes a theory? Well, you have to meaningfully describe what a brain region does. (Why is almost as important; how more important to the path there.) From a sensory standpoint: what information is stored? What processing gain is enacted? How does the stored information impress itself on behavior? From a motor standpoint: how are goals selected? How are the behavioral segments to attain them sequenced? Is the goal / behavior even a reasonable way of factoring the problem? Our dual problem, building the bridge from the other direction, is perhaps easier. Or it could be a lot more money has gone into it. Either way, much progress has been made in AI. One arm is deep function approximation / database compression for fast and organized indexing, aka deep learning. Many people are thinking about that; no need to add to the pile; anyway, as OpenAI has proven, the common solution to many problems is to simply throw more compute at it. A second is deep reinforcement learning, which is hideously sample and path inefficient, hence ripe for improvement. One side is motor: rather than indexing raw motor variables (LRUD in a video game, or joint torques with a robot..) you can index motor primitives, perhaps hierarchically built; likewise, for the sensory input, the model needs to infer structure about the world. This inference should decompose overwhelming sensory experience into navigable causes ... But how can we do this decomposition? The cortex is more than adept at it, but now we're at the original problem, one that the paper above purports to make a stab at. | |||||
{1418} |
ref: -0
tags: nanophotonics interferometry neural network mach zehnder interferometer optics
date: 06-13-2019 21:55 gmt
revision:3
[2] [1] [0] [head]
|
||||
Deep Learning with Coherent Nanophotonic Circuits
| |||||
{1463} | |||||
All-optical spiking neurosynaptic networks with self-learning capabilities
| |||||
{1446} | |||||
PMID-29074582 A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs
| |||||
{1434} | |||||
Audio AI: isolating vocals from stereo music using Convolutional Neural Networks
| |||||
{1426} | |||||
Training neural networks with local error signals
| |||||
{1419} | |||||
All-optical machine learning using diffractive deep neural networks
| |||||
{1174} | |||||
Brains, sex, and machine learning -- Hinton google tech talk.
| |||||
{1391} | |||||
Evolutionary Plasticity and Innovations in Complex Metabolic Reaction Networks
Summary thoughts: This is a highly interesting study, insofar that the authors show substantial support for their hypotheses that phenotypes can be explored through random-walk non-lethal mutations of the genotype, and this is somewhat invariant to the source of carbon for known biochemical reactions. What gives me pause is the use of linear programming / optimization when setting the relative concentrations of biomolecules, and the permissive criteria for accepting these networks; real life (I would imagine) is far more constrained. Relative and absolute concentrations matter. Still, the study does reflect some robustness. I suggest that a good control would be to ‘fuzz’ the list of available reactions based on statistical criteria, and see if the results still hold. Then, go back and make the reactions un-biological or less networked, and see if this destroys the measured degrees of robustness. | |||||
{1348} | |||||
Heller Lecture - Prof. David Kleinfeld
| |||||
{1269} |
ref: -0
tags: hinton convolutional deep networks image recognition 2012
date: 01-11-2014 20:14 gmt
revision:0
[head]
|
||||
{913} | |||||
PMID-21499255[0] Reversible large-scale modification of cortical networks during neuroprosthetic control.
Other notes:
____References____
| |||||
{1007} | |||||
IEEE-5910570 (pdf) Spiking neural network decoder for brain-machine interfaces
____References____ Dethier, J. and Gilja, V. and Nuyujukian, P. and Elassaad, S.A. and Shenoy, K.V. and Boahen, K. Neural Engineering (NER), 2011 5th International IEEE/EMBS Conference on 396 -399 (2011) | |||||
{993} | |||||
IEEE-1439548 (pdf) Interpreting spatial and temporal neural activity through a recurrent neural network brain-machine interface
____References____ Sanchez, J.C. and Erdogmus, D. and Nicolelis, M.A.L. and Wessberg, J. and Principe, J.C. Interpreting spatial and temporal neural activity through a recurrent neural network brain-machine interface Neural Systems and Rehabilitation Engineering, IEEE Transactions on 13 2 213 -219 (2005) | |||||
{968} |
ref: Bassett-2009.07
tags: Weinberger congnitive efficiency beta band neuroimagaing EEG task performance optimization network size effort
date: 12-28-2011 20:39 gmt
revision:1
[0] [head]
|
||||
PMID-19564605[0] Cognitive fitness of cost-efficient brain functional networks.
____References____
| |||||
{323} |
ref: Loewenstein-2006.1
tags: reinforcement learning operant conditioning neural networks theory
date: 12-07-2011 03:36 gmt
revision:4
[3] [2] [1] [0] [head]
|
||||
PMID-17008410[0] Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity
____References____ | |||||
{862} |
ref: -0
tags: backpropagation cascade correlation neural networks
date: 12-20-2010 06:28 gmt
revision:1
[0] [head]
|
||||
The Cascade-Correlation Learning Architecture
| |||||
{859} | |||||
Learning by Playing: Video Games in the Classroom
| |||||
{789} | |||||
I've been reading Computational Explorations in Cognitive Neuroscience, and decided to try the code that comes with / is associated with the book. This used to be called "PDP+", but was re-written, and is now called Emergent. It's a rather large program - links to Qt, GSL, Coin3D, Quarter, Open Dynamics Library, and others. The GUI itself seems obtuse and too heavy; it's not clear why they need to make this so customized / panneled / tabbed. Also, it depends on relatively recent versions of each of these libraries - which made the install on my Debian Lenny system a bit of a chore (kinda like windows). A really strange thing is that programs are stored in tree lists - woah - a natural folding editor built in! I've never seen a programming language that doesn't rely on simple text files. Not a bad idea, but still foreign to me. (But I guess programs are inherently hierarchal anyway.) Below, a screenshot of the whole program - note they use a Coin3D window to graph things / interact with the model. The colored boxes in each network layer indicate local activations, and they update as the network is trained. I don't mind this interface, but again it seems a bit too 'heavy' for things that are inherently 2D (like 2D network activations and the output plot). It's good for seeing hierarchies, though, like the network model. All in all looks like something that could be more easily accomplished with some python (or ocaml), where the language itself is used for customization, and not a GUI. With this approach, you spend more time learning about how networks work, and less time programming GUIs. On the other hand, if you use this program for teaching, the gui is essential for debugging your neural networks, or other people use it a lot, maybe then it is worth it ... In any case, the book is very good. I've learned about GeneRec, which uses different activation phases to compute local errors for the purposes of error-minimization, as well as the virtues of using both Hebbian and error-based learning (like GeneRec). Specifically, the authors show that error-based learning can be rather 'lazy', purely moving down the error gradient, whereas Hebbian learning can internalize some of the correlational structure of the input space. You can look at this internalization as 'weight constraint' which limits the space that error-based learning has to search. Cool idea! Inhibition also is a constraint - one which constrains the network to be sparse. To use his/their own words: ... given the explanation above about the network's poor generalization, it should be clear why both Hebbian learning and kWTA (k winner take all) inhibitory competition can improve generalization performance. At the most general level, they constitute additional biases that place important constraints on the learning and the development of representations. Mroe specifically, Hebbian learning constrains the weights to represent the correlational structure of the inputs to a given unit, producing systematic weight patterns (e.g. cleanly separated clusters of strong correlations). Inhibitory competition helps in two ways. First, it encourages individual units to specialize in representing a subset of items, thus parcelling up the task in a much cleaner and more systematic way than would occur in an otherwise unconstrained network. Second, inhibition greatly restricts the settling dynamics of the network, greatly constraining the number of states the network can settle into, and thus eliminating a large proportion of the attractors that can hijack generalization.." | |||||
{776} | |||||
http://www.willamette.edu/~gorr/classes/cs449/intro.html -- descent resource, good explanation of the equations associated with artificial neural networks. | |||||
{724} | |||||
| |||||
{695} | |||||
Alopex: A Correlation-Based Learning Algorithm for Feed-Forward and Recurrent Neural Networks (1994)
| |||||
{669} | |||||
PMID-19191602 A New Hypothesis for Sleep: Tuning for Criticality.
| |||||
{497} | |||||
http://dotpublic.istumbler.net/
| |||||
{7} |
ref: bookmark-0
tags: book information_theory machine_learning bayes probability neural_networks mackay
date: 0-0-2007 0:0
revision:0
[head]
|
||||
http://www.inference.phy.cam.ac.uk/mackay/itila/book.html -- free! (but i liked the book, so I bought it :) | |||||
{92} | |||||
with the extended kalman filter, from '92: http://ftp.ccs.neu.edu/pub/people/rjw/kalman-ijcnn-92.ps with the unscented kalman filter : http://hardm.ath.cx/pdf/NNTrainingwithUnscentedKalmanFilter.pdf | |||||
{40} |
ref: bookmark-0
tags: Bayes Baysian_networks probability probabalistic_networks Kalman ICA PCA HMM Dynamic_programming inference learning
date: 0-0-2006 0:0
revision:0
[head]
|
||||
http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html very, very good! many references, well explained too. | |||||
{39} | |||||
http://www.numenta.com/Numenta_HTM_Concepts.pdf
| |||||
{20} |
ref: bookmark-0
tags: neural_networks machine_learning matlab toolbox supervised_learning PCA perceptron SOM EM
date: 0-0-2006 0:0
revision:0
[head]
|
||||
http://www.ncrg.aston.ac.uk/netlab/index.php n.b. kinda old. (or does that just mean well established?) |