m8ta
You are not authenticated, login. |
|
{1544} | |||||
The HSIC Bottleneck: Deep learning without Back-propagation In this work, the authors use a kernelized estimate of statistical independence as part of a 'information bottleneck' to set per-layer objective functions for learning useful features in a deep network. They use the HSIC, or Hilbert-schmidt independence criterion, as the independence measure. The information bottleneck was proposed by Bailek (spikes..) et al in 1999, and aims to increase the mutual information between the layer representation and the labels while minimizing the mutual information between the representation and the input:
Where is the hidden representation at layer i (later output), is the layer input, and are the labels. By replacing with the HSIC, and some derivation (?), they show that
Where are samples and labels, and -- that is, it's the kernel function applied to all pairs of (vectoral) input variables. H is the centering matrix. The kernel is simply a Gaussian kernel, . So, if all the x and y are on average independent, then the inner-product will be mean zero, the kernel will be mean one, and after centering will lead to zero trace. If the inner product is large within the realm of the derivative of the kernel, then the HSIC will be large (and negative, i think). In practice they use three different widths for their kernel, and they also center the kernel matrices. But still, the feedback is an aggregate measure (the trace) of the product of two kernelized (a nonlinearity) outer-product spaces of similarities between inputs. it's not unimaginable that feedback networks could be doing something like this... For example, a neural network could calculate & communicate aspects of joint statistics to reward / penalize weights within a layer of a network, and this is parallelizable / per layer / adaptable to an unsupervised learning regime. Indeed, that was done almost exactly by this paper: Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks albeit in a much less intelligible way. Robust Learning with the Hilbert-Schmidt Independence Criterion Is another, later, paper using the HSIC. Their interpretation: "This loss-function encourages learning models where the distribution of the residuals between the label and the model prediction is statistically independent of the distribution of the instances themselves." Hence, given above nomenclature, (I'm not totally sure about the weighting, but might be required given the definition of the HSIC.) As I understand it, the HSIC loss is a kernellized loss between the input, output, and labels that encourages a degree of invariance to input ('covariate shift'). This is useful, but I'm unconvinced that making the layer output independent of the input is absolutely essential (??) | |||||
{1552} | |||||
Modularizing Deep Learning via Pairwise Learning With Kernels
I think in general this is an important result, even if its not wholly unique / somewhat anticipated (it's a year old at the time of writing). Modular training of neural networks is great for efficiency, parallelization, and biological implementations! Transport of weights between layers is hence non-essential. Classes still are, but I wonder if temporal continuity can solve some of these problems? (There is plenty of other effort in this area -- see also {1544}) | |||||
{1550} |
ref: -2011
tags: government polyicy observability submerged state America
date: 09-23-2021 22:06 gmt
revision:0
[head]
|
||||
The Submerged State -- How Invisible Government Policies Undermine American Democracy. By Suzanne Mettler (I've not read this book, just the blurb, but it looks like a defensible thesis) : Government polyicy, rather than distributing resources (money, infrastructure, services) as directly as possible to voters, have recently opted to distribute indirectly, through private companies. This gives the market & private organizations more perceived clout, perpetuates a level of corruption, and undermines American's faith in their government. So, we need a better 'debugger' for policy in america? Something like a discrete chain rule to help people figure out what policies (and who) are responsible for the good / bad things in their life? Sure seems that the bureaucracy is could use some cleanup / is failing under burgeoning complexity. This is probably not dissimilar to cruddy technical systems. | |||||
{1543} |
ref: -2019
tags: backprop neural networks deep learning coordinate descent alternating minimization
date: 07-21-2021 03:07 gmt
revision:1
[0] [head]
|
||||
Beyond Backprop: Online Alternating Minimization with Auxiliary Variables
This is interesting in that the weight updates can be cone in parallel - perhaps more efficient - but you are still propagating errors backward, albeit via optimizing 'codes'. Given the vast infractructure devoted to auto-diff + backprop, I can't see this being adopted broadly. That said, the idea of alternating minimization (which is used eg for EM clustering) is powerful, and this paper does describe (though I didn't read it) how there are guarantees on the convexity of the alternating minimization. Likewise, the authors show how to improve the performance of the online / minibatch algorithm by keeping around memory variables, in the form of covariance matrices. | |||||
{1487} |
ref: -0
tags: adaptive optics sensorless retina fluorescence imaging optimization zernicke polynomials
date: 11-15-2019 02:51 gmt
revision:0
[head]
|
||||
PMID-26819812 Wavefront sensorless adaptive optics fluorescence biomicroscope for in vivo retinal imaging in mice
| |||||
{1459} |
ref: -2018
tags: Michael Levin youtube talk NIPS 2018 regeneration bioelectricity organism patterning flatworm
date: 04-09-2019 18:50 gmt
revision:1
[0] [head]
|
||||
What Bodies Think About: Bioelectric Computation Outside the Nervous System - NeurIPS 2018
| |||||
{1445} |
ref: -2018
tags: cortex layer martinotti interneuron somatostatin S1 V1 morphology cell type morphological recovery patch seq
date: 03-06-2019 02:51 gmt
revision:3
[2] [1] [0] [head]
|
||||
| |||||
{1444} |
ref: -2012
tags: parvalbumin interneurons V1 perceptual discrimination mice
date: 03-06-2019 01:46 gmt
revision:0
[head]
|
||||
PMID-22878719 Activation of specific interneurons improves V1 feature selectivity and visual perception
| |||||
{1413} | |||||
PMID-24711417 Evidence for a causal inverse model in an avian cortico-basal ganglia circuit
| |||||
{1410} | |||||
Structure discovery in Nonparametric Regression through Compositional Kernel Search
| |||||
{1401} | |||||
PMID-27225074 Somatostatin-expressing neurons in cortical networks.
| |||||
{1195} | |||||
PMID-21270781[0] How advances in neural recording affect data analysis.
____References____
| |||||
{1198} | |||||
PMID-22049097[0] Mechanically adaptive intracortical implants improve the proximity of neuronal cell bodies.
____References____
| |||||
{288} | |||||
PMID-11240278[0] Functions of mammalian spinal interneurons during movement
____References____ | |||||
{984} | |||||
IEEE-6114258 (pdf) Towards a Brain-Machine-Brain Interface:Virtual Active Touch Using Randomly Patterned Intracortical Microstimulation.
____References____ O'Doherty, J. and Lebedev, M. and Li, Z. and Nicolelis, M. Towards a Brain #x2013;Machine #x2013;Brain Interface:Virtual Active Touch Using Randomly Patterned Intracortical Microstimulation Neural Systems and Rehabilitation Engineering, IEEE Transactions on PP 99 1 (2011) | |||||
{998} | |||||
The Coming War on General Computation "M.P.s and Congressmen and so on are elected to represent districts and people, not disciplines and issues. We don't have a Member of Parliament for biochemistry, and we don't have a Senator from the great state of urban planning, and we don't have an M.E.P. from child welfare. " | |||||
{300} | |||||
Motor learning by field approximation.
____References____ | |||||
{883} | |||||
Response to Jonah Lehrer's The Web and the Wisdom of Crowds: Lehrer is right on one thing: culture. We're all consuming similar things (e.g. Rebecca Black) via the strong positive feedback of sharing things that you like, liking things that you share, and becoming more like the things that are shared with you. Will this lead to a cultural convergence, or stable n-ary system? To early to tell, but probably not: likely this is nothing new. Would you expect music to collapse to a single genre? No way. Sure, there will be pop culture via the mechanisms Lehrer suggests, but meanwhile there is too much to explore, and we like novelty too much. Regarding decision making through stochastic averaging as implemented in democracy, I have to agree with John Hawk here. The growing availability of knowledge, news, and other opinions should be a good thing. This ought to be more than enough to counteract the problem of everyone reading say the NYTimes instead of many varied local newspapers; there should be no impoverishment of opinion. Furthermore, we read blogs (like Lehrer's) which have to compete increasingly honestly in the attention economy. The cost of redirecting our attention has gone from that of a subscription to free. Plus, this attention economy ties communication to reality at more points - each reader, as opposed to each publisher, is partially responsible for information amplification and dissemination. (I mean I just published this damn thing and almost zero cost - is that not a great thing?) | |||||
{818} | |||||
Say you have a program, called from a perl script, that may run for a long time. Get at the program's output as it appears? Simple - open a pipe to the programs STDOUT. See http://docstore.mik.ua/orelly/perl/prog3/ch16_03.htm Below is an example - I wanted to see the output of programs run, for convenience, from a perl script (didn't want to have to remember - or get wrong - all the command line arguments for each). #!/usr/bin/perl $numArgs = $#ARGV + 1; if($numArgs == 1){ if($ARGV[0] eq "table"){ open STATUS, "sudo ./video 0xc1e9 15 4600 4601 0 |"; while(<STATUS>){ print ; } close STATUS ; }elsif($ARGV[0] eq "arm"){ open STATUS, "sudo ./video 0x1ff6 60 4597 4594 4592 |"; while(<STATUS>){ print ; } close STATUS ; }else{ print "$ARGV[0] not understood - say arm or table!\n"; } } | |||||
{786} |
ref: -0
tags: linux keyboard international characters symbols
date: 10-01-2009 14:09 gmt
revision:1
[0] [head]
|
||||
Need to type international symbols and characters on your keyboard, e.g. for writing in another language? Do this: cp /usr/share/X11/locale/en_US.UTF-8/Compose ~/.XCompose xmodmap -e 'keycode 115 = Multi_key Multi_key Multi_key Multi_key' xmodmap -e 'keycode 116 = Multi_key Multi_key Multi_key Multi_key' Where 115 and 116 are the windows keys on my keyboard. (You can find this out for your keyboard by running 'xev'); Then:
yay! And now for something completely unrelated but highly amusing, at least in title: Optimal Brain Damage | |||||
{756} |
ref: life-0
tags: education wikinomics internet age college university pedagogy
date: 06-11-2009 12:52 gmt
revision:0
[head]
|
||||
Will universities stay relevant? and the rest of the wikinomics blog
| |||||
{752} | |||||
http://www.theatlantic.com/doc/200501/kirn -- goood.
| |||||
{724} | |||||
| |||||
{634} |
ref: RAzsa-2008.01
tags: nAChR nicotinic acetylchoine receptor interneurons backpropagating LTP hippocampus
date: 10-08-2008 17:37 gmt
revision:0
[head]
|
||||
PMID-18215234[0] Dendritic nicotinic receptors modulate backpropagating action potentials and long-term plasticity of interneurons.
____References____ | |||||
{590} | |||||
It is not obvious how to run an external command in ocaml & get it's output from stdin. Here is my hack, which simply polls the output of the program until there is nothing left to read. Not very highly tested, but I wanted to share, as I don't think there is an example of the same on pleac let run_command cmd = let inch = Unix.open_process_in cmd in let infd = Unix.descr_of_in_channel inch in let buf = String.create 20000 in let il = ref 1 in let offset = ref 0 in while !il > 0 do ( let inlen = Unix.read infd buf !offset (20000- !offset) in il := inlen ; offset := !offset + inlen; ) done; ignore(Unix.close_process_in inch); if !offset = 0 then "" else String.sub buf 0 !offset ;; Note: Fixed a nasty string-termination/memory-reuse bug Sept 10 2008 | |||||
{545} |
ref: notes-0
tags: telecommunications FCC wireless regulation government
date: 02-26-2008 04:18 gmt
revision:2
[1] [0] [head]
|
||||
http://news.zdnet.com/2010-1035_22-6231729.html
| |||||
{503} | |||||
quote: Consumers also pay high taxes for telecommunication services, averaging about 13 percent on some telecom services, similar to the tax rate on tobacco and alcohol, Mehlman said. One tax on telecom service has remained in place since the 1898 Spanish-American War, when few U.S. residents had telephones, he noted. "We think it's a mistake to treat telecom like a luxury and tax it like a sin," he said. from: The internet could run out of capacity in two years comments:
| |||||
{394} | |||||
so, you want to control the LEDs on a BF537-STAMP board? You'll need a linux box with a serial port, then will need to do a few things:
set serverip 192.168.1.149 set ipaddr 192.168.1.200 tftpboot 0x1000000 blink bootelf 0x1000000
| |||||
{479} | |||||
http://pespmc1.vub.ac.be/books/IntroCyb.pdf -- dated, but still interesting, useful, a book in and of itself!
| |||||
{476} | |||||
{465} | |||||
good explanation of 32-bit CRC (from the blackfin BF537 hardware ref): | |||||
{354} | |||||
____References____ | |||||
{352} |
ref: bookmark-0
tags: postmodernism pseudoscience Alan Sokal
date: 04-23-2007 03:47 gmt
revision:0
[head]
|
||||
http://www.physics.nyu.edu/faculty/sokal/pseudoscience_rev.pdf
| |||||
{148} |
ref: Vidal-2000.01
tags: EEG ERN error negativity conflict resolution 2000
date: 0-0-2007 0:0
revision:0
[head]
|
||||
PMID-10686362 Is the 'error negativity' specific to errors?
| |||||
{145} | |||||
PMID-12899266 Response error correction-a demonstration of improved human-machine performance using real-time EEG monitoring
| |||||
{44} | |||||
http://www.jneurosci.org/cgi/reprint/24/12/2989.pdf
|