m8ta
You are not authenticated, login.
text: sort by
tags: modified
type: chronology
{1548}
hide / / print
ref: -2021 tags: gated multi layer perceptrons transformers ML Quoc_Le Google_Brain date: 08-05-2021 06:00 gmt revision:4 [3] [2] [1] [0] [head]

Pay attention to MLPs

  • Using bilinear / multiplicative gating + deep / wide networks, you can attain similar accuracies as Transformers on vision and masked language learning tasks! No attention needed, just a in-network multiplicative term.
  • And the math is quite straightforward. Per layer:
    • Z=σ(XU),,Z^=s(Z),,Y=Z^V Z = \sigma(X U) ,, \hat{Z} = s(Z) ,, Y = \hat{Z} V
      • Where X is the layer input, σ\sigma is the nonlinearity (GeLU), U is a weight matrix, Z^\hat{Z} is the spatially-gated Z, and V is another weight matrix.
    • s(Z)=Z 1(WZ 2+b) s(Z) = Z_1 \odot (W Z_2 + b)
      • Where Z is divided into two parts along the channel dimension, Z 1Z 2Z_1 Z_2 . 'circleDot' is element-wise multiplication, and W is a weight matrix.
  • You of course need a lot of compute; this paper has nice figures of model accuracy scaling vs. depth / number of parameters / size. I guess you can do this if you're Google.

Pretty remarkable that an industrial lab freely publishes results like this. I guess the ROI is that they get the resultant improved ideas? Or, perhaps, Google is in such a dominant position in terms of data and compute that even if they give away ideas and code, provided some of the resultant innovation returns to them, they win. The return includes trained people as well as ideas. Good for us, I guess!

{1495}
hide / / print
ref: -0 tags: multifactor synaptic learning rules date: 01-22-2020 01:45 gmt revision:9 [8] [7] [6] [5] [4] [3] [head]

Why multifactor?

  • Take a simple MLP. Let xx be the layer activation. X 0X^0 is the input, X 1X^1 is the second layer (first hidden layer). These are vectors, indexed like x i ax^a_i .
  • Then X 1=WX 0X^1 = W X^0 or x j 1=ϕ(Σ i=1 Nw ijx i 0)x^1_j = \phi(\Sigma_{i=1}^N w_{ij} x^0_i) . ϕ\phi is the nonlinear activation function (ReLU, sigmoid, etc.)
  • In standard STDP the learning rule follows Δwf(x pre(t),x post(t)) \Delta w \propto f(x_{pre}(t), x_{post}(t)) or if layer number is aa Δw a+1f(x a(t),x a+1(t))\Delta w^{a+1} \propto f(x^a(t), x^{a+1}(t))
    • (but of course nobody thinks there 'numbers' on the 'layers' of the brain -- this is just referring to pre and post synaptic).
  • In an artificial neural network, Δw aEw ij aδ j ax i \Delta w^a \propto - \frac{\partial E}{\partial w_{ij}^a} \propto - \delta_{j}^a x_{i} (Intuitively: the weight change is proportional to the error propagated from higher layers times the input activity) where δ j a=(Σ k=1 Nw jkδ k a+1)ϕ \delta_{j}^a = (\Sigma_{k=1}^{N} w_{jk} \delta_k^{a+1}) \partial \phi where ϕ\partial \phi is the derivative of the nonlinear activation function, evaluated at a given activation.
  • f(i,j)[x,y,θ,ϕ] f(i, j) \rightarrow [x, y, \theta, \phi]
  • k=13.165 k = 13.165
  • x=round(i/k) x = round(i / k)
  • y=round(j/k) y = round(j / k)
  • θ=a(ikx)+b(ikx) 2 \theta = a (\frac{i}{k} - x) + b (\frac{i}{k} - x)^2
  • ϕ=a(jky)+b(jky) 2 \phi = a (\frac{j}{k} - y) + b (\frac{j}{k} - y)^2

{1488}
hide / / print
ref: -0 tags: multimode fiber imaging date: 11-15-2019 03:10 gmt revision:2 [1] [0] [head]

PMID-30588295 Subcellular spatial resolution achieved for deep-brain imaging in vivo using a minimally invasive multimode fiber

  • Oh wow wowww
  • Imaged through a 50um multimode optical fiber!
  • Multimode scattering matrix was inverted through a LC-SLM

{1338}
hide / / print
ref: -0 tags: ZeroMQ messaging sockets multithreading date: 05-03-2016 06:10 gmt revision:0 [head]

ZeroMQ -- much better sockets framework than native TCP/UDP sockets.

  • Bindings for Ocaml, too.
  • Supports Erlang-like concurrency.

{1199}
hide / / print
ref: -0 tags: histology atryocytes immune response electrode arrays lund multiple exacerbate date: 01-24-2013 19:56 gmt revision:1 [0] [head]

PMID-23091629 Multiple implants do not aggravate the tissue reaction in rat brain.

  • After six weeks, the astrocytic scar surrounding the middle out of five implants was significantly smaller compared to the single contralateral implant, suggesting that an intrahemispheric interaction might be taking place, reducing the astrocytic response around the central implant.
  • Weak (?) staining for ED1 in this study?
  • -- after 6 weeks.
  • Thought: every paper has a different method for quantify immune response, GFAP staining in this case.

{979}
hide / / print
ref: Obeid-2004.02 tags: Obeid multichannel telemetry wireless recording date: 01-15-2012 22:06 gmt revision:3 [2] [1] [0] [head]

PMID-14757342[0] A multichannel telemetry system for single unit neural recordings

  • 16 channels; only transmit 12.
  • 45 minute battery life, 4W power consumption.
  • Uses a 486 index-card sized PC running DOS.
    • TCP/IP connection from host PC to wearable computer; UDP transmission of neural data.
  • 802.11b via a WAN ethernet card
  • 235g
  • AFE see [1]
  • 100mW radiated power.
  • Latency 680us input to output.
  • Did not notice any problems due to multipath.
  • See also PMID-17945926[2] for similar work

____References____

[0] Obeid I, Nicolelis MA, Wolf PD, A multichannel telemetry system for single unit neural recordings.J Neurosci Methods 133:1-2, 33-8 (2004 Feb 15)
[1] Obeid I, Nicolelis MA, Wolf PD, A low power multichannel analog front end for portable neural signal recordings.J Neurosci Methods 133:1-2, 27-32 (2004 Feb 15)
[2] Parthasarathy J, Hogenson J, Erdman AG, Redish AD, Ziaie B, Battery-operated high-bandwidth multi-channel wireless neural recording system using 802.11b.Conf Proc IEEE Eng Med Biol Soc 1no Issue 5989-92 (2006)

{996}
hide / / print
ref: Najafi-1986.12 tags: Najafi implantable wired recording Michigan array multiplexing silicon boron MEA date: 01-05-2012 03:07 gmt revision:8 [7] [6] [5] [4] [3] [2] [head]

IEEE-1052646 (pdf) An implantable multielectrode array with on-chip signal processing

  • "The major reason for the slow progress in the understanding of neural circuits has been the lack of adequate instrumentation."
  • previous photolithographic: [4],[5]. Their first publication: [7].
  • Kensall Wise, not Stephen.
  • Single shank
  • 10 recording sites spaced at 100um
  • Amplifying 100x, b/w 15kHz., multiplexing.
  • width: 15um near tip, 160um at base.
  • 3 leads (!) power, ground, data.
  • 6um LOCOS enhancement and depletion NMOS technology -- not CMOS. (latter is prone to latch-up)
  • 5mW power.
  • boron dope silicon, etch back non doped portion with ethylenediamine-pyrocatechol (EDP) water solution.
  • must not have any substrate bias!

____References____

Najafi, K. and Wise, K.D. An implantable multielectrode array with on-chip signal processing Solid-State Circuits, IEEE Journal of 21 6 1035 - 1044 (1986)

{744}
hide / / print
ref: Merletti-2009.02 tags: surface EMG multielectrode recording technology italy date: 01-03-2012 01:07 gmt revision:2 [1] [0] [head]

PMID-19042063[0] Technology and instrumentation for detection and conditioning of the surface electromyographic signal: state of the art

  • good background & review of surface EMG (sEMG) - noise levels, electrodes, electronics. eg. Instrumentation amplifiers with an input resistance < 100MOhm are not recommended, and the lower the input capacitance, the better: the impedance of a 10pf capacitor at 100hz is 160MOhm.
  • Low and balanced input impedances are required to reduce asymmetric filtering of common-mode power-line noise.

____References____

[0] Merletti R, Botter A, Troiano A, Merlo E, Minetto MA, Technology and instrumentation for detection and conditioning of the surface electromyographic signal: state of the art.Clin Biomech (Bristol, Avon) 24:2, 122-34 (2009 Feb)

{1002}
hide / / print
ref: Fan-2011.01 tags: TBSI wireless recordings system FM modulation multiplexing poland date: 01-03-2012 00:55 gmt revision:5 [4] [3] [2] [1] [0] [head]

PMID-21765934[0] A wireless multi-channel recording system for freely behaving mice and rats.

  • Light enough that rats can use it: 4.5g
  • 15 or 32 channels.
  • Good list of the competiton; they note Szuts et al [31], [1], {1003}, [2], {1004}, {1005}
  • Why are there so many authors?
  • Morizio and Henry Yin last authors.

____References____

[0] Fan D, Rich D, Holtzman T, Ruther P, Dalley JW, Lopez A, Rossi MA, Barter JW, Salas-Meza D, Herwik S, Holzhammer T, Morizio J, Yin HH, A wireless multi-channel recording system for freely behaving mice and rats.PLoS One 6:7, e22033 (2011)
[1] no Title no Source no Volume no Issue no Pages no PubDate
[2] Szuts TA, Fadeyev V, Kachiguine S, Sher A, Grivich MV, Agrochão M, Hottowy P, Dabrowski W, Lubenov EV, Siapas AG, Uchida N, Litke AM, Meister M, A wireless multi-channel neural amplifier for freely moving animals.Nat Neurosci 14:2, 263-9 (2011 Feb)

{857}
hide / / print
ref: -0 tags: nvidia linux driver vmalloc multiple cards grub date: 02-09-2011 13:49 gmt revision:2 [1] [0] [head]

http://www.nvnews.net/vbulletin/showthread.php?t=141845 -- when running multiple nvidia cards on one linux computer with a 32-bit kernel, you may run out of kernel memory while loading the video drivers. To fix this, pass vmaloc=256M to the kernel prior boot - e.g. by editing /boot/grub/menu.lst (grub 1) or /boot/grub/grub.cfg (grub 2)

If you want to make the change permanent with all kernels, edit

/etc/grub.d/10_linux

and add vmalloc=256M to the end of

linux   ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}

see also the Nvidia driver release notes

{410}
hide / / print
ref: notes-0 tags: multirate downsample DSP filter date: 08-09-2007 19:14 gmt revision:0 [head]

{400}
hide / / print
ref: bookmark-0 tags: phase converter gilbert cell analog multiplication RF bipolar transistors phase detector modulator date: 07-23-2007 20:48 gmt revision:0 [head]

http://www.electronics.dit.ie/staff/ypanarin/Lecture%20Notes/DT021-4/7AnalogMultipliers.pdf

{229}
hide / / print
ref: notes-0 tags: SNR MSE error multidimensional mutual information date: 03-08-2007 22:33 gmt revision:2 [1] [0] [head]

http://ieeexplore.ieee.org/iel5/516/3389/00116771.pdf or http://hardm.ath.cx:88/pdf/MultidimensionalSNR.pdf

  • the signal-to-noise ratio between two vectors is the ratio of the determinants of the correlation matrices. Just see equation 14.