You are not authenticated, login.
text: sort by
tags: modified
type: chronology
{231} is owned by tlh24.{725} is owned by tlh24.
hide / / print
ref: -0 tags: diffusion models image generation OpenAI date: 12-24-2021 05:50 gmt revision:0 [head]

Some investigations into denoising models & their intellectual lineage:

Deep Unsupervised Learning using Nonequilibrium Thermodynamics 2015

  • Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli
  • Starting derivation of using diffusion models for training.
  • Verrry roughly, the idea is to destroy the structure in an image using diagonal Gaussian per-pixel, and train an inverse-diffusion model to remove the noise at each step. Then start with Gaussian noise and reverse-diffuse an image.
  • Diffusion can take 100s - 1000s of steps; steps are made small to preserve the assumption that the conditional probability, p(x t1|x t)N(0,I)p(x_{t-1}|x_t) \propto N(0, I)
    • The time variable here goes from 0 (uncorrupted data) to T (fully corrupted / Gaussian noise)

Generative Modeling by Estimating Gradients of the Data Distribution July 2019

  • Yang Song, Stefano Ermon

Denoising Diffusion Probabilistic Models June 2020

  • Jonathan Ho, Ajay Jain, Pieter Abbeel
  • A diffusion model that can output 'realistic' images (low FID / low log-likelihood )

Improved Denoising Diffusion Probabilistic Models Feb 2021

  • Alex Nichol, Prafulla Dhariwal
  • This is directly based on Ho 2020 and Shol-Dickstein 2015, but with tweaks
  • The objective is no longer the log-likelihood of the data given the parameters (per pixel); it's now mostly the MSE between the corrupting noise (which is known) and the estimated noise.
  • That is, the neural network model attempts, given x tx_t to estimate the noise which corrupted it, which then can be used to produce x t1x_{t-1}
    • Simpicity. Satisfying.
  • The also include a reweighted version of the log-likelihood loss, which puts more emphasis on the first few steps of noising. These steps are more important for NLL; reweighting also smooths the loss.
    • I think that, per Ho above, the simple MSE loss is sufficient to generate good images, but the reweighted LL improves the likelihood of the parameters.
  • There are some good crunchy mathematical details on how how exactly the the mean and variance of the estimated Gaussian distributions are handled -- at each noising step, you need to scale the mean down to prevent Brownian / random walk.
    • Taking these further, you can estimate an image at any point t in the forward diffusion chain. They use this fact to optimize the function approximator (a neural network; more later) using a (random but re-weighted/scheduled) t and the LL loss + simple loss.
  • Ho 2020 above treats the variance of the noising Gaussian as fixed -- that is, β \beta ; this paper improves the likelihood by adjusting the noise varaince mostly at the last steps by a ~β t~\beta_t , and then further allowing the function approximator to tune the variance (a multiplicative factor) per inverse-diffusion timestep.
    • TBH I'm still slightly foggy on how you go from estimating noise (this seems like samples, concrete) to then estimating variance (which is variational?). hmm.
  • Finally, they schedule the forward noising with a cosine^2, rather than a linear ramp. This makes the last phases of corruption more useful.
  • Because they have an explicit parameterization of the noise varaince, they can run the inverse diffusion (e.g. image generation) faster -- rather than 4000 steps, which can take afew minutes on a GPU, they can step up the variance and run it only for 50 steps and get nearly as good images.

Diffusion Models Beat GANs on Image Synthesis May 2021

  • Prafulla Dhariwal, Alex Nichol

In all of above, it seems that the inverse-diffusion function approximator is a minor player in the paper -- but of course, it's vitally important to making the system work. In some sense, this 'diffusion model' is as much a means of training the neural network as it is a (rather inefficient, compared to GANs) way of sampling from the data distribution. In Nichol & Dhariwal Feb 2021, they use a U-net convolutional network (e.g. start with few channels, downsample and double the channels until there are 128-256 channels, then upsample x2 and half the channels) including multi-headed attention. Ho 2020 used single-headed attention only at the 16x16 level. Ho 2020 in turn was based on PixelCNN++

PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications Jan 2017

  • Tim Salimans, Andrej Karpathy, Xi Chen, Diederik P. Kingma

which is an improvement to (e.g. add selt-attention layers)

Conditional Image Generation with PixelCNN Decoders

  • Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, Koray Kavukcuoglu

Most recently,

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

  • Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen

Added text-conditional generation + many more parameters + much more compute to yield very impressive image results + in-painting. This last effect is enabled by the fact that it's a full generative denoising probabilistic model -- you can condition on other parts of the image!

hide / / print
ref: bookmarks-0 tags: EMG schematic amplifier prosthetic myopen date: 01-03-2012 23:08 gmt revision:3 [2] [1] [0] [head]

hide / / print
ref: -0 tags: FIR LMS decorrelation adaptive filter matlab myopen date: 01-03-2012 03:35 gmt revision:7 [6] [5] [4] [3] [2] [1] [head]

LMS-based adaptive decorrelator, xn is the noise, xs is the signal, len is the length of the signal, delay is the delay beyond which the autocorrelation function of the signal is zero but the acf of the noise is non-zero. The filter is very simple, and should be easy to implement in a DSP.

function [y,e,h] = lms_test(xn, xs, len, delay)
h = zeros(len, 1); 
x = xn + xs; 
for k = 1:length(x)-len-delay
	y(k) = x(k+delay:k+len-1+delay) * h ; 
	e(k) = x(k) - y(k); 
	h = h + 0.0004 * e(k) * x(k+delay:k+len-1+delay)'; 
It works well if the noise source is predictable & stable: (black = sinusoidal noise, red = output, green = error in output)

Now, what if the amplitude of the corrupting sinusoid changes (e.g. due to varying electrode properties during movement), and the changes per cycle are larger than the amplitude of the signal? The signal will be swamped! The solution to this is to adapt the decorrelating filter slowly, by adding an extra (multiplicative, nonlinear) gain term to track the error in terms of the absolute values of the signals (another nonlinearity). So, if the input signal is on average larger than the output, the gain goes up and vice-versa. See the code.

function [y,e,h,g] = lms_test(xn, xs, len, delay)
h = zeros(len, 1); 
x = xn + xs; 
gain = 1;
e = zeros(size(x)); 
e2 = zeros(size(x)); 
for k = 1:length(x)-len-delay
	y(k) = x(k+delay:k+len-1+delay) * h; 
	e(k) = (x(k) - y(k)); 
	h = h + 0.0002 * e(k) * x(k+delay:k+len-1+delay)'; % slow adaptation. 
	y2(k) = y(k) * gain; 
	e2(k) = abs(x(k)) - abs(y2(k)); 
	gain = gain + 1 * e2(k) ; 
	gain = abs(gain);
	if (gain > 3) 
		gain = 3;
	g(k) = gain; 

If, like me, you are interested in only the abstract features of the signal, and not an accurate reconstruction of the waveform, then the gain signal (g above) reflects the signal in question (once the predictive filter has adapted). In my experiments with a length 16 filter delayed 16 samples, extracting the gain signal and filtering out out-of-band information yielded about +45db improvement in SNR. This was with a signal 1/100th the size of the disturbing amplitude-modulated noise. This is about twice as good as the human ear/auditory system in my tests.

It doesn't look like much, but it is just perfect for EMG signals corrupted by time-varying 60hz noise.

hide / / print
ref: notes-0 tags: NXT EMG design myopen date: 01-03-2012 02:49 gmt revision:33 [32] [31] [30] [29] [28] [27] [head]


devices that can be turned off & on to save power (e.g. actually disconnected from power through a P channel MOSFET. must be careful to tristate all outputs before disabling, otherwise we'll get current through the ESD protection diodes )

  1. ethernet
  2. usb reset
  3. usb host power
  4. RS 232
  5. LCD (or at least the 40ma, 6V LED -- the LCD can be disabled in software, and it only consumes 2-3 ma anyway.)
  6. core voltage boost 0.8v to 1.2V
  7. AFE

hide / / print
ref: -0 tags: LDA myopen linear discriminant analysis classification date: 01-03-2012 02:36 gmt revision:2 [1] [0] [head]

How does LDA (Linear discriminant analysis) work?

It works by projecting data points onto a series of planes, one per class of output, and then deciding based which projection plane is the largest.

Below, to the left is a top-view of this projection with 9 different classes of 2D data each in a different color. Right is a size 3D view of the projection - note the surfaces seem to form a parabola.

Here is the matlab code that computes the LDA (from myopen's ceven

% TrainData and TrainClass are inputs, column major here.
% (observations on columns)
N = size(TrainData,1);
Ptrain = size(TrainData,2);
Ptest = size(TestData,2);

% add a bit of interpolating noise to the data.
sc = std(TrainData(:)); 
TrainData =  TrainData + sc./1000.*randn(size(TrainData));

K = max(TrainClass); % number of classes.

%%-- Compute the means and the pooled covariance matrix --%%
C = zeros(N,N);
for l = 1:K;
	idx = find(TrainClass==l);
		% measure the mean per class
	Mi(:,l) = mean(TrainData(:,idx)')';
		% sum all covariance matrices per class
	C = C + cov((TrainData(:,idx)-Mi(:,l)*ones(1,length(idx)))');

C = C./K; % turn sum into average covariance matrix
Pphi = 1/K;
Cinv = inv(C);

%%-- Compute the LDA weights --%%
for i = 1:K
	Wg(:,i) = Cinv*Mi(:,i);
		% this is the slope of the plane
	Cg(:,i) = -1/2*Mi(:,i)'*Cinv*Mi(:,i) + log(Pphi)';
		% and this, the origin-intersect.

%%-- Compute the decision functions --%%
Atr = TrainData'*Wg + ones(Ptrain,1)*Cg;
	% see - just a simple linear function! 
Ate = TestData'*Wg + ones(Ptest,1)*Cg;

errtr = 0;
AAtr = compet(Atr');
	% this compet function returns a sparse matrix with a 1
	% in the position of the largest element per row. 
	% convert to indices with vec2ind, below. 
TrainPredict = vec2ind(AAtr);
errtr = errtr + sum(sum(abs(AAtr-ind2vec(TrainClass))))/2;
netr = errtr/Ptrain;
PeTrain = 1-netr;

hide / / print
ref: -0 tags: openGL GLEW extensions nvidia Cg Linux date: 12-03-2010 23:47 gmt revision:0 [head]

http://www.gamedev.net/community/forums/topic.asp?topic_id=432583 -- bump, it helped me solve a problem with cgGLGetLatestProfile() failing!! (the class I was using to examine openGL extensions was writing to the openGL supplied string)

hide / / print
ref: work-0 tags: fur openGL directX shell hull algorithm date: 11-03-2010 15:47 gmt revision:0 [head]

http://www.xbdev.net/directx3dx/specialX/Fur/index.php -- for future reference. Simple algorithm that seems to work quite well. Can be done almost entirely in vertex shader...

hide / / print
ref: notes-0 tags: wireless nordic headstage bridge neurorecord pictures photo EMG myopen date: 03-12-2009 02:33 gmt revision:4 [3] [2] [1] [0] [head]

hide / / print
ref: -0 tags: myopen EMG recordings NLMS noise date: 07-29-2008 18:32 gmt revision:2 [1] [0] [head]

Myopen amplifiers & analog/digital filters & NLMS are working properly! Below, a recording from my deltiod as I held my arm up: (only one EMG channel active, ground was my knee))

Yellow traces are raw inputs from ADC, blue are the output from the IIR / adaptive filters; hence, you only see 8 of the 16 channels. Read from bottom to top (need a -1 in some opengl matrix somewhere...) Below, the system with no input except for free wires attached to one channel (and picking up ambient noise). For this channel, NLMS could not remove the square wave - too many harmonics - but for all other channels the algorthim properly removes 60hz interference :)

Now, let me clean this EEG paste off my shoulder & leg ;)

hide / / print
ref: notes-0 tags: flock OSS opensource Mozilla LGPL money corporations browser comment embedded mobile opera date: 12-20-2007 16:23 gmt revision:0 [head]

I'm posting my comments about http://flock.com/ here just in case they are removed from the actual site

This is all very interesting. I just downloaded it, and flock seems to work well. I'm probably not going to use it unless there is some demonstrable technical superiority (e.g. leaks less memory than firefox), as the social sites just distract me from getting work done.

Anyway, I have a question: how are you going to make money? How are you paying the developers? If you are not and it is all OSS, where is the source? It seems like the VC's are just throwing money away for the (hypothetical) good of the social-network crowd. Or, rather, you are indirectly funding the popularity of sites that flock makes it easy to get at. Are these sites (e.g. facebook) paying you? Wait -- flock allows you to look at content and not the ads. They are not paying you.

Perhaps you are moving along the lines of Opera, and intending to get people addicted to flock to a degree that they demand it on their mobile devices. Mobile devices are closed (for now .. check google), hence you can make money licensing software to phone manufacturers. I imagine that you'll have to rewrite the Mozilla core to do this (unless phones become significantly more powerful - not likely, they are battery devices. ) Mozilla is (L)GPL - you'll have to release the source. To the best of my knowledge, with non-physical goods money can only be made from gradients in knowledge (pun.. intended), therefore you will have to keep the source closed. If this is the case, you'll be able to make money (on this, i don't know what else you have planned) for a while, and when you can no longer, I hope you open the source like netscape.

Technically, though, excellent job! your website is also very pretty!

hide / / print
ref: bookmark-0 tags: open source cellphone public network date: 11-13-2007 21:28 gmt revision:2 [1] [0] [head]


  • kinda high-level, rather amorphous, but generally in the right direction. The drive is there, the time is coming, but we are not quite there yet..
  • have some designs for wireless repeaters, based on 802.11g mini-pci cards in a SBC, 3 repeaters. total cost about $1000
  • also interesting: http://www.opencellphone.org/index.php?title=Main_Page

hide / / print
ref: bookmark-0 tags: blackfin ARM buglabs opensource fsf java date: 10-19-2007 20:14 gmt revision:7 [6] [5] [4] [3] [2] [1] [head]

buglabs, makers of plug n' play configurable handheld platform based on open-source technologies

  • uses the ARM11 processor.
    • 0.24mw/mhz; blackfin: 0.15mw/mhz (measured, by me; DSP was running relatively intense elliptic filters on 2m samples/sec data); blackfin wins again!
  • also uses Java. beh, I don't like java at all. Lua is way better.
  • see their list of open-source software that they use/employ, eg. http://aptana.com/ & http://www.blackdown.org/ (blackdown is a port of Sun's JDK to Linux).
  • references http://emoglen.law.columbia.edu/publications/maine-speech.html -- information should be free!
    • quote: [...]the chief technology officer of the Microsoft Corporation, Craig Mundie, made a public speech, in which he said that my client the Free Software Foundation (the Free Software Foundation, and only the Free Software Foundation) was destroying the global software industry. hah!
    • software in the twenty-first century is becoming a public utility, not a product
    • goal of a programmer: Write once, run everywhere. minimize duplicated effort.
    • It's an emergent property of connected human minds that they do create. :) :)
    • still, you have to pay the programmers and developers .. like me .. they need to live. even if the software is free, there must be some way of recouping the cost of making software, information, and music. This is going to require management, probably governmental, but without obviating the natural human inclination to compete.
      • Actually, I dare venture that even without present-day capitalism & with proper education, humans will compete, better things will be created.

hide / / print
ref: bookmark-0 tags: eeg oss openeeg recording linux date: 0-0-2007 0:0 revision:0 [head]


hide / / print
ref: bookmark-0 tags: OpenGL shaders vertex pixel fragment GLSL Linux programming C++ date: 0-0-2006 0:0 revision:0 [head]


easy to compile on my debian system - all the development libraries had debian packages!

also of interest :