Quiz 2, decoded: the likelihood half of GenAI.

Straight from the identity. \(\log p_\theta(x)\) is a constant once \(\theta\) is fixed, so \(J = \log p_\theta(x) - \text{gap}\): a smaller gap gives a strictly larger ELBO, and subtracting the two equations kills the constant, leaving \(J_1-J_2=\text{gap}_2-\text{gap}_1\). Option 3 is a trap — nothing forces the ELBO to be negative; with a continuous density \(\log p_\theta(x)\) can be positive.

A discrete latent means the expectation is just a weighted sum over the two states — no sampling needed.

This exact question, with these exact numbers, appeared in two separate papers. Bank it. The only trap is forgetting the \(-\log q\) term (which gives −3.80) or using \(\log_{10}\).

The whole difficulty is the index convention. \(\gamma_{ij}\) is "point \(i\), component \(j\)", so for component 2 you take the second subscript = 2 entries: \(\gamma_{12}=0.1\) and \(\gamma_{22}=0.8\) — a column, not a row. Grabbing the row \((0.2, 0.8)\) gives 13.0, which is wrong. This question appeared verbatim in two papers.

Evaluate both densities at \(x=7\) with \(\mathcal N(x|\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-(x-\mu)^2/2\sigma^2}\):

Watch the variance vs standard deviation. The second component has \(\sigma^2=4\Rightarrow\sigma=2\), so the normalising constant is \(1/\sqrt{2\pi\cdot4}=1/\sqrt{8\pi}\), not \(1/\sqrt{8\pi^2}\) and not \(1/\sqrt{2\pi}\).

2025 T3: the densities are handed to you, so it is pure arithmetic: \(\gamma_{n1}=\frac{0.4(0.10)}{0.4(0.10)+0.6(0.05)}=\frac{0.04}{0.07}=0.5714\).

Practice Q5: if the Gaussians are identical they are a common factor in numerator and denominator and cancel, leaving \(\gamma_1=\pi_1/(\pi_1+\pi_2)=2\pi_2/3\pi_2=2/3\) — independent of \(x\). This is the cleanest possible test of whether you understand that responsibilities are a normalised product of prior and likelihood.

\(\mu_1,\mu_2\) and one mixing weight. \(\alpha_2=1-\alpha_1\) is determined by the constraint so it is not free, and \(\sigma^2\) is given as known and shared. Parameter-counting questions always hinge on spotting exactly these two things: which constraint removes a degree of freedom, and which quantity is declared known.

The last option is the subtle one. At convergence \(\alpha_j = \frac1N\sum_n\gamma_{nj}\) — the average responsibility across the dataset. Individual points still have their own posteriors \(\gamma_{nj}\), which differ from each other and from \(\alpha_j\). They would only coincide if every point had identical responsibilities.

Direct test of "probabilistic representation": the net emits every parameter of the distribution it models. If the question had written \(p_\theta(x|z)=\mathcal N(x;\hat x_\theta(z), I)\) — covariance fixed to identity, as in the lecture notes — then the decoder would output the mean only, and the reconstruction term reduces to squared error.

The last option is the trap, and it is a good one: \(x\to\text{Enc}\to z\to\text{Dec}\to\tilde x\) is the training / reconstruction path, not the generative path. Generation never sees an input \(x\) — you draw \(z\) from the prior and push it through the decoder. The original paper's wording ("the generative process (Inference)") is confusing on purpose; go by generative = prior-first.

Three separate discriminators between the options, and you can kill three options without doing any algebra:

1. 1.Nested expectations. Because \(q\) factorises as \(q(z_2|x)q(z_1|z_2,x)\), the inner expectation over \(z_1\) must sit inside the outer one over \(z_2\). Options 1 and 2 flatten it.
2. 2.KL against the conditional prior. The \(z_1\) KL must be against \(p_\theta(z_1|z_2)\), not the marginal \(p_\theta(z_1)\) — that is what the model actually defines.
3. 3.Signs. KL terms are always subtracted from an ELBO. Option 4 adds them.

This is the bridge question to diffusion: a DDPM is exactly this, with \(T\) layers instead of 2 and the \(q\)'s frozen.

If you had used \(\sigma^2=e^{1.6}=4.953\) you would get \(-4.93\) — outside the accepted range. The \(\tfrac12\) is the whole question.

The official key applies \(z=\mu+\Sigma\odot\epsilon\) literally, treating the diagonal entries as standard deviations:

Read the distractors — they are a map of the three conventions. Option 2 is what you get treating the entries as variances (\(\mu+\sqrt{\Sigma}\epsilon\)); option 3 is \(\mu+\epsilon\) with unit \(\sigma\). The examiner wants the formula as written on the slide: \(z=\mu_\phi(x)+\Sigma_\phi(x)\epsilon\). When a paper writes \(\Sigma\) and gives you the matrix directly, plug it in unchanged; reach for the square root only when the symbol is explicitly \(\sigma^2\) or \(\log\sigma^2\).

All the randomness is pushed into \(\epsilon\), which is a constant as far as \(\phi\) is concerned; given \(\epsilon\), the map \(\phi\mapsto z\) is a differentiable deterministic function, so backprop reaches the encoder. Option 1 is false — the KL has a closed form and needs no sampling at all. Option 4 is a genuinely tempting near-miss: reparameterized ("pathwise") gradients are usually lower-variance than the score-function alternative, but that is a bonus, not the reason it is essential. Without the trick the gradient does not merely have high variance — it cannot be computed at all.

Note how the two minus signs turn into plus when you subtract a negative log-variance. That is where nearly all the arithmetic errors live.

Part (b) is the sign-convention question in miniature. "Reconstruction mismatch (negative log-likelihood) = 5.2" means \(\mathbb E_q\log p_\theta(x|z)=-5.2\). The ELBO is being maximised, so both contributions are negative and the answer is negative. In the minimisation convention the same numbers give \(+6.36\).

Reading \(\Sigma\)'s diagonal as the variances, so \(\log\sigma_j^2=-0.8,-1.2,0.4\):

For (b), remember the sign: \(J=\underbrace{(-186.4)}_{\mathbb E\log p_\theta(x|z)}-\underbrace{D_{KL}}_{\ge0}\).

Part (c) is the conceptual payload: the loss is quadratic in the mean error, so halving the error (0.4→0.2) divides the KL by 4. Part (d): the minimum is at zero error, \(\mu_\theta=\mu_q=1.2\), where KL = 0. Practice Q14 is the same idea as a comparison — the model whose mean is closer to \(\mu_q\) wins, so \(\mathcal N(1.3,0.16)\) beats \(\mathcal N(1.1,0.16)\) against a true mean of 1.5.

This is exactly the DDPM consistency term (§12) — which is why "training a diffusion model" is really just "regression onto \(\mu_q\)".

\(194.8+0.5(3.6)=194.8+1.8=196.6>196.0\). Trivial arithmetic, but note the applied framing: a VAE trained only on normal data reconstructs normal inputs well and anomalies badly, so a high reconstruction error is an anomaly signal. Expect more application-flavoured wrappers like this — the maths inside stays elementary.

Same arithmetic, three sign outcomes decided by one word in the stem. Also note 2025 T3 plants \(D=5\) as a pure distractor — the latent dimension does not enter the loss assembly at all once you are handed \(\mathcal L_{KL}\).

Everything except sharper reconstructions. A bigger \(\beta\) puts more weight on "make \(q\) look like the prior", which simultaneously (4) pulls the posterior to the prior, (3) squeezes information out of the channel from \(x\) to \(z\), and (1) — the empirical finding the \(\beta\)-VAE paper is famous for — encourages axis-aligned, disentangled factors. The cost is paid entirely by reconstruction quality, so option 2 is exactly backwards. With no negative marking, if you are confident of 4 and 3, take 1 as well.

The same phenomenon from two directions — a favourite examiner move. The KL is the only force pushing \(q_\phi(z|x)\) toward a fixed distribution, so it is the driver; and KL \(\to0\) everywhere is the definition of the symptom.

Option 4 in Q77 is the interesting distractor: a maximal ELBO would be a good outcome, but collapse maximises only the KL half while the reconstruction half rots — the total ELBO is not at its best.

Practice Q4 flips it into a comparison: encoder A has per-dimension KL ≈ 0, encoder B ≈ 1.5, same reconstruction quality. The right reading is "B uses its latent more informatively; A has partially collapsed toward the prior." Higher KL = the posterior is allowed to move away from the prior = the code is carrying information.

Both dropped items exist for the same reason: they are artefacts of a continuous Gaussian posterior. VQ-VAE's posterior is a deterministic \(\arg\min\) onto a finite codebook — there is no \(\sigma\) to sample and no continuous density to measure a KL against. What survives is what any autoencoder needs: a decoder and a reconstruction loss.

\(D=32\) is a distractor and so is the trailing 32 in the shape. Once a vector has been quantized, all that travels to the decoder is which entry it was — an index. The codebook dimension affects the codebook's own storage, never the payload. Appeared in two papers, both times graded exact-match, so do the \(\log_2\) carefully.

It is a squared L2 norm — no square root. The sg[·] (stop-gradient) changes nothing about the value; it only stops gradient flowing into the codebook from this term, so that the commitment loss pushes the encoder toward the codebook rather than the reverse.

The entire schedule description is decoration. You are handed \(\bar\alpha_{200}\), so \(\beta_1,\beta_T,T\) never enter the arithmetic. Appeared in two papers with identical numbers. The only real trap is taking \(\sqrt{1-\bar\alpha_t}=1-\sqrt{\bar\alpha_t}\) — it is not.

At \(t=1\), \(\bar\alpha_1=\alpha_1=1-\beta_1\approx1\) (since \(\beta_1=10^{-4}\)), so \(\sqrt{1-\bar\alpha_1}\approx0\) — barely any noise, \(x_1\approx x_0\). At \(t=T\), \(\bar\alpha_T\) is a product of 1000 numbers each \(<1\), so \(\bar\alpha_T\approx0\) and \(\sqrt{1-\bar\alpha_T}\approx1\) — pure noise, matching \(q(x_T)\approx\mathcal N(0,I)\). Option 4 confuses the per-step variance \(\beta_t\) with the cumulative one.

Q8 is the defining triple plus the closed form; the false option contradicts the whole point — only the reverse process is learned.

Q9's false option is a good one: a VAE's \(k\ll d\) compression is typical but not required, and a DDPM explicitly does not compress at all — so "necessarily lower-dimensional" fails for both.

Q11's last option is the subtle trap. With \(\bar\alpha_t=0.1\) the signal coefficient is \(\sqrt{0.1}\approx0.32\) — small, but not zero. Information about \(x_0\) is heavily degraded, never fully erased at finite \(t\). "Approaches \(\mathcal N(0,I)\)" is a limit statement, not an equality.

Just rearrange the forward marginal \(x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon\) for \(x_0\): subtract the noise term, divide by \(\sqrt{\bar\alpha_t}\).

Options 3 and 4 differ only by the bars. The marginal from \(x_0\) is governed by the cumulative \(\bar\alpha_t\), never the per-step \(\alpha_t\) — an unbarred alpha in a formula relating \(x_t\) to \(x_0\) is always wrong. Options 1 and 2 fail a cruder check: they multiply by \(\sqrt{\bar\alpha_t}\) where the inversion must divide.

The negative predicted noise is the whole trap. Subtracting a negative adds. Rushing gives \(\frac{0.5-0.1095}{0.8367}=0.467\), outside the accepted band.

Options 3 and 4 are eliminated instantly — the posterior mean must depend on both \(x_t\) and \(x_0\), and both of those drop one. That leaves the \(\tilde\beta_t\) fraction as the only real decision.

Two independent ways to settle it: (i) match the derived form \(\sigma_q^2=\frac{(1-\alpha_t)(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\) and read off \(1-\alpha_t=\beta_t\); (ii) the sanity check — conditioning on more information (both \(x_t\) and \(x_0\)) must reduce uncertainty, so we need \(\tilde\beta_t<\beta_t\), which forces the smaller quantity \(1-\bar\alpha_{t-1}\) on top.

Note the mean here is written in the \(\epsilon\)-substituted form, since \(\frac{x_t-\sqrt{\bar\alpha_t}x_0}{\sqrt{1-\bar\alpha_t}}=\epsilon\) — algebraically identical to the blend-of-\(x_t\)-and-\(x_0\) form above.

\(x_0=1.5\) and \(\bar\alpha_{400}=0.75\) are pure distractors. \(L_{\text{simple}}\) compares noise to predicted noise and nothing else; you never need to form \(x_t\). The examiner supplies enough information to compute \(x_t\) precisely to see whether you know that you don't have to. Recognising which given quantities are irrelevant is a recurring skill in this paper.

This concept has now been examined three times in three different forms — as a plain MCQ, as a true/false, and (Practice Q12) as a student's wrong claim to be refuted. Learn the refutation sentence:

The final step must return a clean sample, so injecting noise at the very end would only degrade it. Practically: in the sampling loop the noise term is gated as z = randn_like(x) if t > 1 else 0. When one option is "All of these" and the others are each individually defensible, it is almost always the answer.

Setting \(\Sigma_\theta=\Sigma_q\) makes the log-det and trace terms of the Gaussian KL cancel, collapsing it to \(\frac{1}{2\sigma_q^2}\|\mu_\theta-\mu_q\|^2\) — and from there to \(\|\epsilon-\hat\epsilon_\theta\|^2\). The simplification of the objective is the reason.

Option 1 is the near-miss and deserves attention: it is a true statement — \(\sigma_q^2\) genuinely depends only on the schedule — but it describes a consequence, not the motivation. When a distractor is true but answers a different question, it is still wrong.

Everything else in the stem is a distractor: \(32\times32\times3=3072\), the trunk layer, the ReLU (zero parameters) and the second head for \(\log\sigma^2\). Only the \(\mu\)-head is asked for. If you had counted both heads you would get 16040; the trunk alone is \(3072\times400+400=1{,}229{,}200\). Same question, same numbers, two papers.

Indexing a 1-D schedule with a [B] tensor gives a [B] result, which must become [B,1,1,1] to broadcast against [B,C,H,W]. [:, None, None, None] adds those three axes as a view — no data copied.

• Option 1 — gather returns the right values but stays [B]: no broadcast axes, so the multiply against 4-D fails.
• Option 4 — F.embedding expects a 2-D weight matrix, not a 1-D buffer.
• Option 2 is the interesting one, and the two papers word it differently. 2026 T1 writes .view(-1, 0, 1, 1), which is plainly invalid — a zero-sized dimension. 2025 T2 writes .view(-1, 1, 1, 1), which is functionally correct; there it loses only on the "most memory-efficient" tiebreak. If you see the 0 variant, eliminate it instantly.