The End Term: from pixels to preferences.

Denominator first: \(1-\bar\alpha_t=0.09\).

The 2-second check that beats the arithmetic: \(\mu_q\) is a convex combination of \(x_0\) and \(x_t\), so \(c_1+c_2\) must be \(\approx1\). Only the pair \((0.456,0.544)\) and the decoy \((0.5,0.5)\) satisfy that, and \(c_1=c_2\) would need \(\bar\alpha_{t-1}=\bar\alpha_t\). Answer identified without computing either fraction.

\(\sqrt{1-\bar\alpha_t}=\sqrt{1-0.36}=\sqrt{0.64}=0.8\), so the score is \(-[0.8,-1.6]^\top/0.8=[-1.0,\,2.0]^\top\).

Read the distractors — they are a checklist of the three errors above. \([-0.8,1.6]^\top\) is forgetting to divide at all; \([-1.25,2.5]^\top\) is dividing by \(\sqrt{\bar\alpha_t}=0.6\); \([-0.48,0.96]^\top\) is multiplying by \(1-\bar\alpha_t=0.64\) instead of dividing by its root. Every option preserves the sign flip, so the sign is free here — but the FN twin, and the 2025 T2 version below, make you supply it.

Note on the FN twin. The same question with \(\bar\alpha_t=0.49\) has a broken key — see the flag in §00. This AN version is the one to trust; the numbers work out exactly.

Direct, without Tweedie: \(\log q(x_t|x_0)=-\frac{\|x_t-\sqrt{\bar\alpha_t}x_0\|^2}{2(1-\bar\alpha_t)}+\text{const}\), so

The numerator \(x_t-\sqrt{\bar\alpha_t}x_0\) is \(\sqrt{1-\bar\alpha_t}\,\epsilon\) by definition, and one power of the root cancels. That cancellation is precisely why the answer has \(\sqrt{1-\bar\alpha_t}\) and not \(1-\bar\alpha_t\) — option 3 is the trap for anyone who stops one line early.

Always compute the difference vector first — it is the only step where a sign can go wrong.

Distractor 1, \([3.2,-1.7]^\top\), is what you get from \(\epsilon_u+s\,\epsilon_c\) in the first coordinate — forgetting to subtract \(\epsilon_u\) inside the bracket. Distractor 4 is \(s\big(\epsilon_c-\epsilon_u\big)\) alone, dropping the leading \(\epsilon_u\). Sanity check that catches both: the guided vector must lie on the ray through \(\epsilon_u\) and \(\epsilon_c\), beyond \(\epsilon_c\). Here that ray runs down-and-right at slope \(-1\) from \([0.2,0.1]^\top\); only \([2.0,-1.7]^\top\) sits on it.

This is the conceptual pay-off of the whole section, and it is worth stating as a principle: guidance and the sampler are orthogonal concerns. Guidance only changes what number you feed in as \(\hat\epsilon\); the sampler only decides what to do with \(\hat\epsilon\). So CFG drops into DDIM, DDPM, or any other sampler with no retraining — exactly as it drops into a latent diffusion model.

Verbatim in both 2025 T3 papers, options merely reordered. Bank it. The three distractors are all false of DDIM: the network is unchanged (option 1); DDIM uses fewer steps (option 2); and it needs the same one forward pass per step (option 3 — that is CFG, which needs two).

This option is a three-fact summary of the entire section, and it is the best single sentence to memorise about DDIM:

• Deterministic — because \(\sigma_t=0\) kills the \(\sigma_t z\) term.
• Non-Markovian — because \(q_\sigma(x_{t-1}|x_t,x_0)\) conditions on \(x_0\) as well as \(x_t\).
• Matches the marginals — which is why the DDPM-trained network is still the correct network, and why you may skip steps.

Option 3 inverts the convention and is the one to watch: \(\eta=1\) is DDPM, \(\eta=0\) is DDIM.

Get here by checking the two endpoints, not by recalling the formula. The definition is \(\sigma_t=\eta\sqrt{\tilde\beta_t}\), so the variance picks up \(\eta^2\). Test it: \(\eta=0\) must give variance 0 (DDIM) and \(\eta=1\) must give \(\tilde\beta_t\), the DDPM posterior variance. Option 1 passes both. Option 3 fails at \(\eta=0\) (gives full DDPM noise — backwards). Options 2 and 4 use \(\beta_t\), the forward step variance, not the posterior variance \(\tilde\beta_t=\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t\).

Worth meeting now rather than in §14, because it shows the whole back half of the course in one line. The AR factorisation \(p_\theta(x_t|x_{<t})\) already is a policy \(\pi_\theta(a_t|s_t)\): the conditioning context \(x_{<t}\) is the state, the emitted token \(x_t\) is the action. No reinterpretation of the model is needed — only a change of what you optimise. The AR factorisation is the reason RLHF is possible at all.

Step 0 — do this before reading the sub-questions. Compute all three projections in full. Each is a \(3\times3\) times \(3\times2\) product; nine dot products of length 3, about ninety seconds.

(a) \(q_2=(3,1)\). Raw scores against the three keys: \(3(3)+1(1)=10\), \(3(1)+1(2)=5\), \(3(2)+1(3)=9\). Divide each by \(\sqrt2\):

(b) Softmax the scaled scores. Subtract the max (7.0711) first for stability, then exponentiate:

Two ways to shortcut option-matching here. First, the output must be a convex combination of \((2,1),(3,2),(1,3)\), so both coordinates must lie in \([1,3]\) — that alone kills the options \([0.37,0.30,0.33]\) and \([0.66,0.02,0.32]\), which are attention weights mistaken for the output (a deliberate trap, and note they even have three components rather than two). Second, of the two survivors \([1.66,1.85]\) and \([1.70,1.67]\): token 1 gets two-thirds of the weight and its value is \((2,1)\), which has first coordinate greater than second — so the output should too. Only \([1.70,1.67]\) does.

The given exponentials are a gift — they tell you the answer's shape before you compute anything. Only two distinct values are supplied for three scores, so two of the three scaled scores must be equal. Expect a tie.

(a) \(q_1=(1,2)\). Raw scores: \(q_1^\top k_1=1+6=7\), \(q_1^\top k_2=1+2=3\), \(q_1^\top k_3=3+4=7\). The lowest is 3 → token 2. Note the tie at 7 the given exponentials predicted. This sub-part needs no scaling and no softmax at all — softmax is monotone, so ranking is unaffected. Two marks for three dot products.

(b) Scaled: \(7/\sqrt2=4.9497\), \(3/\sqrt2=2.1213\), \(4.9497\) — matching the supplied \(e^{4.950}\) and \(e^{2.121}\) exactly.

(c) \(Z_1=0.4857(3,2)+0.0287(1,2)+0.4857(2,1)=(2.4570,\;1.5143)\), and \(2.4570+1.5143=3.9713\approx\mathbf{3.97}\).

The 10-second route to (c). The question asks for a sum of coordinates, so sum first and weight second. Row sums of \(V\) are \(5,3,3\), hence \(0.4857(5)+0.0287(3)+0.4857(3)=2.4285+0.0861+1.4571=3.9717\). One weighted sum instead of two — and it dodges the vector arithmetic entirely.

Do not compute the matrix. Position \((1,2)\) of \(QK^\top/\sqrt{d_k}\) is one dot product: row 1 of \(Q\) with row 2 of \(K\), divided by \(\sqrt4=2\). Everything else on the page is a distraction.

Note that \(Q\) and \(K\) are given directly here — no \(X\) and no \(W\) matrices to multiply, unlike the 2026 T1 clusters. Read which the paper gives you; projecting a matrix that was already projected is a costly and easy mistake. The one real trap is transposing: \((QK^\top)_{ij}=q_i\cdot k_j\), so the second index selects the key.

(a) Row 3 of \(S\) is \([1,0,3,2]\). Query 3 may see positions 1, 2, 3 and not 4, so only the last entry becomes \(-\infty\): \([1,0,3,-\infty]\). One mark for reading one row.

The zero is the trap. Option 3 masks position 1 because its score is... no, because its score looks small; option 4 masks position 2 because its score is literally \(0\). Neither is a mask. A score of 0 is a perfectly ordinary score — it contributes \(e^0=1\) to the softmax, which is why the paper bothers to supply \(e^0=1\). Masking depends on position only, never on value.

(b) Token 4 is the last query, so nothing is masked — row 4 is \([2,1,0,4]\) in full.

"Jointly" means add the numerators, not the probabilities — same thing here, but adding numerators over one shared denominator is faster and avoids two roundings. Sanity check: \(e^4=54.6\) is 83% of the mass on its own, so the answer for the other three positions together must be small. 0.154 for two of them is plausible; anything above 0.5 would not be.

(a) Exactly 0. Position 4 is in the future for query 3, so it is masked, and \(e^{-\infty}=0\). One mark, no arithmetic — and the fastest mark in the whole paper. Do not be tempted to compute a softmax.

(b) Row 4 is \([0,1,2,5]\), fully visible, and softmax is monotone: the largest score always wins the largest weight. Largest is 5 at position 4. Two marks, no arithmetic.

(c) Two different rows with two different denominators — that is the entire difficulty, and where the 3 marks are.

The mistake this question is built to catch is normalising both rows over all four positions. Row 2 sees only two positions, so its denominator has two terms — and \(e^5=148.413\) is supplied precisely so that anyone dividing by a whole-matrix sum gets a plausible-looking wrong number. Use it only where it belongs: nowhere in this sub-part.

Second row is \([1.0,\,2.0,\,1.0]\); query 2 sees positions 1 and 2, so the third entry is masked: \([1.0,\,2.0,\,-\infty]\).

Or skip the exponentials entirely. With only two survivors, softmax is the logistic function of the difference: \(P=\sigma(2.0-1.0)=\sigma(1)=1/(1+e^{-1})=0.731\). Worth remembering — a two-way softmax is always a sigmoid of the gap, which turns this into one keystroke.

The trap is including the masked third column: \(7.389/12.825=0.576\), which is why the accepted range (0.71–0.75) excludes it. The official key is typo'd as "0.71 to 075" — read it as 0.75.

(a) One \(\mathbb R^D\to\mathbb R^D\) layer has a \(D\times D\) weight matrix and a length-\(D\) bias: \(D^2+D\). Three such layers: \(3(D^2+D)=3D^2+3D\). Option 1 is the weights-only answer — correct arithmetic, wrong question. The phrase "including both weights and biases" is doing real work here; it is there to separate options 1 and 2.

(b) \(A\in\mathbb R^{N\times N}\) because every one of the \(N\) queries scores every one of the \(N\) keys. So \(N^2=100^2=\mathbf{10{,}000}\).

\(D=16\) is a red herring, and a well-built one. The head dimension is summed away by the dot product: \(q_i^\top k_j\) is a scalar whatever \(D\) is. Anyone who multiplies by 16 gets 160,000. Remember the shape chain — \(A\) is \(T\times T\), never \(T\times d_k\) — and this is a two-mark gift. It is also the origin of the transformer's \(O(T^2)\) cost, which is the whole motivation for §18.

"Evenly split across heads" is the phrase that decides it: \(d_k=D/h\), so the \(h\) per-head matrices of size \(D\times\frac{D}{h}\) tile exactly the same \(D\times D\) budget as one single-head matrix. Concatenating \(h\) slices of width \(D/h\) reconstructs width \(D\).

The intuition worth carrying: multi-head attention buys diversity of attention patterns, not capacity. If instead each head kept the full width \(d_k=D\) (not the standard construction) the count really would be \(h\) times larger — which is what option 2 is fishing for.

This asks about an activation, not a parameter count — so \(B\) and \(L\) matter and \(h\) does not.

The reasoning that makes \(h=8\) irrelevant is the shape-preservation property: concatenating 8 heads of width \(512/8=64\) gives width 512 again, and \(W^O\) maps \(512\to512\). The block's output is always \(B\times L\times d_{\text{model}}\) regardless of how it was internally partitioned — which is precisely why the residual connection \(x+\mathrm{sublayer}(x)\) is well-defined. \(h=8\) is there to tempt you into \(\times8\) or \(\div8\).

(a) \(W_O\) maps the concatenation back to model width. Concatenated width is \(h\,d_v=12\times64=768=d_{\text{model}}\), so \(W_O\in\mathbb R^{768\times768}\). The distractor \(768\times64\) is the size of one head's projection; \(64\times64\) and \(12\times64\) are dimensional nonsense. Check the consistency of the given numbers — \(h\,d_k=d_{\text{model}}\) holds here, confirming the standard even split.

(b) \(4\times512^2=4\times262{,}144=1{,}048{,}576\) (which is \(2^{20}\), i.e. exactly 1M — a nice self-check). Note how the option list maps to misreadings: \(262{,}144\) counts one matrix, \(786{,}432\) counts three (forgetting \(W_O\)), \(524{,}288\) counts two. The whole question is "how many matrices did they ask about". Read the stem twice, then multiply.

Step 1 — the projection. \(z_c\) is a row vector, so \(z_cW^C\) means "dot \(z_c\) with each column of \(W^C\)":

Step 2 — the residual. \(u=[1,2,0,1]+[3,4,5,2]=[4,6,5,3]\). Forgetting this addition is the biggest single risk in the cluster, and it corrupts both sub-answers.

Step 3 — the requested coordinate only. \(\sqrt{1.25}=1.11803\):

Notice the option design in (a): the four choices are the \(2\times2\) grid of "did you add the residual" (mean 4.5 vs 4.0) against "did you use the biased variance" (1.25 vs 1.50 — and \(5/3=1.67\) would be the \(n-1\) answer, with 1.50 as a nearby decoy). The option list is the pitfall list. And do not normalise all four coordinates in (b) — you need one.

(a) \(\mu=0\), so \(\sigma^2=\frac{4+0+4+0}{4}=2\) and \(\sigma=\sqrt2\). Divide through: \((2/\sqrt2,\,0,\,-2/\sqrt2,\,0)=(1.414,0,-1.414,0)\). Symmetry shortcut: the input is antisymmetric so \(\mu=0\) for free, and the zeros must stay zero — which alone eliminates every distractor.

(b) Deviations are \((-1,0,1)\); divide by 0.8165 to get \((-1.2247,\,0,\,1.2247)\); then scale by \(\gamma\):

The decoy \((-0.612,0,1.225)\) applies \(\gamma\) in reverse order — a pure sequencing error. Note the middle coordinate is 0 whatever \(\gamma_2\) is, because \(x_2\) equals the mean.

(c) Two marks of reading comprehension. \(f\) is the identity so \(f(\mathrm{LN}(x))=\mathrm{LN}(x)=(1,-1)\), and \(y=(2,0)+(1,-1)=(3,-1)\). The point being tested is the order of operations: here the normalisation is applied inside the sublayer path (pre-norm), so the residual adds the original \(x\), not the normalised one. Answer \((2,0)\) if you forget the sublayer, or \((1,-1)\) if you forget the residual — both are on the option list.

Under the stated independence approximation each layer multiplies the variance by \(1.1\):

The physics worth taking away: residual streams make variance grow geometrically with depth. That is exactly what the LayerNorm after each block exists to prevent — it rescales the stream back to zero mean and unit variance, so a 96-layer model does not blow up by \(1.1^{96}\approx11{,}000\times\).

Build the PE vector coordinate by coordinate. With \(d_m=4\) the four slots pair up as \((2i,2i{+}1)\) for \(i=0,1\), and the two supplied argument values (3 and 0.03) tell you the two wavelengths before you compute anything:

Then add element-wise: \([2+0.141,\;1-0.990,\;0+0.03,\;2+0.9996]=[2.141,\;0.010,\;0.03,\;2.9996]\).

The whole question turns on one coordinate. Slot 1 uses \(\cos3=-0.990\), which nearly cancels \(x_1=1\) to give \(0.01\). Option 2 has \(1.99\) — the sign of the cosine flipped. Option 3 misreads the exponent (\(10000^{2i/d_m}=100\), so \(0.03\) not \(0.30\)); option 4 forgets slot 0 entirely. Compute coordinate 1 first and you are done in ten seconds.

(b) PE is added, so \(\tilde x\in\mathbb R^{d_m}\) just like \(x\) — dimensionality preserved (killing "increases the dimensionality", which would describe concatenation). It does not "replace" semantic information — the sum retains both, and downstream linear layers can separate them because the two carry different structure. Nor does it normalise anything.

The FN paper offered options 1–4 and the AN paper offered 1, 2, 3 and 5; in both, the "learned parameters" option is the only false one. That is the pattern to remember: sinusoidal PE is a fixed, closed-form function of position, with no gradient and no parameters. (Learned positional embeddings exist as a separate design choice, but they are not what this formula describes.)

• Order information (2). Restates permutation-equivariance: attention sees a set, so order must be injected at the input.
• Geometric progression (3). The wavelengths \(2\pi\cdot10000^{2i/d_m}\) form a geometric series in \(i\) — ratio \(10000^{2/d_m}\) per pair — so early dimensions oscillate fast and late ones slowly, giving the model a multi-scale ruler. Strictly, the frequencies decay geometrically while the wavelengths grow; the key marks this true because the intended point is the geometric spacing. Do not be talked out of it by the direction of the word "grow".
• Relative positions (4). The real reason for sinusoids: \(\mathrm{PE}(j+k)\) is a fixed linear (rotation) transform of \(\mathrm{PE}(j)\) for any offset \(k\), straight from \(\sin(a+b)\) and \(\cos(a+b)\). So a linear layer can learn to detect "\(k\) positions apart" independently of absolute location.
• Uniqueness (5). With the lowest frequency having wavelength \(2\pi\cdot10000\), no two positions in any practical sequence share an encoding.

MSQ tactic: zero negative marking means select every option you can positively justify. Here, spotting the single false statement is the whole question.

Score all four candidates, then take the top two. Nothing else.

Log-probs are negative, so "highest score" means closest to zero. That sign is the only real trap, and it is what options 1 and 3 are built on: each keeps both children of a single beam, which is what happens if you rank by magnitude or expand only the currently-best beam. Beam search pools candidates across all beams — the winning pair here comes one from each. Note that the best step-\(t\!+\!1\) continuation (\(-0.2\), from 'hi') does not produce the best path, because 'hi' started half a nat behind. That is the entire point of beam search over greedy decoding.

The AN twin is the same question with 'cat'/'dog': \(-1.0,-1.5\) and steps \(-0.5,-1.0\) / \(-0.1,-0.4\), giving \(-1.5\) for ('cat','sits') and \(-1.6\) for ('dog','barks') — again one from each beam.

(a) Keep \(0.5\) and \(\tfrac13\); the surviving mass is \(0.5+\tfrac13=\tfrac56\). So

Option 0.5 is the un-renormalised probability — the whole point of the question is that truncation forces renormalisation, and the kept tokens' probabilities all rise.

(b) Accumulate until the total reaches \(p=0.9\): \(0.5\to0.7\to0.85\to0.95\). Three tokens give only \(0.85<0.9\), so you need the fourth: \(|S_p|=4\) with mass \(0.95\). Then

"Smallest set whose cumulative mass exceeds \(p\)" means you overshoot deliberately — you never stop at 0.85 and you never trim the last token to make the sum exactly 0.9. Option "Size 4, prob = 0.1" is the trap for skipping the renormalisation, and "Size 3, prob = 0" for stopping one token early.

Exactly the identity above, and the reason it is worth 3 marks twice: it is the theoretical licence for everything the exam then asks you to compute. Since the baseline term has zero mean, you may subtract \(V^\pi(s_t)\) for free — the gradient estimator stays unbiased while its variance drops, because \(R_t-V^\pi(s_t)\) is a much smaller number than \(R_t\).

Option 4 is the interesting distractor. \(\nabla_\theta b(s)\) would be the answer if \(b\) depended on \(\theta\) in a way that mattered — and this hints at the real constraint: the baseline must not depend on the action. A baseline \(b(s,a)\) would not factor out of the sum over \(a\), and would bias the gradient. That is why \(V^\pi(s)\) is a legal baseline and \(Q^\pi(s,a)\) is not.

The whole cluster is one table. Build it before answering anything. Compute the reward-to-go values backwards using \(R_t=r_t+\gamma R_{t+1}\) — four multiplications instead of ten.

(a) is the bottom-row \(R_t\): \(R_0=R(\tau)=6.446\). (b) is the row above: \(R_1=4.94\). (c) is the last column summed: \(0.723-0.188+0.160+0=\mathbf{0.695}\).

Three things that make or break this cluster. First, \(R_t\) restarts its discounting at \(t\) — \(R_1=r_1+\gamma r_2+\gamma^2r_3\), not \(\gamma r_1+\gamma^2r_2+\gamma^3r_3\). Dividing \(R(\tau)\) by \(\gamma\) is wrong; the backward recursion makes this impossible to get wrong. Second, \(\hat A_3=0\) exactly (the last reward-to-go is just \(r_3=4\), and \(V^\pi(s_3)=4.0\)), so \(g_3\) is decoration — a deliberate gift. Third, the advantages are small (0 to 1.4) while the returns are large: that shrinkage is the variance reduction the baseline exists to deliver.

Identical machinery, new numbers — which is precisely why the table is worth internalising.

\(\hat g=0.232+0.180-0.100+0=\mathbf{0.312}\). Again \(\hat A_3=0\) because \(V^\pi(s_3)\) is set equal to the final reward — this setter does that every time, so expect the last term to vanish and use it as a check that your table is right.

What the sign of \(\hat g\) means: it is positive here, so the update \(\theta\leftarrow\theta+\alpha\hat g\) increases the log-probability of the sampled actions on balance. Where \(g_t\hat A_t<0\) (at \(t=2\), because \(g_2\) is negative while the advantage is positive) that coordinate is pushed the other way.

(a) \(G_1=r_1+\gamma r_2+\gamma^2r_3=1+0.9(0)+0.81(2)=1+0+1.62=2.62\). The \(r_2=0\) is there to see whether you keep the place of the zero term — skip it and you compute \(1+0.9(2)=2.8\), which is the 2.81 decoy.

(b) The distractors are worth naming, because each is a specific misunderstanding:

• \(\mathbb E[\nabla_\theta\pi_\theta(a|s)]\) — missing the \(\log\). Without it there is no log-derivative trick and the expression is not an expectation under \(\pi_\theta\) at all.
• \(\mathbb E[\nabla_\theta V^\pi(s)]\) — differentiating the value function directly, which needs the dynamics and defeats the purpose.
• \(\nabla J=0\) — confusing the baseline term (which does vanish, §13 above) with the whole gradient.

The \(Q^\pi\)-weighted form is the theorem's canonical statement; the \(A^\pi\)-weighted form used in the course is that same expression with a zero-mean baseline subtracted, and the reward-to-go \(R_t\) is its one-sample estimate. All three are the same object at different levels of estimation.

Same structure in both papers: four true correspondences and one false reward claim. The reward option is false on the word "necessarily", and the reason is the whole motivation for Weeks 11–12 — if the reward were the pretraining log-likelihood, optimising it would just be more pretraining, and alignment would be a no-op. The reward comes from outside the likelihood: a model trained on human preference comparisons (§16).

Exam habit worth forming: in this course's MSQs, an option containing "necessarily", "always", "must" or "guarantees" is false far more often than not. Check whether a single counterexample exists.

Two separate facts, each with a clean justification:

• Output dimension \(=|V|\). At each step the policy emits a full distribution over the vocabulary — one probability per token, a 32000-vector summing to 1. It does not emit a single token; sampling from the distribution does that. (Option 4's "output dim 1" confuses the sampled action with the policy.)
• Trajectory space \(=|V|^T\). Each of the \(T\) positions independently takes any of \(|V|\) values, so the count of distinct length-25 sequences is \(32000^{25}\) — base is the vocabulary, exponent is the length.

Base-versus-exponent is the entire question, and option 2 exists solely to catch the swap. Sanity-check by extremes: with \(T=1\) the space should be exactly \(|V|\), and only \(|V|^T\) gives that. The AN twin uses \(|V|=50000,\;T=20\), answer \(50000^{20}\).

Why this matters beyond the mark: \(32000^{25}\approx10^{112}\) trajectories is why you cannot enumerate, cannot compute the exact expectation, and must estimate \(\nabla J\) from a handful of sampled roll-outs. The astronomical size of this space is the reason for the whole variance-reduction apparatus — baselines, advantages, clipping.

Verbatim in both papers. Each distractor is instructive:

• Pointwise max (2) is the theoretically motivated constraint but is intractable — you would have to bound the KL at every state in a space of \(10^{112}\) contexts. TRPO deliberately relaxes it to an average, which is estimable from the sampled batch.
• Squared KL (3) is not a thing; and note it also reverses the argument order.
• Euclidean distance on \(\theta\) (4) is the deep conceptual error, and worth understanding: a small parameter change can produce an enormous change in the output distribution, and a large one can produce none at all. Trust regions are defined in distribution space, not parameter space — that is the entire idea behind the word "trust region", and the reason plain gradient clipping is not a substitute.

Build the whole table first. All five sub-questions read off it.

\(\text{total}=1.80+(-0.64)+2.40=\mathbf{3.56}\).

The three cases in this one table are exactly the three cases that exist, which is why it is such an economical question:

• \(t=0\): ratio too high, advantage positive. Clipping bites — the answer 1.8 is strictly less than the unclipped 1.95. This is the clip doing its job.
• \(t=1\): ratio exactly on the boundary. \(0.8\) is inside \([0.8,1.2]\), so \(\tilde r=r\) and both terms are \(-0.64\). The interval is closed — do not clip a value that sits on the edge, and do not be spooked by the negative advantage. Note the answer here is simply \(r_tA_t\).
• \(t=2\): ratio too high again, larger advantage, same treatment: \(1.2\times2.0=2.4\).

Two safeguards. (i) Take the ratio of the sampled action's probability only — not a sum over the distribution. (ii) When \(A_t>0\) and \(r_t>1+\epsilon\), the answer is always exactly \((1+\epsilon)A_t\), so you can write it down without computing \(r_tA_t\) at all.

(a)–(d) The full distributions are given, but the ratio uses only the sampled action — the first component:

Both ratios land exactly on a boundary, so clipping never bites and both answers are just \(r\!\cdot\!A\). This paper is testing whether you know the interval is closed — the reverse of the FN paper, where clipping bit twice. If you had wrongly clipped 1.2 down or 0.8 up you would still get the same numbers here, which makes it a forgiving question; the marks are in not panicking at the second components (0.64, 0.52), which are pure distraction.

(e) Now the second components matter, because a KL sums over the whole distribution:

Use the supplied logs to identify which terms you need: \(0.30/0.36=0.8333\) and \(0.70/0.64=1.09375\) are given, so those are state 1's two terms; \(1.25\) and \(0.7692\) belong to state 2 and are decoys here. Two checks: the weights are \(\pi_{\text{old}}\) (the first argument of the KL) — using \(\pi_\theta\) gives the reverse KL, a different number; and the result must be \(\ge0\) even though one term is negative. A tiny value like 0.0084 is exactly what you expect for two nearly-identical distributions — and it is the quantity TRPO would have constrained explicitly and PPO controls implicitly through the clip.

\(r=0.85\) lies inside \([0.8,1.2]\), so \(\mathrm{clip}(r)=r\), both branches equal \(-0.85\), and the \(\min\) is \(-0.85\).

The negative advantage is bait. Option \(-0.80\) is what you get by clipping \(0.85\) to \(0.8\) — which people do because "clip" feels like it should always fire, or because the negative sign makes them apply the clip in the wrong direction. Always check membership of the interval first; if \(r\in[1-\epsilon,1+\epsilon]\) the clip is the identity and there is nothing to do, whatever the sign of \(A\). Option \(-1.00\) forgets \(r\) entirely.

Take the difference first — it is the only input.

The shortcut worth knowing: \(-\log\sigma(\Delta)=\log(1+e^{-\Delta})\), the softplus of \(-\Delta\). One exponential, one log — no division. For \(\Delta=1.1\): \(\ln(1+0.3329)=\ln1.3329=0.2873\).

Three sanity properties that catch sign errors. The loss is always \(>0\). It decreases as \(\Delta\) grows — the model is more confident the winner wins — which is why the AN answer (\(\Delta=1.5\)) is smaller than the FN one (\(\Delta=1.1\)). And at \(\Delta=0\) it is \(-\ln0.5=0.693\), the maximum-confusion value: any answer above 0.693 means the model ranks the pair the wrong way round, so if your loss for a correctly-ordered pair exceeds 0.693 you have flipped the subtraction.

The supplied logarithm tells you the ratio before you compute it — \(\ln4\) means the ratio is 4, i.e. \(0.8/0.2\). Use that as a check that you have the fraction the right way up.

The sign is the whole question. The policy here assigns higher probability than the reference (ratio \(>1\)), so the log is positive and the penalty subtracts — the model is being charged for having drifted. Getting \(3.2\) instead of \(1.8\) means you added. Inverting the ratio to \(\ln(0.25)=-1.4\) would give a spurious bonus.

And note what the penalty is not. It is a per-sample log-ratio, not a full KL divergence — no sum over the vocabulary, no probability weighting. Its expectation under \(\pi_\theta\) is the KL, which is why the two formulations agree in expectation, but a single sampled log-ratio can even be negative (when the policy is less likely than the reference) whereas a KL never is.

Use \(r^*=\beta\log\frac{\pi^*}{\pi_{\text{ref}}}\), dropping the \(\beta\log Z(x)\) term (it is unidentifiable and cancels in every use of the reward):

Recognise \(\ln10=2.3026\) on sight — a tenfold ratio is the setter's favourite because \(0.1\times2.3=0.23\) lands cleanly. Option \(0.10\) is \(\beta\) itself (forgetting the log); \(0.45\) is \(\log_{10}\)-based arithmetic gone astray; \(1.00\) comes from \(\log_{10}10=1\) without the \(\beta\). Natural logs unless stated otherwise.

What the number means: the aligned policy is 10× more likely than the reference to produce this response, and at \(\beta=0.1\) that is scored as a reward of 0.23. Turn \(\beta\) down and the same behavioural change implies a smaller reward — because a weaker leash means less reward is needed to explain the same drift.

\(\sigma(4.0-4.0)=\sigma(0)=\frac{1}{1+e^{0}}=\frac12\). Two seconds of arithmetic — but the concept it tests is the most important property of the DPO gradient, and worth stating properly:

• The weight is the model's own error probability, so DPO is self-annealing. Pairs the model already ranks correctly (\(\hat r_w\gg\hat r_l\)) get \(\sigma(\text{large negative})\approx0\) and contribute almost nothing.
• Pairs it ranks wrongly get \(\sigma(\text{large positive})\approx1\) and dominate the update.
• Indifference sits exactly in the middle at \(0.5\) — maximum uncertainty, half weight.

Do not mistake "indifferent" for "no gradient". Option \(0.0\) is the trap for reasoning "equal rewards, so nothing to learn". The opposite holds: equal rewards mean the model has not yet learned this preference, so the example is highly informative. Zero weight arrives only once the preference is confidently correct.

Multiply, then exponentiate — that is the entire question, and the formula is given. The AN version is \(e^{-1}\) exactly, worth recognising as \(0.368\) without a calculator.

Two checks that make this error-proof. Since \(A<0\) (the stem says "representing decay") the exponent is negative, so \(\bar A\in(0,1)\) — any answer above 1 means a dropped minus sign, and it would also mean an unstable recurrence in which \(x_k=\bar A^kx_0\) explodes. And \(\bar A\) is the per-step memory-retention factor: \(0.472\) means each step keeps 47% of the previous state, so the model's memory decays with a half-life of about one step. Larger \(\Delta\) or more negative \(A\) = shorter memory.

(a) is the harder of the two, because you must read the densities out of a word problem.

The two readings that decide it: "three times as many cars as bikes" makes the real distribution \(3\!:\!1\), i.e. \(0.75\) on cars — not 3, and not \(1/3\). And "produces only car images" makes the generator's density on cars \(1\), not \(0.75\). The answer is below \(0.5\), and that is the meaningful check: the generator over-produces cars relative to the real distribution, so a car is more likely to be fake than real, and an optimal discriminator must say so. Anyone who answers \(0.5\) has assumed convergence \(p_g=p_{\text{data}}\) — which the question has deliberately broken.

(b) \(D^*(x)=\frac{1/4}{1/4+3/4}=0.25\) is the probability of being real. The question asks for generated, so take the complement: \(1-0.25=0.75\). Equivalently \(\frac{p_G}{p_{\text{data}}+p_G}=\frac{3/4}{1}=0.75\). Underline the word "generated" or "real" before computing — the two papers ask opposite directions in consecutive terms, and both 0.25 and 0.75 are exactly-matched answers, so there is no partial credit for the complement.

Compute the two weighted densities, then divide. Do not normalise anything early.

Three traps, all of which appear here by design. (i) \(\sigma_2^2=9\Rightarrow\sigma_2=3\), so the normaliser is \(\frac{1}{3\sqrt{2\pi}}\) and the exponent denominator is \(2(9)=18\). (ii) You cannot drop the \(\frac{1}{\sqrt{2\pi\sigma^2}}\) prefactor, because the two components have different variances so it does not cancel — here \(C_2\) is three times wider and therefore three times flatter, which is most of why the answer is near \(\tfrac12\) despite \(\pi_1\) being much larger. (iii) The answer is a ratio, so no \(2\pi\) arithmetic is wasted if you keep both terms in the same form.

Why both answers sit just above 0.5: the point is placed roughly equidistant in standardised terms between a tall narrow component and a short wide one. The setter tunes it so the two weighted densities nearly tie — which means a small slip in either exponent moves the answer outside the accepted band. Carry four decimals.

The two false options are false for the same underlying reason, which makes them easy to dispatch together: both misattribute a property of the reconstruction term to the KL term.

• VAE blurriness comes from the reconstruction likelihood — a Gaussian \(p_\theta(x|z)\) makes the objective an L2 loss, and L2's optimum is the mean of all plausible images, which is a blur. Latent sampling contributes noise, not blur. (The deeper cause is maximising likelihood at all: it penalises missing a mode far more than covering a bad one, so a VAE hedges. A GAN's discriminator penalises implausibility directly, which is why GANs are sharper but drop modes.)
• The KL term is a regulariser on \(z\), pulling \(q_\phi(z|x)\) toward the prior. It does the opposite of improving sharpness — that is precisely the trade-off \(\beta\)-VAE exposes, and the subject of the next question.

Option 5 is the sharpest thing on this page. Optimising the KL alone is minimised by \(q_\phi(z|x)=p(z)\) for every \(x\) — the encoder ignores its input entirely and the latent carries zero information. That is posterior collapse, stated as an optimisation fact rather than a phenomenon: the reconstruction term is the only thing preventing it.

Reason from the loss, not from memory. This is a minimisation, and \(\beta\) multiplies the KL term. Raising \(\beta\) makes KL violations more expensive, so the optimiser buys KL reduction by spending reconstruction quality. Hence the posterior is pushed toward the prior and reconstructions degrade — the two effects always move in opposite directions, which immediately eliminates options 3 and 4 (both of which have them moving together).

In the limit \(\beta\to\infty\) you reach exactly the posterior collapse of the previous question: \(q_\phi(z|x)=\mathcal N(0,I)\) for all \(x\), zero KL, and a decoder that has no information to reconstruct from. In the limit \(\beta\to0\) you get a plain autoencoder with sharp reconstructions and an unusable, unstructured latent space. \(\beta\) is the dial on the rate–distortion trade-off, and its selling point is that intermediate values buy disentangled latents at a modest reconstruction cost.