Quiz 1, decoded: the adversarial half of GenAI.

Format

• Question types: MCQ single-answer · MSQ multi-select · NAT numeric answer. Marks per question range 1–4.
• Split: roughly half concept-recall (know the exact formula / which statement is false) and half short numeric computation (a trace, a small-distribution divergence, a layer-size product).
• Style: heavy, precise LaTeX notation. It rewards exact recall (the exact objective, the exact conjugate, the exact optimal discriminator) far more than hand-waving.

Topic frequency across the 3 papers

Combined GenAI questions, binned by theme. Longer bar = it shows up more, so it earns more of your revision time.

We are given data \(D = \{x_{1},\dots ,x_{n}\}\) drawn i.i.d. from an unknown distribution \(P_{x}\). The goal of generative modelling is to estimate \(P_{x}\) and learn to sample from it — not to classify, but to synthesise new points that look like they came from \(P_{x}\).

We never observe the densities \(P_{x}\) or \(P_{\theta }\) — only samples from each. So the entire game is: rewrite the divergence as expectations, which the Law of Large Numbers lets us estimate from samples. That single move — divergence → expectation → sample average — is the spine of the whole course.

The generator \(g_{\theta }(z)\) turns cheap noise into data. A second network — the discriminator / critic — measures how far \(P_{\theta }\) is from \(P_{x}\). They are trained adversarially: the critic tries to tell real from fake, the generator tries to fool it. When the critic can no longer win, \(P_{\theta } \approx P_{x}\) and \(g_{\theta }\) has become a sampler for \(P_{x}\).

Data, distributions, and the generator

• \(x_{i} \in \mathbb{R} ^{d}\) — each datapoint is a high-dimensional vector. E.g. a 400×400×3 image ⇒ \(d = 480,000\).
• Samples are independent across examples (\(x_{i} \perp x_{j}\)), but the dimensions within one \(x_{i}\) are not independent (pixels are correlated).
• The generator \(g_{\theta }: Z \to X\) maps a known latent \(z \sim \mathcal{N} (0,I)\) to data space. Its output \(\hat{x} = g_{\theta }(z)\) has an implicit density \(P_{\theta }\) we can sample from but cannot write down.

The one theorem you compute with: Monte Carlo

Any integral against a density is an expectation, and any expectation is a sample average in the limit. This is why "we only have samples" is not a blocker.

Definition & properties

The key exam fact: different choices of \(f\) give different classic divergences — so different generative models. Memorise this table; it is directly asked every semester.

The convex-conjugate trick → variational lower bound

The problem: the integrand contains the ratio \(P_{x}/P_{\theta }\), which is not an expectation of any single random variable. The fix is the Fenchel (convex) conjugate, which rewrites a convex \(f\) as a supremum of linear functions:

Substituting and replacing the pointwise optimum with a function \(T(x)\) (a neural net can only search a restricted class, so equality becomes ≥) yields the punchline of Week 1:

Drill it — f-divergence questions

For GANs we take \(f(u)=u \log u - (u+1)\log (u+1)\), whose conjugate is \(f^{*}(t)=-\log (1-e^{t})\) on \(\operatorname{dom} f^{*}=\mathbb{R} ^{-}\). Choosing the activation \(\sigma _{f}(v)=-\log (1+e^{-v})\) to keep the critic's output in that domain, the value function becomes exactly:

The optimal discriminator — memorise these three lines

Drill it — GAN objective & D*

The alternating loop

• Scheduling is a knob: 1 D-step + 1 G-step, or k D-steps per G-step, etc. A training loop that runs the discriminator update k times inside the loop means the discriminator is updated multiple times per generator update.
• Failure signal: if the discriminator loss collapses to ≈0 while the generator loss shoots up, the discriminator has become too strong — the generator's gradient has vanished.
• Stop when generated-sample quality is good enough; there is no clean "loss went to zero" stopping rule.

GAN as a classifier-guided sampler — and the trap

Intuition: if a classifier \(D_{\omega }\) can no longer tell real from generated, maybe \(P_{\theta }=P_{x}\). Tweak θ until the classifier fails.

DCGAN — Deep Convolutional GAN

Typically \(z \in \mathbb{R} ^{k}\), \(x \in \mathbb{R} ^{d}\) with \(k \ll d\) — the generator must expand dimensionality. A plain GAN uses fully-connected layers then reshapes to an image. DCGAN uses transposed (up-)convolution layers to grow the latent straight up to image size \(\mathbb{R} ^{r\times c\times 3}\) — no reshape needed, and spatial structure is respected throughout.

Conditional GAN (cGAN)

Vanilla GANs give no control. With labelled data \(D=\{(x_{i},y_{i})\}\sim P_{\mathrm{xy}}\) we instead sample from the conditional \(P_{x|y}\). Feed the label \(y\) (one-hot or a text embedding) into both networks:

Counting generator parameters (a guaranteed NAT template)

For linear layers \(a \to b \to c \to \dots\) excluding biases, parameters = sum of consecutive products \(a\cdot b + b\cdot c + \dots\)

Why training saturates: manifold hypothesis + perfect discriminator

• Manifold hypothesis: real data (images) lie on a low-dimensional manifold inside the huge ambient space \(\mathbb{R} ^{d}\). A random binary image is almost never meaningful (the "monkey-at-a-typewriter" problem).
• So the supports of \(P_{x}\) and \(P_{\theta }\) almost surely do not overlap.
• Perfect-discriminator theorem: with non-overlapping supports a discriminator can reach 100% accuracy ⇒ \(D_{f}\) becomes independent of θ ⇒ the generator gradient vanishes. Training saturates.

Wasserstein / Earth-Mover distance

Think of \(P_{x}\) and \(P_{\hat{x}}\) as piles of dirt. A transport plan \(\gamma (x,\hat{x})\) is a joint distribution (a table whose row-sums give one marginal, column-sums the other) telling you how much mass to move from each \(x\) to each \(\hat{x}\). The cost of a plan is the average distance mass travels; Wasserstein picks the cheapest plan.

Making it trainable: Kantorovich–Rubinstein duality

The primal min over all transport plans is intractable. Duality rewrites \(W_{1}\) as a maximisation over 1-Lipschitz functions — which a neural "critic" can represent:

Enforcing the 1-Lipschitz constraint — three methods, ranked

Drill it — Wasserstein & transport

Why invert?

• Feature extraction: the recovered \(z\) vectors are compact features for downstream tasks.
• Editing: to edit an image, invert \(x_{i} \to z_{i}\), apply \(z_{\mathrm{edit}}=f_{\mathrm{edit}}(z_{i})\), then decode \(x_{\mathrm{edit}}=g_{\theta }(z_{\mathrm{edit}})\).

Three ways to make a GAN invertible:

① Bidirectional GAN (BiGAN)

Add an encoder \(E_{\phi }: X \to Z\). The twist: the discriminator no longer judges images alone — it judges (data, latent) pairs, learning the joint statistics.

② Latent-regression inversion

Simpler: keep an ordinary GAN discriminator and bolt on an encoder trained to regress the latent back:

③ Inversion via optimisation

For a single target image and a frozen generator, just optimise the latent directly: \(z^{*} = \operatorname*{arg\,min} _{z} \| x - g_{\theta }(z)\| _{2}^{2}\).

Drill it — inversion

Domain shift: we have labelled source data \(D_{S}=\{(x_{i},y_{i})\}\sim P_{S}\) but must perform at test time on an unlabelled target \(D_{T}=\{\hat{x}_{j}\}\sim P_{T}\) with \(P_{S} \ne P_{T}\) (e.g. the PACS dataset: photo → sketch → cartoon → art). A vanilla classifier trained on \(P_{S}\) fails on \(P_{T}\).

Unsupervised Domain Adaptation (UDA): we want features that are good for classifying and look the same across domains. Three networks share the job:

• Feature extractor \(\phi\) — turns any input (source or target) into a feature vector \(f\).
• Label predictor \(h_{\psi }\) — reads a feature and predicts its class \(y\); trained only on the labelled source.
• Domain discriminator \(D_{\omega }\) — reads a feature and guesses which domain it came from (source or target).

The core trick: train \(\phi\) so the domain discriminator can't tell the two domains apart. If source and target features look identical, a classifier trained on source features automatically works on target features too.

Procedure: push real data \(D_{\mathrm{true}}\) and generated data \(D_{\mathrm{gen}}\) through a pretrained Inception-v3 (trained on ImageNet); take activations from the penultimate layer; fit a Gaussian to each set (mean + covariance); then measure the closed-form Wasserstein-2 distance between the two Gaussians.

Drill it — FID

① The variational lower bound of an f-divergence

The engine of the whole f-GAN / VDM framework. It converts an intractable integral over a density ratio into a supremum of two expectations — which the Law of Large Numbers lets us estimate from samples.

② The optimal discriminator → Jensen–Shannon divergence

The classic result that explains what a GAN actually minimises. Two parts: first solve for the best discriminator, then substitute it back into the value function.

③ Two quick ones worth memorising

• The 3-step recipe: family → divergence → argmin
• \(f(u)\) for KL, reverse-KL, TV — and \(f^*(t)=e^{t-1}\) for KL
• Wasserstein is NOT an f-divergence
• Variational bound is a lower bound (any T works)
• GAN objective with sigmoid D — write it blind
• \(D^* = P_x/(P_x+P_\theta)\); value \(= -\log 4 + 2\cdot\mathrm{JSD}\)
• Disjoint supports ⇒ value = 0, gradients vanish
• Non-saturating loss \(-\log\sigma(a)\) beats minimax
• Manifold hypothesis + perfect-discriminator theorem
• Wasserstein primal (transport plan) & K–R dual
• Weight clipping FAILS; GP / spectral-norm work
• Multiple critic steps per generator step, and why
• p.data.clamp_(-0.01, 0.01) is the in-place clip
• Backward accumulates grad → need zero_grad()
• BiGAN judges (x, E(x)) vs (G(z), z) pairs
• Inversion by optimisation: \(z^* = \arg\min\|x-g(z)\|^2\)
• GRL multiplies gradient by \(-\lambda\)
• FID formula + 1-D shortcut; lower is better
• FID features: penultimate layer of Inception-v3
• Param count: \(\Sigma\) of consecutive layer-size products