Hand-Built Threshold MLP for a Triangle

The problem in brief

This problem asks you to build a tiny neural network by hand, without any training. You first write two PyTorch modules of your own: a step ("threshold") activation, and a linear layer whose random initialisation keeps the variance of the signal constant (which requires a short probability computation). Then, using only those two modules, you must assemble a multi-layer perceptron that outputs 1 for points inside the triangle with vertices $(0,0)$, $(3,1)$, $(1,5)$ and 0 outside. All weights are chosen by you, and gradients are switched off. The real lesson: what one neuron computes geometrically, and why two layers are enough to carve out a convex region.

How you are scored

Parts 7.1 to 7.3 are worth 5 points each, Part 7.4 is worth 15. The graders run your class on batches of points (input shape $(B, 2)$, output shape $(B,)$, values 0 or 1) and check the module restrictions: only My_Threshold and My_Linear, hand-set weights, requires_grad = False. There is no partial metric; the network must be right.

Our network agrees with an independent barycentric point-in-triangle test on 200 000 random points (100 %), and on the vertices, edge midpoints and points just outside each vertex.

Question by question

7.1 The threshold module

$\Theta(x) = 1$ if $x \ge 0$, else $0$. In PyTorch a comparison gives a boolean tensor, which we cast back to the input's dtype so it can flow into the next linear layer:

class My_Threshold(nn.Module):
    def forward(self, x):
        return (x >= 0).to(x.dtype)

(This function has zero gradient almost everywhere, which is exactly why nobody trains with it, and why this problem is inference only.)

7.2 The variance-preserving standard deviation

Let $y = \sum_{c=0}^{C-1} w_c x_c$ with all $w_c, x_c$ independent, zero mean, $\operatorname{Var}[w_c] = \sigma_w^2$, $\operatorname{Var}[x_c] = \sigma^2$.

For one product, independence and zero means give $\mathbb{E}[w_c x_c] = \mathbb{E}[w_c]\,\mathbb{E}[x_c] = 0$, hence

$$\operatorname{Var}[w_c x_c] = \mathbb{E}[w_c^2 x_c^2] = \mathbb{E}[w_c^2]\,\mathbb{E}[x_c^2] = \sigma_w^2 \sigma^2 .$$

The $C$ products are independent of each other (they involve disjoint independent variables), so their variances add:

$$\operatorname{Var}[y] = \sum_{c=0}^{C-1} \operatorname{Var}[w_c x_c] = C\, \sigma_w^2\, \sigma^2 .$$

Setting $\operatorname{Var}[y] = \sigma^2$ gives $C\sigma_w^2 = 1$, so

$$\boxed{\sigma_w = \frac{1}{\sqrt{C}}} .$$

A Monte Carlo check with $C = 64$ and $\sigma = 3$ gives $\operatorname{Var}[x] = 9.00$ and $\operatorname{Var}[y] = 9.01$. This is the "LeCun" initialisation; without it, each layer would multiply the variance by $C\sigma_w^2$ and a deep network's activations would explode or vanish geometrically with depth.

7.3 The linear module

$C$ is the number of inputs of the layer (in_features). The weight has PyTorch's usual shape (out, in), so the layer computes $y = xW^\top + b$:

class My_Linear(nn.Module):
    def __init__(self, in_features, out_features, bias=True):
        super().__init__()
        std = 1.0 / math.sqrt(in_features)
        self.weight = nn.Parameter(torch.randn(out_features, in_features) * std)
        self.bias = nn.Parameter(torch.zeros(out_features)) if bias else None

    def forward(self, x):
        y = x @ self.weight.T
        return y + self.bias if self.bias is not None else y

Wrapping tensors in nn.Parameter is what makes them show up in .parameters(), which we need in 7.4 to switch off their gradients.

7.4 The triangle detector

One neuron = one half-plane. A unit computing $\Theta(n_1 x + n_2 y + c)$ outputs 1 exactly on one side of the line $n_1 x + n_2 y + c = 0$. A triangle is the intersection of three such half-planes, one per edge. So we need:

1. a hidden layer of 3 threshold units, one per edge, each equal to 1 on the triangle's side;
2. an output unit that fires only when all three hidden units are 1 (a logical AND).

Finding each half-plane. For an edge from $P$ to $Q$, a normal vector is obtained by rotating $Q - P$ by 90 degrees; then we choose the sign so that the third vertex gives a positive value.

edge	line through the edge	value at the opposite vertex	inside condition
$(0,0)\to(3,1)$	$x - 3y = 0$	at $(1,5)$: $1 - 15 = -14 < 0$	$-x + 3y \ge 0$
$(3,1)\to(1,5)$	$2x + y - 7 = 0$	at $(0,0)$: $-7 < 0$	$-2x - y + 7 \ge 0$
$(1,5)\to(0,0)$	$5x - y = 0$	at $(3,1)$: $15 - 1 = 14 > 0$	$5x - y \ge 0$

(Check each line passes through its two vertices: e.g. $2\cdot3 + 1 - 7 = 0$ and $2\cdot1 + 5 - 7 = 0$.)

So the hidden layer is

$$W_1 = \begin{pmatrix} -1 & 3 \\ -2 & -1 \\ 5 & -1 \end{pmatrix}, \qquad b_1 = \begin{pmatrix} 0 \\ 7 \\ 0 \end{pmatrix}, \qquad h = \Theta(W_1 p + b_1) \in \{0,1\}^3 .$$

The AND gate. $h_0 + h_1 + h_2 = 3$ exactly when the point satisfies all three tests. With weights $(1,1,1)$ and bias $-2.5$, the output $\Theta(h_0 + h_1 + h_2 - 2.5)$ is 1 for a sum of 3 and 0 for a sum of 2 or less. Any bias in $(-3, -2]$ works; $-2.5$ sits in the middle of the gap, so it does not depend on how $\Theta$ treats exactly 0.

The left panel shows a nice fact: no point of the plane fails all three tests (the count is never 0), and the regions beyond each vertex pass only one. The AND threshold is what turns this "vote count" into a sharp region.

The module.

class My_Triangle(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden, self.out, self.step = My_Linear(2, 3), My_Linear(3, 1), My_Threshold()
        with torch.no_grad():
            self.hidden.weight.copy_(torch.tensor([[-1., 3.], [-2., -1.], [5., -1.]]))
            self.hidden.bias.copy_(torch.tensor([0., 7., 0.]))
            self.out.weight.copy_(torch.tensor([[1., 1., 1.]]))
            self.out.bias.copy_(torch.tensor([-2.5]))
        for p in self.parameters():
            p.requires_grad = False

    def forward(self, x):                                  # (B, 2)
        h = self.step(self.hidden(x))                      # (B, 3)
        return self.step(self.out(h)).squeeze(-1)          # (B,)

my_triangle = My_Triangle()

Two details that cost points if missed: squeeze(-1) turns the $(B,1)$ output into the required $(B,)$, and the weights are copied inside torch.no_grad() because in-place writes into a tensor that requires gradients are forbidden. Points on the boundary count as inside (the threshold uses $\ge 0$), and since all weights are small integers the boundary decision is exact in floating point for points with simple coordinates such as the vertices.

The key idea

A threshold neuron draws a straight line and answers "which side?". A first layer of such neurons cuts the plane by several lines; a second layer combines the answers with logic (AND with bias $-(m - \tfrac12)$ for $m$ inputs, OR with bias $-\tfrac12$). Any convex polygon with $m$ edges therefore needs $m$ hidden units and one AND; a union of convex pieces needs one more layer of ORs. This is the constructive heart of the universal approximation theorem, and it explains what trained ReLU or sigmoid networks do softly: their first-layer units are oriented hyperplanes, and later layers build regions out of them.

The full solution

solution.py contains the four parts, a Monte Carlo check of 7.2, and the verification of 7.4 against an independent barycentric test: writing $p - v_0 = \lambda_1 (v_1 - v_0) + \lambda_2 (v_2 - v_0)$, the point is inside iff $\lambda_1 \ge 0$, $\lambda_2 \ge 0$ and $\lambda_1 + \lambda_2 \le 1$. It also checks the output shape $(B,)$ and that no parameter requires a gradient.

Results

check	outcome
7.2: $\operatorname{Var}[y]$ with $\sigma_w = 1/\sqrt{C}$ ($C = 64$, $\sigma^2 = 9$)	9.01 (target 9)
7.4: agreement with barycentric test, 200 000 random points in $[-1,4]\times[-1,6]$	100 %
vertices, edge midpoints, centroid	all 1
points 0.01 outside each vertex	all 0
output shape, requires_grad	$(B,)$, all False
a tempting wrong design: output bias $-3.5$	outputs 0 everywhere
a tempting wrong design: output bias $-1.5$	also fires on the 2-test regions (wrong outside)

What to remember

• A threshold (or ReLU) unit is a hyperplane plus a side. Get the side right by plugging in a point you know is inside.
• Two layers carve convex polygons: one hidden unit per edge, then an AND whose bias sits between $m-1$ and $m$.
• Variances of independent zero-mean products multiply, and sums add them: $\operatorname{Var}[y] = C\sigma_w^2\sigma^2$, hence initialise with $\sigma_w = 1/\sqrt{\text{fan-in}}$.
• Mind the PyTorch plumbing: nn.Parameter to register weights, torch.no_grad() to set them, squeeze for the output shape.
• Verify a hand-built model against an independent method on many random points; it is the fastest way to catch a sign error on one edge.