Advanced architectures

Graphs, external memory, temporal alignment: beyond the transformer

In this chapter, you will learn to

• Process graph-shaped data
• Give a network an external memory
• Align sequences of different lengths, and train very deep networks

6 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   Not all data are sentences or photos. Roads, molecules, recorded voices: each has its own shape, and the architecture has to fit that shape. - Graphs: weather and travel times. GraphCast, the DeepMind model…

2. The hop count

   You administer a mesh network of radio relays. Most relays merely forward packets; only a few, the gateways, are connected directly to the backbone, the heart of the network. A packet emitted by any relay must…

3. The swarm's mesh

   You join a team operating a swarm of reconnaissance drones deployed over an area. Each drone holds a position on the ground and continuously transmits on a given frequency band. Two drones can establish a direct radio…

4. The network's scratchpad

   You are designing a small controller network that processes a long series of information, but it can hold only a handful of numbers in its own activations at a time. It cannot retain hundreds of cross-referenced facts…

5. A hundred layers without flinching

   You are pushing your networks ever deeper. A shortcut connection has already let you train a few dozen layers where a bare stack stayed stuck. You now aim much higher: a hundred blocks stacked one on top of another.…

6. The hesitant transcriber

   You are developing a speech recognition system. A recurrent network scans a sound clip cut into T very short time steps and, at each step, produces a probability distribution over a small alphabet. This alphabet…