ReLU networks are a standard example of neural networks. We shall discuss the expressivity of neural networks with a focus on their depth. The plan is to start with a general introduction and a survey of basic results, including the duality between ReLU networks and a model for constructing polytopes. We shall then discuss polytopes and see how their properties can help to understand the depth of neural networks. Based on joint works with Bakaev, Brunck, Hertrich and Stade.