Can we identify the parameters of a neural network by probing its input-output mapping? Usually, there is no unique solution because of permutation, overparameterisation and activation function symmetries. Yet, we show that the incoming weight vector of each neuron is identifiable up to sign or scaling, depending on the activation function. For all commonly used activation functions, our novel method 'Expand-and-Cluster' identifies the size and parameters of a target network in two phases: (i) to relax the non-convexity of the problem, we train multiple student networks of expanded size to imitate the mapping of the target network; (ii) to identify the target network, we employ a clustering procedure and uncover the weight vectors shared between students. We demonstrate successful parameter and size recovery of trained shallow and deep networks with less than 10\% overhead in the neuron number and describe an `ease-of-identifiability' axis by analysing 150 synthetic problems of variable difficulty.