M dwarfs are prime targets in the hunt for habitable worlds around other stars. This is due to their abundance as well as their small radii and low masses and temperatures, which facilitate the detection of temperate, rocky planets in orbit around them. However, the fundamental properties of M dwarfs are difficult to constrain, often limiting our ability to characterize the planets they host. Here we test several theoretical relationships for M dwarfs by measuring 23 high-precision, model-independent masses and radii for M dwarfs in binaries with white dwarfs. We find a large scatter in the radii of these low-mass stars, with 25 per cent having radii consistent with theoretical models while the rest are up to 12 per cent overinflated. This scatter is seen in both partially and fully convective M dwarfs. No clear trend is seen between the overinflation and age or metallicity, but there are indications that the radii of slowly rotating M dwarfs are more consistent with predictions, albeit with a similar amount of scatter in the measurements compared to more rapidly rotating M dwarfs. The sample of M dwarfs in close binaries with white dwarfs appears indistinguishable from other M dwarf samples, implying that common envelope evolution has a negligible impact on their structure. We conclude that theoretical and empirical mass-radius relationships lack the precision and accuracy required to measure the fundamental parameters of M dwarfs well enough to determine the internal structure and bulk composition of the planets they host.