"assuming the null hypothesis is true"

This is the crucial caveat. The p-value is computed in a hypothetical world where the null is true. It doesn't tell you the probability that the null *is* true.