Hmm, it's a difficult issue. Rigorously learning the fundamental mathematics may not be very interesting and it may appeal to a very small segment of students as it might seem like it has no 'real' applications, but if you do slog through it then the physics that uses it becomes much easier.
There's a big difference between studying Applied Mathematics and studying Theoretical Physics. The former will involve a
lot of new mathematical objects and learning how these mathematical objects transform and interact with other mathematical objects.
Differential geometry is a perfect example, as you have to learn the different sorts of linearity and which objects exhibit which sort of linearity. For a simple example, the differential operator del
This behaves quite unlike a lot of other operations you might come across. And you also have even weirder graded Leibniz rules.
You can use knowledge of differential geometry to use differential
forms. This is differential geometry that is
independent of co-ordinate systems, and is incredibly useful and simplifies many problems in General Relativity. But the fundamental knowledge of the rules of differential geometry is essential before you can study any interesting problems of rotating black holes or anything like that.
Applied Mathematics uses abstract mathematics to solve abstract problems. Theoretical Physics uses abstract mathematics to solve physical problems.
There is some overlap in what people might be interested in, someone studying string theory or quantum gravity theories may well be interested in various applied mathematics problems too, but a condensed matter physicist will probably have little interest in the abstract problems and be more concentrated on phenomenology.
My favourite mathematical object is the quaternion, I love those guys, but I've never found any use for them.