Well, there are of course many examples but perhaps the Dirac $\delta$-function is the most striking one. With quite some violence to the math of usual calculus, Dirac's idea turned out to be exremely useful: the "function" everywhere zero and very high at $0$ so that the integral is $1$. Applications are ranging from Fourier analysis, (linear) PDE, quantum mechanics, and many more. Now of course we know how to make things rigorous in the framework of Schwartz's test function and distribution theory. But the intuition is clearly from physics.

Similar and also in this framework is the notion of oscillatory integrals...