Abstract: Plasmonic nanostructures and -devices are rapidly transforming light manipulation technology by allowing to modify and enhance optical fields on sub-wavelength scales. Advances in this field rely heavily on the development of new characterization methods for the fundamental nanoscale interactions. However, the direct and quantitative mapping of transient electric and magnetic fields characterizing the plasmonic coupling has been proven elusive to date. Here we demonstrate how to directly measure the inelastic momentum transfer of surface plasmon modes via the energy-loss filtered deflection of a focused electron beam in a transmission electron microscope. By scanning the beam over the sample we obtain a spatially and spectrally resolved deflection map and we further show how this deflection is related quantitatively to the spectral component of the induced electric and magnetic fields pertaining to the mode. In some regards this technique is an extension to the established differential phase contrast into the dynamic regime.