Yes, the high spots are hindering you a bit. I mentioned stretching a wire over the surface just to see how bad things really are without even doing sanding work. This gives you a better idea on plan of attack to be done both best & quickest route without spending time & money applying product & sanding. I probably wouldn't bother trying to tap up on the low spots unless they are pretty bad, you could end up moving metal elsewhere.. Metalworking can be a pretty tedious skill to get near perfection without filler & if you were that damn good at it you'd be telling us how to do it , right?
From what I see here, basically you wasted a little time & money at this point with the high build primer being the level of flattening needed. High build primer shouldn't be overused where filler is needed, as it can shrink a little over time, especially from over use. A skimcoat of filler over the whole area would have been a better start off here, then switching to a high build type primer. For the door started on here, I'd first hammer & dolly on the worst high spots, then sand & or file, spot epoxy, & skimcoat the whole thing with a good smoothing filler such as evercoat metal glaze. Also start out with a longer sanding stick & cut through the first part with coarser paper & light pressure to give the best straightening effect, then switch to finer paper once you're touching everywhere. If you're still seeing untouched areas when most of the filler is looking pretty well sanded, stop & skim the low a little more first.
Now , for the next door, start out better with the metal before epoxy & skimcoat first with filler if it seems to need it.
Not trying to be critical, just trying to help you get there easier in the long run.
EDIT;;;; Just noticed you posting on the evercoat poly primers. These are ok in place of regular filler, though I'd still skim some regular over the worst low spots if needed. 2k primers is what you don't want to overuse in place of actual filler, was thinking you meant 2k products..