Not sure what you mean by "within the same engine". An engine swap is using an engine design family that was never offered in the car.
The example that was used for a long time was a 350 Chevy in a '57 Chevy is not a swap however a 454 Chevy in a '57 Chevy would be. If a '57 Chevy originally had a V8 in it it was either a 265 or a 283 but it was of the "small block Chevy" design none the less. A '57 Chevy never came with the 454 "big block Chevy" design. So, you can run you standard "small block Chevy" in any displacement you want and it's not an engine swap so long as you stick to the same engine design. Once you go to the "big block Chevy" design then you've made a swap because a '57 Chevy was never offered with a big block motor. This is what allows you to run different engine classes in Production ie a 427 cid small block chevy can run B/PRO where a 427 cid big block chevy would have to run as B/GC because a BBC would be a swap assuming we're still talking about the same car.
So, it looks like you want to run a 5.4L motor. If the engine design between the 4.6L and 5.4L is basically the same then it won't be considered a swap. Further, if the 5.4L is just a "punched out" version of the 4.6L then it's not a swap. I don't know anything about mod motors but if the 5.4L and 6.4L motors share the same head bolt pattern, bellhousing bolt pattern, oil pan bolt pattern, etc, then they're the same family and would not be considered a swap. A Windsor motor in a mod motor car would be a swap as would the new 5.0 Coyote motor.
Hope this helps.