The OEM airbag steering wheel is 367mm in diameter. A 350mm wheel is just under 5% smaller, so the amount your hands have to move for the same angular rotation, for example a quarter turn, is similarly 5% less. It is this that makes it feel so much more direct. The weight will only make a tiny, probably unnoticeable, difference.
My EPS equipped Type S came with an airbag wheel and when I changed it, for a 350mm Momo tuner, I was also surprised how much effect it had
I agree that the 320mm (14% smaller) wheel will be too small and likely feel very (too) heavy.
Cheers
Mark