I believe you might want to double check your phone and/or headphones. Or even the Deck itself.
I use it daily and apart from the expected latency, the audio is great with no flaws. No interruptions or jank.
Bluetooth 4.0 is capable of handling around 8 connections simultaneously
Bluetooth controllers actually do behave weirdly on the Deck, the polling rate is sub optimal. You can modify certain system files to help, but you’d have to reset the setting after every update.
But for audio, it’s mostly fine. I really do not have issues using this phone to Deck setup. You could consider a full system reset if it’s that bad on your end, it’s supposed to work well.