I have no such problem ....... I am using PulseAudio and turn off the HDMI in pavucontrol and so it never gets used.
I am certain there are other methods to do this without installing Pulse Audio ...... but i like my GUI controls

maybe look at setting your default audio card to something other than the HDMI ....