Right, exactly what I do also. It seems either the configurations in .xine or possibly .pulse mess up the audio sometimes, especially if they're kept over several upgrades (KDE3 => KDE4?).
But recreating a clean home directory, and also completely cleaning out the /tmp folder, have solved problems for me.
Fingers crossed for you, CheeseQueen452!