You won't be able to save passwords in private mode... but otherwise there are few drawbacks... So here's how...
Bring up "about:config"
Set "browser.privatebrowsing.autostart" to "true"
Restart FF, and you should be running in private mode (private browsing options will be grayed out under Tools)