2 to 4 are more than enough and if you monitor that partition you will see that it is not used with all that ram
i have 1280mbs and haven't seen activity on it on many months, when i had 512 mbs the swap was merely used up to 200 mbs or less, usually it used 70, 30 or less
as blackbird says is a safety net and under normal circumstances can be seen as a waste of hard disk space