I've noticed that memory is still being held by the system, even with no apps running.

This is PCLinuxOS 2010 KDE running in VirtualBox 3.1.6 on a Mac OS X 10.6.3 host.
I did a boot recently, did a 'free' - saw about 480Meg (out of 1500 Meg allocated to the VM) in use and no swap space in use. I've run a few apps (FireFox, Synaptic, a few games) where I do my usual thing (open it, do my work in the app, close it). I also minimize the VirtualBox window when I'm not using it (and need to run my Mac apps).
After doing some of the above stuff for awhile, I did another 'free' - and noticed about 1400Meg in use and about 470Meg of swap space (out of 3000Meg allocated for swap). No apps are running, so that memory usage looks awfully high.
Is this normal behavior? I'm concerned that resources aren't being released as they should.