My computer has only 4 GB of RAM and when I build (amd64) I usually have firefox and google chrome open at the same time, plus I am usually either straming music with Amarok or watching some MOOC videos. My swap memory is hardly used, just occasionaly.
So perhaps there is something else going on, Michael.
cheers Michele