6GB, does this mean you are the PowerCLI snapin in a 64-bit PowerShell engine ?
i had occasional issues with PS v3 as well.
Since restarting the session fixes the problem temporarily it could be a memory leak issue, but if that is due to PowerCLI or PS v3 itself is not clear to me.