I'm in full agreement that the 3gb hardware limitation of ram defeats much of the 64bit advantages, but the point I'm making is there are other limits besides the max ram that are improved with x86-64 architecture. It's just far superior in many ways.
It depends heavily on your workload.
amd64 binaries are going to have the benefit of way more registers... but they're also going to have the overhead that comes with 64-bit addressing. So as to whether you'll see a speed-up or a slow-down for your workload depends on exactly what you're doing.
My general recommendation is that if you have < 4GB of RAM, run 32-bit, if only for the memory savings. But hey, if you benchmark your code and figure out that amd64 > x86, by all means use it!
Need help with Linux or FreeBSD? Catch me on IRC: I'm ThinkRob on FreeNode and EFnet.
Current laptop: none
Current workstation: IBM Intellistation 9228 (running FreeBSD 9.1) - blackbird