I had this same problem a couple of days ago. This is what I did to solve it. I am using transmission as my BT client but use Firefox as my browser. These steps may help you.
In Firefox, under file > preferences, click on the applications tab. Scroll down to "magnet" and select to "always ask". Then click on the magnet file and it will ask you which program to use to open it. In transmission, I had to manually "browse" to the program in /usr/bin and select it for it to work. I guess this is because firefox recognized "transmission" but when I browsed to the actual file, the file name was transmission-gtk and not merely "transmission". At any rate, after I did this, it worked fine.
Hope this helps.
Desktop: AMD 3.6ghz X4, Gigabyte MB, 8GB ram, Multiple PCLOS's, 184.108.40.206-pclos3.pae.bfs
Server: Intel 3.4ghz X2, 4GB RAM, SATA, gnome2, *.pae.bfs, Headless, Servers: LAMP, FTP, Samba, Print, PXE, BT
HTPC: Phenom II X4 2.8ghz, 4GB Ram, 1TB SATA, PCLinuxOS-KDE, NVIDIA N220GT HDMI, *.pae.bfs