I know it's late in the game to comment on this thread, but it may be useful for other Xfce users seeking an answer to the panel button rearrangement issue, until the moving problem is fixed.
1. go to ".config/xfce4/panel" directory in your Home folder.
2. Open the "panels.xml" file with a text editor, after first taking a backup of it.
3. Under the <items> section is the list of buttons, as they appear on the panel, left to right; each entry references a file in the ".config/xfce4/panel" directory ... just open it with a text editor if you need further information about the button.
4. Just delete an line/entry, e.g., <item name="launcher" id="10"/> if you don't want it to appear in the panel.
5. Move whole lines up/down to get the right sequence of buttons that you like.
6. If you add a launcher in the panel using the desktop applet and it isn't in the right place, use the above techniques to put it where it suits you.
7. SAVE the changed "panels.xml" file when you are finished. I suspect a log out, and then back in again, will reflect the new changes you have made ... can't remember if necessary.
Also, a big +1 to the technique mentioned to consolidate to one panel (from the default 2) mentioned above, e.g., reverse the appearance (top to bottom, bottom to the top area), then right-click, MOVE, items from the old top one (e.g., tasks) from the old top to the new bottom, then delete the unneeded panel.