The "danger" is just that the more you have installed, the more likely you are to miss a dependency, so on a main install where you've got all sorts of stuff set up and working there's a high probability you'll miss something someone with a more minimal install will need as well.
It won't do any harm to build on a main install, but you have a greater chance of producing a broken package with a missing "BuildRequires:" line. Similarly, testing on anything more than the most minimum install means you are more likely to get away with a missing "Requires:" line.
It's about making sure you haven't already got something installed others might not have.