You say the boys live separately - I’m reading this that the boys do not live together in the same cage. If this is the case then it is ok for you to bring the girls downstairs. It’s if the boys live together in one cage that you can’t bring the girls downstairs. The risk isn’t to the girls bond, it is to the boys bond (you should not bring sows into a boar room as it will cause the boars to fight with each other), but if the boys do not live in the same cage, then the boys will obviously not be able to fight with each other.
However, bringing the girls downstairs is still going to cause a reaction in each of the single boys. You must make sure the boys cages are both secure so they can’t escape their cages to get in with the girls (it can and does happen).