What age actually are they?
Were all four actually in the same cage?
Keeping two neutered males with two females would cause huge problems. You can only ever keep one neutered boar with any number of sows. Two boars with sows will not work out long term. So yes it is likely them having been around sows has caused the problem.
Equally, Once the sows were removed, the boys would have gone into a period of establishing as a pair but having been in the presence of sows and then going to a boar pair will be seen as a demotion in status. Boars don’t take kindly to that and once a boar has lived with a sow, he may never want to live with a boar again in the future
As I say, if it was a full on fight then bonding them back again, particularly if they are going to be put back in the presence of sows would not work. A bonded pair of boars always needs to be kept well away from a sow pair and never in the same cage.