How much is your boy weighing now? He will hopefully put on some weight in the next few days as babies grow quickly. On average a healthy piggy weighs around 250g at 3 weeks old.
If he is ill or has other major issues or is under 200g, I would leave him with mum a bit longer, as he won't be able to make babies in that state and you can leave him a bit longer. If he is healthy, active and putting on weight well every day (please weigh daily), and if mummy is pushing him away and weaning him, then please separate at about three weeks or soon after, as he won't get much more milk anyway. Make sure that he has a cosy and a (safe) cuddly toy that are smelling of his mum to snuggle into at first in order to buffer the shock of a separation.
I would recommend to have him bonded with an older "uncle" boar under expert supervision at a good rescue; this ensures that he also has somebody to teach him the piggy ways and to socialise him. A good age gap will also go a long way towards keeping the relationship stable and the dominance clear when your little boy is going through the hormonal teenage months.
However, the key to any successful boar bond is character compatibility - it really makes all the difference if you can have your little boy meeting and bonding with likely boars under expert supervision; the fall-out risk is a LOT lower that way than if you just chuck together two youngsters (even if they are brothers). It is really worth travelling to a rescue further away for that kind of added security! Ideally, you find a rescue that offers residential boar bonding that offers this time consuming service (not all rescues can afford to); "speed dating" at a good rescue is the next best option.
Sadly, both your closest good standard rescues that we can vouch for are a bit further away, but still at a manageable distance, either in Barnsley or Keighley (they are both on the map of recommended rescues tha you have been given in a previous post). I can recommend them most warmly. You and your little boy will be in very safe hands there! I have rehomed from either rescue myself.
Boars are usually neutered once their testicles have fully descended and they are a good weight; that means a wait until 4-6 months. The success of a neutering operation very much depends on finding an experienced vet with a good track record in piggy neutering. Sadly, the risk of post complications and even death at worst is still far more common than it should be.
Even with a successful operation, your boy then faces another 6 weeks post op wait until he is 100% safe to go with sows. I have a baby from a supposedly safe over 5 weeks post op boar (not one of mine), just to prove that point!
If you are thinking going down the neutering route, it will take a lot of time and some very thorough research on your part in order to minimise all the risks. While I am a big fan of neutered boars living with sows, I'd rather see a little chap live happily in company much sooner!
PS:
IF there is really a fall-out (and with careful choosing and bonding, the risk should be well below 10%), your boy is then at the right age for neutering, so it could be used as Plan B.
Unoftunately, whichever way you decide to go, there is no absolute guarantee of success, but by going a bit out of your way, you can reduce any risks considerably.