What you are describing is very common in terms of rough play or what is better known as 'playful aggression'. In these types of interactions there are times when it might appear that one child is being singled out but this is in fact purposeful and evident in all mammalian species including humans.
Over time what will happen is roles will change and your son may appear as one of the aggressors...it is this form of reciprocation that is so important and also a sign if things are not going well...if your son never gets to be the one instigating the playful aggression then this type of play is not positive and should be stopped or mediated.
If, on the otherhand, he is happy to be the one on the ground as you suggested given he became very upset by your husband's intervention then you should leave him and the others to their own devices. It may sound counterintuitive and you may want to interject but best to pause and let the boys sort things out...boys are very good at being able to tell when someone is not happy within the dynamics of playful aggression...also, all the boys have a vested interest in ensuring everything is positive...as long as everyone is happy, then the chances of more playful behaviour are good. To that end the best advice I can give is to be cautious in looking at those activities with the eyes of a protective adult and allow your son to mediate the play and let you know if things are not going well, and he will!