improve visibility groups functionality


Hello,
This scenario is causing me issues with visibility groups, and it does not seem there is an easy solution to do it:
- some users, let's say group A, have to see all leads/deals/contacts
- some others users, group B, should only see their own leads/deals/contacts
- ownership of leads/deals/contact can change from someone in group A to someone in group B, or from someone in group B to group A
I tried to configure two visibility groups, one "group A", with a default visibility set to "Owner's visibility group", and a "group B", child of group A, with a default visibility set to "Owner only".
This works, as long as there is no ownership change. If the owner of a deal change from group B to group A, as the visibility is still "owner only", only the owner can see the deal, not all group A.
Is there a way to use the visibility groups only based on the "viewer", not on the "owner" ?
For example, is user is part of this group, he can see all deals/leads, and he's part of this other group, he can only see it's own deals ?
Thanks,
Mathieu