Have a group of users who can only access one of several pipelines

Hi all

I'm looking for a way to achieve the following:

I want to achieve the following: I have pipelines A and B. I want some of my users to see everything for all deals in both pipelines. Then I want another group users who can only see the deals (with all their details, including notes) in pipeline B. 

Is that possible to achieve somehow?

I can achieve something halfway userful with visibility groups. However, with two drawbacks:

a) The users with access to pipeline B can see the deals and their details but not the corresponding contacts involved. I can live with that.

b) The users can also not see the notes with deals in pipeline B. That is a showstopper for me.

Any ideas?

Thank you!

myke.