Restrict Audience Directory to Specific Audience Segment

Just to add a use case for Audience Directory; I might want everyone to be able to see the directory but only a certain segment to be IN the directory.