Is there a way to restrict visibility of assets for certain users/groups?

Yes, you can enable Targets in your Media Library. This means that you can define the visibility of your assets for different departments, markets, and more.

Note: this is only available for our Enterprise customers. See here or here to learn more.