Hi,
you probably already understand this but I want to put stress on this. In case of UI5 you can't trust client. So for example if you a have a service that gives you PO details and it is consumed by UI5. You want to hide some fields (e.g. pricing) based on user's authorization. You can't send all data to client and then hide some fields based on user's authorization in UI5. You have filter out data in backend service if a user does not have authorization for them. Same is true for authorization checks for various activities. So your UI5 team should read info about user's authorization to hide the fields/buttons that are not available for a user, But even if they don't do it the user won't see any values in these fields because the backend service won't send any data.
Regarding your original question. If you want to have a really granular access control then you have to pay for it. There is no magic trick to avoid this. You can split your fields into multiple groups and assume that if field does not have a group then it's displayed to every users regardless authorization. This should minimize number of records. Also lower number of groups means less maintenance.
Cheers