Vamshi,
Restricting a specific field in an Infotype to be viewable only for one set of user while not viewable by another set of users is possible. However, it is not done through security settings, specially for Infotypes where sub-types are not available. You can explore extending Infotypes e.g. 0002 to include subtypes to gain more control but such enhancements need to properly evaluated.
One non-security related resolution for your scenario is to explore editing the table T588M in SM30. Here you can edit fields and use MOLGA+PSA combinations to restricting your user populations to separate PSAs and there by separate field view capabilities.
Regards,
Shivraj Singh