Setting the "Auto-Delete Empty Dynamic Groups" function to "On" in a Perspective will automatically remove any groups in the Perspective that have no assets categorized in them (0 Assets).  This feature however will not remove empty groups if the Perspective  is referenced in a Policy.  This is done so that a Policy that is dependent upon a Perspective and its groups can operate properly. 

To test for dependencies simply click the "Trash-can" icon in a group as if you were manually deleting a group.  A list of dependent policies will be show in a dialog.  You may wish to remove the dependency in the policy and then perform any maintenance on the Perspective.