Access groups are configuring user-access context.
When you create a new user, you define "home group", which is also home access group for a selected user. If you create user with home group "Tvaika" all of his objects (policies / tasks, etc ...) will be located in this group. By default, it´s set to "all", therefore the reason why you need to remove the filter.
In permission sets, you define access to objects (object is in fact anything inside ESMC, including computers, tasks, policies). If you grant read only access for a user to group "Tvaika" and set "policies", they will be able to see them. In your case, you just misunderstood the filter. It´s not "assignment filter" (to which group it is assigned to) but "access rights filter".
