We are looking to utilize Public Groups to restrict access to particular DocGen Packages to certain Nintex users beyond System Admins; however, Nintex has confirmed that the security setting is not working as expected.
Per Nintex:
Unfortunately, it is a known limitation that Availability/Security settings only work with Roles and Profiles and do not function with Groups. I completely understand how this might be disappointing.