A custom group is any group that you create outside of the default groups that come installed with your server or assigned to a new project.
A custom group can be either a server-level group or a team project group. Server-level groups can have permissions at both the server and the team project level. Server-level permissions include the ability to create new projects, administer shelved changes, and change the process template, among others. Team project groups can only have permissions to perform project operations, such as start a build or publish test results.
For more information on the various permissions available to Team Foundation Server groups, see Team Foundation Server Permissions.