A custom group is any group that you create outside of the default groups that are created when you create a new team project or a new team project collection.
A custom group can be either a collection-level group or a project-level group. Collection-level groups can have permissions at both the collection and the project level. Collection-level permissions include the ability to create new projects, administer shelved changes, and change the process template, among others. Project-level groups can only have permissions to perform project operations, such as start a build or publish test results.
For more information on the various permissions available for custom groups, see Team Foundation Server Permissions.