Under point d) of step 1 of configuring the Service Behaviour, a caveat is mentioned that the custom authentication scheme is actually using a subset of windows authentication.
No indication is given of exactly what that means. Indeed, in practise I have tried to setup a custom authentication with Basic authentication over Http or Https on the webHttpBinding - and it would appear that are no acceptable configuration settings that actually make it work...