If you follow the spec, detaching a display doesn't work:

Setting the position flag and the width and height to zero doesn't make any sense (and it doesn't work if you try).

To detach a display, you need to set the dmFields to DM_POSITION | DM_PELSWIDTH | DM_PELSHEIGHT, and set dmPelsWidth, dmPelsHeight, dmPosition.x and dmPosition.y to 0.

I also found that the spec was incorrect, but I didn't need to set dmPosition.x or dmPosition.y to 0. It was enough to use dmFields = DM_POSITION | DM_PELSWIDTH | DM_PELSHEIGHT and dmPelsWidth = dmPelsHeight = 0.