The description for File.Delete and the Intelli-sense both say that an exception is thrown if the specified file does not exist. This is not the case. It quietly continues if the file does not exist.
I believe the current behavior is what's intended (i.e., quietly ignore missing file). Please update the documentation and Intelli-sense.