I've just been playing around with Microsoft's Sandcastle CTP, a replacement for NDoc (RIP). No matter whose batch script I used, Sandcastle Helper, Sandcastle Batch, I kept getting the same error message which was stopping the generation of the chm.

HHC6000: Error: An internal file could not be created. Make certain there is enough disk space on the drive where you are compiling your file.
HHC5007: Error: Fatal navigational compilation error. This is likely the result of an invalid contents (.hhc) file.

But I finally came across SandcastleGUI which works great, and is simpler to use than editing a batch script.