If you want CKEditor to output valid HTML4 code instead of XHTML, you should configure the behavior of the dataProcessor.
For some tips on how to achieve this, check the Output Formatting section of CKEditor 3.x Developer's Guide as well as the Output HTML (output_html.html) and Output XHTML (output_xhtml.html) samples that can be found in the _samples directory of CKEditor installation package.
If, for example, you want CKEditor to output the self-closing tags in the HTML4 way, creating <br> elements instead of <br />, configure the selfClosingEnd setting in the following way.
CKEDITOR.on( 'instanceReady', function( ev )
{
ev.editor.dataProcessor.writer.selfClosingEnd = '>';
});