Hello,
first of all, the indexer does not look for transformations to PDF, it looks for transformations to plain text. This might find a round-trip transformer, i.e. one that first converts to PDF, then to plain text - but such a round-trip transformer may need additional setup.
second of all, as I said, the properties only "declare" that a transformation should be considered supported. They do not technically force a transformation to be supported if the transformer actually fails or its implementation rejects it despite the configuration.
Furthermore, the OpenOffice.Html2Pdf transformer already is a pipeline. This pipeline works with input documents that OpenOffice can convert to ODT internally (HTML => ODT) and then to PDF (ODT => PDF). Also, the maxSourceSizeKBytes config should have been for OpenOffice.extensions.newMimetype.html.maxSiurceSizeKBytes and OpenOffice.Html2Pdf.extensions.newMimetype.pdf.maxSiurceSizeKBytes to be complete.
Now, either your new mimetype is HTML or it isn't. If it is you shouldn't have to do anything - if it isn't, that could be a possible reason for OpenOffice possibly rejecting it when using only OpenOffice.Html2Pdf - if your new pipeline contains a transformer that first converts to "real HTML" before passing it, that would explain why it's now working.
Regards
Axel