User blog comment:Observer Supreme/Is there something I'm missing?/@comment-30395642-20170521081631

There's a small scene pre-timeskip where Doflamingo's talking to the ringleader of the slave auction house on Saboady and he claims that he's toning down his slavery business because he's focussing on SMILES. That would imply that his smile production plan was fairy recent.

Even without that though dwarves are superstrong so they wuld be useful for slave labour even before smiles.