First get a channel list from your operator/provider. Mine (willy.tel in Hamburg) can be found here and looks like this:
Name Language Freq QAM Chan Bouquet CNN Int. english 786 256 K60 english
Now in MythTV backend setup, open the channel scan and choose “complete scan (tuned)” and enter frequency and QAM (modulation). The frequency field expects a value in Hz, so the 786 from my example becomes 786000000 (these frequencies are usually given in MHz).
Leave the other fields as they are.
When you start the scan, MythTV will find other transponders as well.