First get a channel list from your operator/provider. Mine (willy.tel in Hamburg) can be found here and looks like this:

Name       Language  Freq  QAM   Chan Bouquet
CNN Int.   english   786   256   K60   english

Now in MythTV backend setup, open the channel scan and choose “complete scan (tuned)” and enter frequency and QAM (modulation). The frequency field expects a value in Hz, so the 786 from my example becomes 786000000 (these frequencies are usually given in MHz).
Leave the other fields as they are.

When you start the scan, MythTV will find other transponders as well.