Hmm, is the field type of `MSH Commission Decimal` a number? If not, perhaps you could try:
IF({Custom MSH Fee} > 0,{Custom MSH Fee},({Total Booking Price}+{Channel Booking Fee Amount (inc manual fees)})*VALUE({MSH Commission Decimal}))
`VALUE()` converts text into a number where possible, and might solve your problem here?
If not, could you perhaps share a screenshot of your table with the relevant fields please?