Okay so one way to do this is to add some sneaky formula logic to the rollup formula:
IF( values, AVERAGE(values))
So IF the rollup has values (or if there are linked records), do the AVERAGE, but IF NOT, it outputs nothing.
One thing to be aware of are 'ghost' or null values that may throw this off. So alternatively you could toggle on "Only include records from the..." option and filter to just records with a value in the Exchange rate field