Welcome to the Airtable community!
Even though the single select values look like numbers, they are not numbers to Airtable. You need to convert them to number in your formula.
AVERAGE(
VALUE({Evaluator 1 score}),
VALUE({Evaluator 2 score}),
VALUE({Evaluator 3 score})
)
You will also probably want to adjust the formatting options for the formula field to show the number of decimal places you want.