Welcome to the community, @Moez_Tharani!
You could create a formula field that gives you the “grouping phone number” for each record. Something like this:
SWITCH(
{Direction},
"External", {Phone Number Field 1},
"Internal", {Phone Number Field 2}
)
Replace the field names with your actual field names.
Then, you can group by that formula field.