You would not be able to enter a formula in the field in your screen shot, it is just looking for a string to match.
Have you taken a look at the DeDupe application? It is free and will show you all duplicates and give you options to delete or merge.
There are some other steps you can take if the DeDupe app is not sufficient, feel free to let me know here and I can guide you to another path.
I know of three ways of finding duplicates.
Method 3 is the only way I know that you then can use color by field or conditions like you showed.
I’ve used it in the waaaay past in my total n00b days, before the Dedupe App, and it works. A lot of work and fields to get there though, but it givs live ‘commentary’ in your base without you running a script or app.
Might not be the way for you, but it’s a way.