Welcome to the Airtable community!
Are you using the Standard REST API? (“filterByFormula” is a feature of the Standard REST API, whereas “formulas” usually refer to a formula field within a table.
In Airtable, building queries works very differently from building queries in other relational databases. As you’ve discovered, linked records are stored as arrays of record ids. However, if you use the “filterByFormula” parameter, it should work exactly as it works in the user interface. You should be able to build your formula in a formula field, and once it works, copy the formula exactly (substituting variable names as necessary).
If you would like additional help, can you post screen captures of your base and some code samples so that we can better understand your situation?