Are the linked records in a different table?
Can you remove the user from the backlink field in the [Users] table? If there is a one-to-many relationship (versus a many-to-many relationship) this is usually the easier way.
If you have a many-to-many relationship, I recommend scripting. It is possible to have a non-scripting process that involves formula fields and multiple reads on the records. However, the non-scripting method would require splitting the logic up across multiple fields and possibly multiple automations, making the system harder to maintain.