Correct. That’s the default operation: to use the layout of the first view of a given table when showing linked record details. That’s not because of any special connection between the records and that first view (more on this below). It’s just a choice made by the developers because (guessing) it’s probably the easiest to implement.
Records aren’t tied to specific views. View filters only control record visibility. They don’t create a relationship or connection between the record and the view. Even if they did, that still wouldn’t make the problem any easier to solve because more than one view could show the same record. If View A and View B both have filter settings that make Record 1 visible, which view’s layout should take precedence when displaying that record’s link elsewhere?
Does that make things more clear?