Hmm, what you said about the primary view is correct; it always uses the field order of the first view of that table
In the screenshot below, it uses the field order of 'Grid view'

Here's a link to a base where I've just tested this and where it works as expected
I'd be happy to take a look at this for you if you could invite me to a base where this isn't working for you!