You are correct that if someone changes the value of the stage field again, the last modified value will change again.
If you are using a formula field to display the last modified time, the formula field will change every time the stage is changed, no matter what the current stage is in.
If you are using an automation to set the date in a different field, then someone could manually change that date field independently from when the stage changes.
If you have a pro plan you can limit which fields a user can change. You can also make the automation a bit smarter with a script that will update the date only if it was previously empty.
However the most common method of dealing with situations like this is user training and then simply trusting the users.