You can try using a formula to extract the date portion from your original date field, regardless of the "include time" setting. This way, you'll always have the intended date without rounding.