Per this article, both <a> tags and <img> tags are supported in email automations. Formatting <div> tags does not seem to work. This article is not specific to Outlook emails, but I find these tags work in both Airtable branded emails and Google emails, so they probably work in Outlook emails.
You cannot embed an html button, but you can use a link that will open a webpage. You may even be able to use an image inside the link, where the image looks like a button.
