It can’t be done automatically — that is, without human intervention — entirely within Airtable, but it can be done with minimal human intervention in Airtable or automatically with the assistance of a third-party integration service (e.g., Zapier, Integromat).
Essentially, what you need to do is create a formula field containing the desired calculation; you then copy the calculated value and paste it into the linked-record field. That will cause the record in [Table A] to link to the appropriate record in [Table B], should it exist, or it will create the appropriate record in [Table B] and link to it.
That copy can be a simple, three-keystroke sequence (assuming the formula and linked-record fields are adjacent): Ctrl-C, Right-Arrow,Ctrl-V. Alternatively, you can grab the [fill handle](https://support.airtable.com/hc/en-us/articles/208330098-Quickly-filling-cells-using-fill-handle) on the calculated field and drag it into the linked-record field. Or, if you don't need the link made immediately upon creating the record in[Table A], you canLeft-Clickon the header for the calculated field, thus marking the entire column, pressCtrl-Cto copy all cells in the column,Left-Clickon the header for the linked-record field (ditto), and pressCtrl-V` to paste the entire column’s worth of values.
Finally, you can use such a service as Zapier or Integromat to perform the copy/paste. (Actually, the ‘copy’ action is implied: Trigger on a new record being created and save the value of the calculated field into the linked-record field. In Zapier, this is counted as a two-step Zap and, as such, can be executed from a free account.)