That’s correct. Cross-table comparisons can’t be done because the chart app only connects to an individual table.
Another option (just a variation on that master table idea) would be to sync your individual year tables to a single master table (look up multi-source syncing in the docs). If your individual tables have a date field, the year identifier could be handled automatically with a formula field that pulls out the year.