Omni doesn’t seem ready to be a real, live feature that promises to do “Quick data analysis and pattern recognition across records” as even the most basic capability that Airtable describes (per https://support.airtable.com/v1/docs/using-omni-ai-in-airtable#faqs).
Background: I have built an Airtable base for volunteer shift tracking for a special event, with a multiple volunteer areas, and multiple shifts per area. Each volunteer has a record, and there is a field where after the event, it was logged if they showed up or not. I am now asking Omni AI to analyze how many volunteers showed up, per volunteer area.
Issues/Process: First, Omni completely hallucinates data and presented a summary table of the show up rates -- but upon probing deeper, it arbitrarily picked volunteers and included them as part of areas they were not in. Next, I uncover that the analysis did not actually even use the boolean/checkmark field that describes whether a volunteer showed up or not. After those two issues, I find out Omni was ignoring some shifts for no reason. After teeth-pulling I eventually got Omni to a point where it addressed these issues, and acknowledged “Thank you for pointing out the discrepancy… a complete and accurate analysis requires a thorough aggregation of all relevant records.” (i.e. it did not have this requirement beforehand!).
So, then I ask for it to run it’s analysis again, but there is a hard stop (built-in by Airtable?) where it will always create a summary or partial “demonstration” and just gets stuck in a loop asking for confirmation of a full analysis, even after I explicitly ask for the complete analysis. This approach might be great for initial checking for issues, like what came up, so that is appreciated, but is also is cutting off getting the full analysis with “showing your work” so I can spot further hallucinations.
Conclusion/question: Am I missing something about the expectations or limitations of Omni, or Airtable’s intents / roadmap? It appears Airtable has shifted to focus on being an “app-building platform” now (vs online database service, years ago when I started using Airtable for this) -- but with such an emphasis on AI capabilities in and on this platform, these basic data analysis hallucinations and walls seem very problematic.
Addendum:
After more teeth-pulling, I think Omni is willing to give and show me the full analysis now, across multiple messages where I have to confirm for each volunteer area. I would hope to think that Omni would be doing this data analysis anyway, if it were truly doing a “complete and accurate” analysis to eventually show me an accurate summary table based on my data. But perhaps somehow splitting such analysis across multiple messages and needing multiple prompts to ask to get that analysis is something that Airtable has deemed better, e.g. to track / account for the computational effort to actually achieve accurate “data analysis” results?
tl;dr
Omni AI hallucinates basic data analysis results, even though Airtable says the feature is useful for “Quick data analysis and pattern recognition across records“ and more.
