Weaknesses of AI chatbots revealed in MRS experiment

Commercial chatbots powered by artificial intelligence respond inaccurately and inconsistently to queries about UK census statistics, according to a report published by the MRS Census and GeoDemographics Group (CGG). The report finds that most AI models, when asked a question about GDP figures for the UK, got the answer wrong or refused to give an answer at all. Some, subsequently asked an identical question, produced a different answer entirely.

More information on the experiment and its results can be found on Research Live here.

Jane Frost, CEO of the MRS, recently recounted a personal story on a similar theme. She asked an AI model to look up the year when she was awarded her CBE, and was surprised to be given entirely the wrong date. Jane added that as the Honours List is public data, she had assumed that this would be a simple task for AI. Not so, it seemed.