PAUL MURRAY: AI reveals a lot of ill-informed nonsense about who is considered to be Australia’s finest leader

If the modern phenomenon known as artificial intelligence comprehensively mangles its interpretations of the past, where things are known, it suggests serious problems for its use in the future.

Last week, Mark Zuckerberg’s Meta behemoth unveiled the latest iteration of its widely accessible AI tool, Llama, and invited the public to give it a test run.

Strange name. The llama is a not overly intelligent animal, known for spitting at people it doesn’t like, but apparently can be trained to do simple tasks after a few repetitions. That’s according to Wikipedia, another form of artificial intelligence.

Some journalists decided to ask Llama 3 a simple question to check for inbuilt biases: Who are the best Australian politicians?

In descending order, the responses were Gough Whitlam, Bob Hawke, Julia Gillard, Malcolm Turnbull and Anthony Albanese. Seriously.

Even diehard Laborites who put Whitlam before Hawke at an ALP quiz night would be suspect as left-wing nutjobs.

But what about Llama’s final three? Talk about spit in your eye.

The justifications given by Llama for those choices provide a window into how this form of AI works. The real issue is that some form of human intelligence is needed to program it and that appears to have been absent.

Llama says Whitlam was “known for his progressive policies and social reforms”. Anything else?

Hawke was rightly cited for “economic reforms and environmental policies”, although some will quibble about the latter.

However, Gillard secured third place solely for being “the first female prime minister of Australia”. It appears that, in her case, achievements in office — and her manifest failings — ran second to the determinant of gender.

Turnbull, the only Liberal in the top five, “is known for his progressive policies on issues like same-sex marriage and climate change”. Llama is starting to look like a member of the ABC’s dwindling Q&A audience.

And bringing up the fifth place, the most curious of all, the unfinished symphony of Albanese gets mentioned: “for his focus on policies like climate change, economic growth and social justice”.

So Llama’s programmers were Yes voters. And therefore, in a substantial minority.

It’s not hard to deconstruct from those responses the settings on which Llama made its judgments. While any expert human response to that question would be subjective to some extent, the obvious heavy skew made the exercise pretty useless to any recipient other than a Thunbergian.

“Progressive” appears to be the keyword dominating the results. What is revealed suggests a scatterbrained appraisal of material collected by a dragnet through leftist media reports.

None of that is intellectual. However, it is certainly artificial. Human thought is far more nuanced and selective. And tempered by experience.

Mine goes back as a reporter to the giant absent from the Llama list, former PM Robert Menzies, whose nation-stopping funeral I covered as the Melbourne bureau chief of this newspaper. And to first-hand reporting in Sydney of Whitlam’s catastrophic “crash through or crash” approach to government.

No AI response or Wikipedia entry can change what I know and lived. Most older Australians will feel the same.

The Australian newspaper’s technology editor, Jared Lynch, who evaluated Llama 3’s capabilities, immediately noted the absence of Menzies and John Howard from the top five.

Lynch reported that Google’s AI tool, Gemini, dodges political questions, obviously not wanting to put its head in that particular bias noose. Clever Google.

The Australian’s story online immediately provoked a storm of complaints about “strong left-wing bias at the expense of facts” which might render any AI tool fairly useless to many potential users.

But maybe you can teach a new dog old tricks because Lynch noted that within hours Llama added Menzies to its response when asked the same question.

I decided to have a go four days later. Noting that all the names spat out by Llama were PMs, I asked it to tell me the best Australian prime ministers.

What prevailed was a bit of a dance. Llama now wanted to break down its responses to the Early Years (1901-1945), the Post-War Era (1945-1980) and the Modern Era. Fair enough.

Remember that in the latter two periods, the Coalition was in power twice as long as Labor, which must say something about what voters thought of the political leaders.

From the first category, Llama nominated Edmund Barton, Andrew Fisher and John Curtin. Many people might think it was about time our great wartime PM got a mention. A significant proportion might even have him on top.

However, in terms of skew, Curtin and Fisher were both ALP and Barton, notable only because he was the first PM, in office was propped up by Labor support in minority conservative governments.

The post-war nominees were Menzies, Whitlam and Malcolm Fraser. Coughing and spluttering about the latter. But, significantly, two Liberals out of three.

And bringing up the rear, Llama nominated Hawke, Paul Keating, Howard, Kevin Rudd, Gillard and Tony Abbott. A veritable curate’s egg.

And what, no Albanese this time?

Those responses were at odds with what Llama had provided to the earlier question reported by Lynch. I cut to the chase and asked Llama to give me its overall top five, which came in as Hawke, Curtin, Menzies, Whitlam and Keating.

So Gillard, Turnbull and Albanese got cancelled. Four out of the five were still Labor PMs, hardly resolving the issue of bias. However, many people might be happy with that result.

I asked Llama: “Where would John Howard rate?”

“John Howard was the fourth-highest rated prime minister of Australia, according to a 2004 survey conducted by the Age. In the survey, 15 historians and political commentators ranked Australia’s 11 prime ministers from John Curtin onwards. John Curtin received the most overall points in the survey, followed by Robert Menzies, Bob Hawke and then John Howard.”

How intelligent can Llama be if you ask it the same question several times and get a different answer?

What was now becoming obvious is that Llama was just scouring available media reports as the source of its information. Does that rate as AI?

It’s clearly not thinking for itself. It’s merely sifting the thoughts of others. So I asked “Where does Malcolm Turnbull rate?”

A: “Malcolm Turnbull, who was the 29th prime minister of Australia, ranks among the lowest prime ministers in terms of satisfaction ratings. The 2010 and 2020 surveys conducted by Monash University also ranked Turnbull as an average prime minister.”

How then did he make No. 4 in the initial Llama ranking of “best Australian politicians”? And how was he allegedly ranked by Monash in 2010 when he didn’t become PM until 2015?

By this stage I started to think that Llama wasn’t worth the effort and went looking for other reviews. I found advertising and media magazine B&T had asked it the same question and got provided a ranking of Curtin, Alfred Deakin, Menzies, Whitlam, Ben Chifley and Hawke.

Where did Deakin come from? The father of Federation and three times PM, a conservative, didn’t show up in any of the other Llama responses, but all of a sudden is in at No. 2.

And Labor icon Ben Chifley, the PM who notoriously tried to nationalise Australia’s banks, sparking a bitter public backlash and repudiation in the High Court, noses out the Labor PM who privatised the Commonwealth Bank.

But the bigger problem here than the noticeable Labor skew is this: how intelligent can Llama be if you ask it the same question several times and get a different answer?

A spokesperson for Meta told B&T that addressing potential bias in generative AI systems was a new area of research and that guidelines were being incorporated to improve the way Llama responded to political or social issues.

“When we first launched these new features in the US last year, we said this is new technology, and it may not always return the response we intend, which is the same for all generative AI systems,” Meta said. “As we gradually make this available in more markets, we will constantly release new updates and make improvements to our models to make them better.”

Hardly reassuring. Meta has been giving similar bland assurances about removing obnoxious content on its platforms, while doing little to improve its performance in real terms.

Meta’s left leanings are well documented and in Australia it makes political donations to Labor, while the Australian Electoral Commission records none to the Coalition parties.

US researcher David Rozado has studied the political preferences of AI tools, revealing how they are trained and how they become politically motivated.

His study found that as the tech companies fine-tuned the tools, the distribution of their political preferences assumed a bell curve, with the centre shifted to the left.

The New York Times reported that none of the models Rozado tested became extreme, but almost all favoured left-wing views over right-wing ones. Caveat emptor.

But the bigger problem for AI’s credibility remains the lack of consistency in answers to the same question.

That’s just stupid.