If someone went to Warren Buffet and asked him to invest in a black box that gives great outputs, but we don’t understand it…do you think he would invest in it? Chances are that he would not since his long-time mantra has been “don’t invest in something you do not understand.” Currently, there is a frenzy of companies and investors jumping on the AI bandwagon.
Traditionally the gold standard in science was to test theories with data. Looking at patterns from data (mining) was traditionally frowned upon as voodoo science. However, it is these very patterns from which we use (human) reasoning to generate theories that we can then scientifically test. LLMs with their predictive bent are simultaneously generating and testing theories without any scientific underlay. Many of the patterns identified are not apparent to humans. Also, since these models do not “understand” anything, the explanation layer that is typical in most scientific theories is glaringly missing.
A simple way to delineate the limitations of LLMs is as a data problem vs. a reasoning problem.
The way humans understand situations is by reasoning them out. LLMs have a reasoning problem. Complex logical (deductive) reasoning is not its forte. While the output might fortuitously appear to be logical – the reasoning is based on data patterns in the training set. So, simple claims like X is the parent of x, may not logically lead to the deduction the x is the child of X. This is one of the causes of hallucinations. When the training data is broad (as it is in the case of foundational models), then the chance of hallucination is higher. When the data is highly specialized, the model can perform well within a domain. However, good quality specialized data is tough to get.
So, within a constraint and an objective function (modulated as a prompt), the LLM can do well (like play chess). The AI resolves the reasoning problem with a massive corpus of specialized data (millions of chess games) that allows it to make excellent predictions. But if we change the chessboard from 8X8 to 9X9 or introduce a new piece – then the training set (based on regular chess) makes it a poor player of the “new” chess, and the reasoning problem manifests itself.
There are grounds to believe that technology will improve its reasoning ability. Perhaps scaling the AI and bigger and better training will mitigate the issue. Larger data sets have led to greater predictive capability…but the relationship is unclear (i.e., there is no equivalent of Moore’s Law here). Curated training data sets will help as would more specialized training of foundational models. Refining models with human feedback can help too. Reasoning engines can also be programmed into the LLM.
It seems that many companies in their gold rush are jumping on the AI bandwagon prematurely. I know a company that uses voice samples to diagnose depression, and another is using facial analysis and deep learning to detect phenotypes that correlate with rare genetic diseases. But are such commercial applications too premature? A common edict in the digital age is to not sit on the sidelines and to move fast with rapid experimentation. However, until we can get to a point where we trust the AI or keep the human with knowledge and authority firmly in the loop, we should restrict its use (and investment) to low risk decisions and not high risk (life and death ones). If we don’t, we might be hallucinating about our return on investment in AI.
Leave a Reply