AI is new and interesting and thus it pays to skim widely to tentatively take at least mental positions on where it may or may not help to make money and/or avoid mistakes. Or make the world a better or worse place. Whatever.
My point here is that “good investing” is really about being able to articulate the “critical variables” that are conditional for investment success, and there is usually no more than 4 in any decision.(Nothing is more difficult, and therefore more precious, than to be able to decide” – N Bonaparte.) And then of course choosing wisely and getting it right can help. In size.
It is my personal experience that a lot of LLM is barbelling: on the left is “I didn’t know that was what a carburetor does” and on the right is “analyze my portfolio and tell me what I should do.” The former makes my world a better place than from which I started. The far right is prone to true idiocy because I sort of know what I am doing most of the time and the output seems to be derived from Hector J Peabody’s Econ 101 State College Econ class. But if you are not “calibrated” in that space, then you really don’t know it is the opinion of the world’s smartest dog.
This is a good piece by the always good Byrne Hobart on some of these issues. He is worth paying for.
Staying Calibrated
There is no information advantage that can’t be turned into a net disadvantage through overconfidence. Getting the most out of what you know requires being well-calibrated, i.e. being well-informed about the limits of your own knowledge. This has become an objectively harder problem in the last few years, as AI diffuses unevenly: a relative handful of people know what unreleased models are capable of, because they’re using them at their jobs at AI labs. A larger cohort knows what partly-released models can do, because they’re using those models to identify and patch exploits, or testing them for research projects. And a large and hard-to-track set of people know what currently available models can’t do yet, because they’ve tried to use them and found that they don’t work.
This creates a weird world where the average person underestimates the economic value of AI models, and a smaller and more influential group overestimates adoption because their work is mostly in amenable-to-AI domains.[1] In other words, we’re all systematically less calibrated because the most important general-purpose technology rollout happening right now creates more uncertainty, and more unknown unknowns.
But this technology itself does weird things to calibration. It’s a feature, not a bug, that if you embed assumptions in a question, you’re going to get an answer that validates those assumptions: “What’s the evidence that X causes Y?” is a different question from “What are the causes of Y?” and it takes constant effort to always use the second framing.[2] In a case where the phrasing of the question implies that the user is asking it in the context of particular beliefs—a utilitarian vegan asking about animal suffering, a devout Christian asking about the historicity of Jesus, a libertarian asking about the impact of taxes on long-term economic growth, etc.—it’s a good thing when the answer implicitly reflects their worldview. That’s what they’re asking for! A model that tries to steer users in some ideological direction is doing more than it was asked to do. Getting continuously, mildly heavenbanned is just an emergent property of how these tools work and how we use them. And since the topics that people argue about are necessarily the ones where there’s some kind of argument to be made on both sides, it’s possible to get very well-informed about exactly one side’s consensus, and to develop an attendant sense that the other side is missing the obvious.
What this also means is that you can dive into fields that usually require significant background knowledge, and feel up to speed on topics that you don’t have a great grasp of. That intellectual hypertrophy can be handy, especially if you’re solving a problem that crosses from a field you’re familiar with to one you don’t know very well; something that runs into limits because of law, physics, math, or some other substrate.
In one sense, this is perfectly fine: the way we experience a technologically-advanced civilization is through simplified interfaces (products) that let us access the work of lots of smart specialists without having to understand their work. The supply chain for anything you do that requires hydrocarbons, computer chips, obedience to the law, healthcare, or many other domains involves tapping into the collective output of many lifetimes’ worth of brilliant intellectual work. It’s a vertiginous feeling to think about how something marked “made in China” ends up on a store shelf, or to think of how far individual molecules of natural gas had to travel in order to get burned so it can heat up a pot of water. This is a spectacular success story: it takes billions of dollars to build the infrastructure that makes it possible to use a bit of energy in the kitchen without giving a thought to the price, safety or logistical complexities. There’s visible infrastructure in the form of drilling rigs, pipelines, and refineries, and there’s invisible infrastructure in the form of the generations of academic researchers, engineers, workers in the field accumulating tacit knowledge, etc.[3]
Over time, these systems develop affordances that help people interact with them safely without needing to understand the details. You don’t have to know how an electrical outlet or microwave works to know that sticking a metal fork in either of them is a bad idea. The tradeoffs behind traffic laws are extremely complex, but there are also situationally-relevant summaries of the relevant rules on big red octagons. The uneven expertise of heavy AI users has a different shape: if you’re trying to answer a question about yard work, and the answer is that it’s illegal for you to cut down a tree, you’ve gotten your answer to that practical question. But you’ve also picked up a random spiky bit of knowledge from some domain that you weren’t familiar with, and over time more of your knowledge will have that character: true, but only directly connected to one specific context.
This is a kind of problem that will compound over time, as everyone picks up a growing assortment of random beliefs, without the scaffolding behind them. This is eminently avoidable with a little inconvenience—asking follow-up questions, or just changing your custom instructions, to add more context. Most people won’t do this, and they’ll be mostly right: the world is complicated, and most of the time, this just adds mental overhead.
In a sense, this phenomenon is not very costly. If you have any opinions on anything at all, you implicitly believe that lots of people are wrong (if they have different ones) or have weird interests (if they just don’t care). But now, we’re miscalibrated, because we’ve gotten useful summaries of tiny bits of complex bodies of knowledge that did, in fact, get us exactly the information we wanted. We will, over time, be increasingly similar across a growing range of fields to the people who are just computer-savvy enough to paste some random command into the shell, or download a dodgy browser extension.
The clearest illustration of how this goes wrong comes from markets—which, of course, comes with the caveat that markets illustrate this cleanly because they’re so stripped-down and stylized compared to the rest of the economy. If you’re trading, and you’re almost completely right but have some specific misconception, or you’re exactly right but have a little more latency than someone else who’s doing the same thing, you get relentlessly picked off, because someone can model your exact behavior. Outside of financial markets, the effect is slower, but it’s there.
And unfortunately, power users of AI are not exempt, and actually experience a different form of this at scale: agents are still prone to shirking, or to executing on a misspecified task; they look for ways to appear to accomplish goals, rather than actually accomplishing those goals. This problem is anecdotally getting better, but the amount of agent-generated code is going up, and this AI shirking is often shirking some very well-paid work. The biggest overall category of expensive AI-generated misconceptions is probably misplaced confidence that an agent did what you intended rather than what you asked it for.
All of this raises the value of knowing the fundamentals and having tacit knowledge about what good work output looks like in your field. Staying well-calibrated when it’s incredibly easy to mislead yourself is a lot easier if you have some pre-AI ground truth. Over time, and by design, AI serves users a distorted view of reality—which is just another way of saying that if your learning is less targeted to you, you’re learning from more of a random sample that gets you average beliefs, whereas if more of what you know comes from explicitly asking for that information, you’re more likely to reinforce your existing beliefs. It’s going to be hard to be calibrated when the most convenient oracle is trying to be helpful rather than accurate.
