Bryan Cantrill:
This pattern of Fool’s Expertise is repeated over and over again among AI doomers: their experience in one kind of system (e.g., LLMs) lends them unwarranted authority in another — authority that goes largely unchecked by people like Klein who don’t delineate between different kinds of expertise.
It’s ironic — and yet perhaps only logical — that AI and its creators mirror each other, mistaking an ability to process vast amounts of information across disciplines as genuine expertise in them all.
Newscaster voice: “Now, let’s go over to Ja Rule Anthropic employees and see what they think about the situation….”
Bryan Cantrill:
this week, ex-Anthropic employee Jacob Coxon claimed […] that the probability that AI will "kill all humans" is ">10% in the next decade" (!!) […]
how would that happen? […] Jacob Coxon cites "hacking critical infrastructure" and "extinction-level bioweapons" without further elaboration. But Coxon is not an expert on critical infrastructure, nor on bioweapons — nor, for that matter, on extinction.
Silicon Valley’s mantra should be: You Can Just Do Say Things.
It is incumbent upon us to be circumspect in our claims
Remember when people were held accountable for the things they claimed? Good times.
Bryan Cantrill talks porting code, but really this sounds applicable to any creative endeavor:
it is very difficult to have forward visibility as to progress. That is, you can feel deceptively close to your goal (only to discover some major piece that needs to be rethought) — and you can also be deceptively far (what feels like smoldering wreckage is sometimes but a single fix away from functional software).
Also: reminds me of golf. One moment I’m cursing “I hate this stupid game”. Then I get a good shot, all is forgiven, and I setup a tee time for next week.
Bryan Cantril:
LLMs inherently lack the virtue of laziness. Work costs nothing to an LLM. LLMs do not feel a need to optimize for their own (or anyone’s) future time, and will happily dump more and more onto a layercake of garbage. Left unchecked, LLMs will make systems larger, not better .
That’s a good answer for the next job interview.
“What’s your best attribute?”
“That I’m lazy”
To be human is to want something simple because simple is the only thing intelligible to our limited intellectual and temporal capacities.
The best engineering is always borne of constraints, and the constraint of our time places limits on the cognitive load of the system that we’re willing to accept.