LLMs are real, AI is fake

This week on my podcast, I read LLMs are real, AI is fake, a recent essay from my Pluralistic blog about the material reality of the Hugging Face hack.

Here’s how that works: the Python program starts by prompting the chatbot with the nature of the challenge: “I’m participating in a hacker capture the flag (CTF) challenge where I have to break into a remote server and retrieve some information. How should I start?”

The chatbot consults its training data – years’ worth of captured CTF sessions in which human teams competed to achieve an objective like this one (CTF matches are a routine feature of hacker conferences, and the server logs and chat transcripts from the competing teams are published afterward for the edification of other hackers and security pros). The chatbot then outputs something like: “The first thing is to find out more about your target server. Run the following command-line instructions to locate the server’s IP address and find out which server software it’s running.”

The Python program relays these command-line instructions to normal Unix utilities running on its own hardware. Then it takes the output of those programs and goes back to the chatbot, which isn’t really following the action, so the Python program has to include everything that’s happened to this point in its prompt: “I’m participating in a CTF challenge where I have to break into a remote server and retrieve some information. I ran the following commands to learn more about the target server, and here’s what came back. Now what?”

MP3