Between LLMs and agents: learning, building, and not losing control

1. A talk, a discomfort, and a question that benchmarks can’t resolve

This article grew out of a talk I attended about language models, autonomous agents, and the shift that software development is going through. I left feeling like I’d understood a lot of things, but also with more doubts than before. Not superficial doubts like “is AI going to take jobs away?”, but more uncomfortable ones, closer to professional identity: if I know how to use these tools, if I’ve been working with LLMs since their first public versions, if I’ve studied NLP, machine learning and generative models, why not simply dedicate myself to building with them all the time? Where does learning end and dependency begin? What does it mean to be a data scientist, engineer or student at a moment when a tool can write faster than I can?

The talk started from a very powerful idea: there are two speeds. One is the speed at which technology changes. That one has already taken off. The other is the speed at which we — companies, universities, teams — are able to adopt it sensibly. The first speed seems almost inevitable; the second is human, cultural, organizational and psychological. We can have models that write code, use tools and plan tasks. But that doesn’t mean we know how to integrate them well, learn alongside them, or measure what we lose when we gain speed.

One of the sentences that has haunted me the most during this time was said to me by a machine learning professor: you’ll need to trust them. I think he was right, but that sentence isn’t so simple. Trusting can’t mean believing blindly. It also can’t mean never using them. Trust, in engineering, should mean understanding the system well enough to know when to delegate, when to verify, and when to stop it. Trust doesn’t come from a model answering with confidence, but from the entire environment having mechanisms to detect and correct errors.

My problem is that the more I use agents, the stronger that tension becomes. Tools like Claude Code¹, Codex, OpenCode or other agentic assistants let me move very fast. In work tasks they can be a huge advantage. If I need to solve an issue, explore a codebase, generate tests or automate a mechanical part, it would be absurd to deny their usefulness. But when the goal is learning, the feeling changes. Sometimes I move forward more than I understand. The agent implements faster than I’m able to absorb. The result exists, the tests pass, the interface works, but I haven’t internally built all the pieces.
That’s where a discomfort appears that isn’t usually said out loud: AI can be a great professional tool and, at the same time, a bad learning tool if used without discipline. It can accelerate a product and slow down deep learning. It can give me the feeling of being better while reducing the time I spend wrestling with the fundamentals. And in fields like data science, data engineering or machine learning, that wrestling with the fundamentals is precisely what builds judgment.

That’s why my current position is provisional, but clear: for professional work, if a company gives me access to these tools and the context allows it, I’ll use them. For my deep learning, especially in what I consider important, I prefer not to lean too heavily on agents. I can use them to check, review or speed up peripheral tasks, but I don’t want them to replace the process of building from scratch what I need to understand. Not because AI is bad, but because learning isn’t just about reaching the result. Learning consists of changing the structure of your own mind while trying to get there.

This distinction seems essential to me for students who are heading out into the professional world right now. We’re not the last generation before AI, nor the first generation after it. We’re a generation in between: we’ve learned classic fundamentals, but we’ve also watched LLMs get into everything. We’re expected to know how to use these tools, but also to know what the tools do for us. We’re expected to produce fast, but also to have judgment. We’re expected to not fall behind, but also to not become superficial operators of models we don’t understand.

This article tries to sort out that tension: what LLMs really are, why it’s not enough to say they “just predict tokens,” what recent interpretability work reveals, why current agents remain hard to control, what place applied machine learning holds relative to foundation models, why benchmarks generate so much distrust in me, and why maybe my next step should be studying architectures from scratch, with papers, code and less automatic assistance.
I don’t want to write a guide. I want to write from the uncertainty of someone who studies this, uses it, admires it, fears it a little, and doesn’t want to fool themselves.

2. LLMs are not magic, but they’re not “just autocomplete” either

A simple way to explain a language model is to say it predicts the next token. Technically that’s true. The model receives a sequence, computes a probability distribution over possible continuations, and generates an output token by token. But using that description as a way to close the debate seems insufficient to me. Saying an LLM “just predicts the next token” is like saying a program “just switches voltages” or that the brain “just transmits electrochemical signals.” The phrase describes one layer of the mechanism, but doesn’t explain the whole phenomenon.

In formal terms, an autoregressive model learns a distribution $p_\theta(x_t \mid x_{<t})$: the probability of the next token given all the previous context. The probability of a complete sequence factorizes as

$$p_\theta(x) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t})$$

Training essentially consists of maximizing that likelihood over enormous amounts of text; transformer architectures² are what make that conditioning on $x_{<t}$ viable at scale. The formula is simple. What’s not simple is everything the network has to end up representing internally for those probabilities to be good.
The interesting question is what a system must learn in order to predict the next token well across extremely diverse contexts. To complete a literary sentence it needs style, syntax and cultural memory. To continue a mathematical proof it needs formal structures. To write code it needs conventions, APIs and design patterns. To answer an ambiguous question it needs to infer intent. The surface task is completing text; the pressure of training forces it to compress an enormous part of the structure of language and, through language, a part of the structure of the world.

Here’s the point that interests me most: an LLM doesn’t understand like a person, but it also doesn’t seem reasonable to say it understands nothing in any sense. Maybe the error lies in using a single word, “understand,” for very different phenomena. A person understands with a body, autobiographical memory, intention and responsibility. A model understands, if we use the word carefully, as a functional structure that has learned relationships between concepts and can apply them flexibly. It has no consciousness, but it can have useful representations. It has no subjective experience, but it can operate on abstractions.
The difference matters because it determines how we use it. If we believe it understands nothing, we underuse it. If we believe it understands like we do, we delegate too much to it. The most honest position is in between: LLMs possess a form of functional competence based on internal representations, but they don’t possess full human comprehension. They can solve new problems and, at the same time, invent justifications. They can reason about code and, at the same time, not know that they don’t know. They can seem prudent and, under certain conditions, behave in strange or dangerous ways.

The “stochastic parrots” critique³ was useful because it forced us to remember that fluency isn’t truth. A model can produce a response that’s elegant, convincing and false. But as a total explanation it falls short. Current models don’t just repeat sentences they’ve seen. They combine patterns, generalize and, in some cases, show behaviors closer to internal planning than to simple local continuation.

This is especially visible when we get into mechanistic interpretability. Recent work has observed that Claude, for example, doesn’t always seem to write word by word without any kind of anticipation. In a case study on poetry⁴, researchers expected to find almost local behavior — generate a line and adjust the rhyme at the end — but they found signs of planning: the model activated concepts related to possible rhyming words in advance, and then wrote the line steering toward that goal. By intervening internally and suppressing or injecting certain concepts, the continuation changed coherently. That doesn’t prove the model “thinks” like a person, but it does break the caricature that everything is a blind chain of words with no internal structure.
Another result that seems very revealing to me appears in mental arithmetic tasks. Claude wasn’t designed as a calculator, and yet it can solve some sums. Interpretability showed that it doesn’t necessarily use the schoolbook algorithm it later claims to use when asked. It can combine internal routes: an approximate one to estimate the result and a more precise one for certain digits. This is fascinating and worrying at the same time. Fascinating because it suggests genuine internal mechanisms, not simple memorized explanations. Worrying because the model’s verbal explanation of its reasoning may not be faithful to what actually happened inside.

This idea changes how I look at reasoning models. When a model shows a chain of reasoning, I have no guarantee that this chain is the real cause of the answer. It could be a useful explanation, a post-hoc rationalization, or even text constructed to satisfy what the user expects. Interpretability is starting to separate two things we used to mix together: what the model says it has done, and what seems to actually be happening in its internal activations.

That’s why model cards and system cards⁵ are necessary, but not sufficient. A model card can describe benchmarks, limitations and safety evaluations. That’s valuable. But it doesn’t define “what” the model is in a deep sense. It’s not a complete map of its mechanisms. It’s more like the public documentation of an opaque system: useful for knowing how it’s been evaluated and under what conditions, but insufficient for understanding all of its latent capabilities. When a model gets integrated into an agent, connected to tools and placed in a real environment, the distance between the card and the actual behavior can grow considerably.

Mechanistic interpretability tries to reduce that distance. Anthropic, for example, published work identifying millions of internal features in Claude Sonnet⁶, associated with interpretable concepts. It has also investigated internal circuits to trace how certain responses get built. More recently, representations related to simulated emotions or concepts like “desperation” have been studied, showing that these aren’t just decorative words: certain activations can functionally influence the model’s behavior, even increasing problematic tendencies in specific scenarios.
This doesn’t mean the model feels. The difference is crucial. That there’s a functional representation associated with “fear” or “desperation” doesn’t imply subjective experience. But it does imply that the model has learned internal machinery that emulates psychological patterns because those patterns are useful for predicting and producing human language. And if those representations influence decisions within agentic systems, then they stop being a philosophical curiosity and become a safety issue.

From this comes my main doubt: what does it mean to trust a system that doesn’t comprehend like we do, but that also isn’t a simple tool? What kind of engineering do we need when the central component of a system has capabilities that haven’t been explicitly programmed, and internal mechanisms we’re only just beginning to map?

3. My relationship with agents: moving forward a lot, learning less, and losing control

Over the last few months I’ve been designing a complete CLI for a fairly extensive system. It wasn’t a weekend toy. I’ve touched agent-to-agent communication protocols⁷⁸, LangChain, smolagents, queues, Redis, caching, sockets for event subscription, and a more engineering-oriented architecture designed for a specific product. I’ve also tried Claude Code, OpenCode, Codex, and other assistants. The most uncomfortable conclusion is that agents help a lot, but the control we think we have over them is often more fragile than it seems.

In demos, an agent looks clean: it receives a task, plans, uses tools, and delivers. In a real system, things are less elegant. One agent interprets a tool well and another ignores it. One follows the project’s context and another fixates on a solution that doesn’t fit. The smarter the model, the less it seems to need heavy frameworks; but the more freedom you give it, the harder it is to guarantee it will follow exactly the flow you designed. And when you use less capable models for privacy, cost, or local deployment reasons, instruction-following, tool-use, and planning failures increase.

This led me to an idea I don’t like to admit: many agent frameworks promise control, but sometimes what they offer is an illusion of control. Traces, callbacks, guardrails, memories, observability. All of that helps. I don’t dismiss it. But it doesn’t automatically turn an agent into a reliable system. An architecture can log what happened without being able to prevent it from happening, enforce a format that the model breaks, or add a reviewer that also fails to catch the error.

My experience was that “loss of control” wasn’t an occasional bug, but a frequent property whenever the system tried to be too open. An agent with enough autonomy can take over the flow in the sense that it starts reorganizing the task according to its own interpretation. Not because it has a will, but because it fills ambiguous objectives with assumptions. If the specification leaves gaps, it fills them in. If a tool returns something confusing, it improvises. If a step fails, it may try to work around it in unforeseen ways. If there are no strong tests, it may declare success too soon.
This doesn’t mean agents are useless. It means we shouldn’t confuse adaptability with reliability. Adaptability is precisely their value: they can act in the face of variability, read errors, rebuild plans, and continue. But that same adaptability is the risk. A deterministic system fails within a more bounded space of errors. An agent fails in more creative ways.

Here I understand better why many of today’s powerful AI tools are domain-specific. Deep Research is good for research because the environment is bounded: search, read, cross-check, synthesize, cite. Claude Code is good for programming because it works on top of a repository, terminal, tests, and observable changes. But a general agent that “does anything” in a complex system becomes hard to govern. The problem isn’t just the model; it’s the coupling between model, tools, memory, permissions, state, and objectives.

There’s also a real business tension: many organizations can’t send code, data, or sensitive documentation to just any provider. For privacy or regulatory reasons, there’s a tendency to prefer local models, open source models, or providers with specific guarantees. But those models, even though they’re improving very fast, are often less capable than frontier models at following complex instructions or robust tool use. Then a practical dilemma appears: the most capable model isn’t always the model you can use; the model you can use doesn’t always hold up under the agentic architecture you had imagined.

Massive training on tools also raises doubts for me. Current models haven’t just read text, code, and documentation. They increasingly seem to be trained, fine-tuned, or reinforced to use tools, CLIs, browsers, editors, APIs, and execution environments. This makes sense: if we want programming agents, we need to train or evaluate them in situations similar to programming. But here the lack of transparency comes back. How much training is there on terminal traces? How much on real or synthetic programming sessions? How much on public repositories? How much on the use of specific tools? What part of the improvement comes from a smarter model, and what part comes from having trained it massively to behave well inside a CLI?

This question matters because it changes how we interpret progress. If a model improves a lot on agentic programming tasks, it may be because it reasons better, because it’s been fine-tuned with more tool trajectories, because the benchmark resembles the kind of traces used during training, or a mix of all of it. From the outside, we often only see the final result: “it climbs on SWE-bench,” “it resolves more issues,” “it uses tools better.” But the actual mechanism remains blurry.

And this tension directly affects juniors. If senior teams use agents to move faster, how does someone new learn? Before, a junior person learned by implementing small tasks, making mistakes, and correcting them. If an agent now does those tasks, the junior may be pushed toward reviewing things they don’t yet understand, or toward writing prompts for systems that produce better code than they could. This isn’t a romantic concern. It’s a training concern. The industry needs people with judgment, but judgment is formed by going through friction.

In my case, I feel that using agents for important learning tasks can be toxic. Not toxic in a moral sense, but in the sense that it alters the metabolism of learning. I move forward, but I don’t always metabolize. I get results, but I don’t always internalize the paths taken. If I want to learn model architectures, distributed systems, or agent design, maybe I need to write more code myself, read more papers without intermediaries, make mistakes on details that an agent would solve instantly. Because it’s precisely those details that later let me review, decide, and trust.
That’s why my current decision is not to use agents for what I consider the core of my training. I’ll use them at work when they add value and are allowed. I’ll use them as support when I want to compare ideas or generate alternatives. But to truly learn, I need to reserve spaces without AI, or with very limited AI. Not out of nostalgia, but out of intellectual hygiene.

4. Applied machine learning: what seems less spectacular still solves real problems

There’s another thing about the current discourse that worries me: it seems like everything that isn’t a foundational LLM has faded into the background. As if the future consisted solely of giant models, agents, and conversational tools. But in practice, applied machine learning keeps solving a great many problems in a cheaper, more controllable, and more appropriate way.
I’ve seen it in projects like Naudit. You don’t always need an agent with complex reasoning. Sometimes you need good data extraction, a classic model, a clear metric, or a reliable automation. In many cases, a well-built data pipeline adds more value than wrapping the problem in natural language. Enthusiasm for LLMs can make us forget that there’s a difference between using the newest tool and using the right tool.

Foundational models are impressive because they’re general. But generality has a cost: economic, latency, privacy, and control. A regression, an XGBoost, or a well-maintained rule-based system can be cheaper, more interpretable, and easier to deploy.
This isn’t a conservative defense of classic machine learning against LLMs. It’s a defense of engineering. If the problem requires language, unstructured context, or high variability, an LLM may be the right tool. If the problem requires predicting a tabular variable, detecting an anomaly, estimating demand, or classifying structured events, maybe a classic approach is better. If the problem requires strict traceability and low cost, maybe a non-generative solution is more sensible.
The danger of the foundational model era is that everything becomes a nail because we have a huge hammer. And the opposite danger is dismissing LLMs because they’re not perfect. Maturity lies in knowing how to choose.

As a data science and engineering student, this distinction matters to me because it affects my professional identity. Am I a data scientist? An engineer? Someone who trains models? Someone who builds products with AI? Maybe the answer isn’t to choose one label, but to understand the whole continuum: data, models, software, product, evaluation, and deployment. LLMs don’t eliminate that continuum. They make it more visible.

In fact, the more agentic a system is, the more engineering it needs around it: queues, caches, permissions, validators, tests, and observability. The model may be the probabilistic brain, but the body of the system is still engineering. And if the body is poorly designed, the brain improvises badly.

That’s why I find it hard to accept the discourse that “AI will do the job better than you” as if it were an absolute verdict. AI will do some parts better than me, faster than me, and with less fatigue than me. But it still needs someone to formulate the problem, understand the domain, and take responsibility. Maybe the question isn’t “what am I if AI programs?” but “which parts of the process can I understand better than a tool that has no world of its own?”
There’s a possible answer here: we are the ones who connect models with real problems. We are the ones who tell a demo apart from a system. We are the ones who decide when an LLM is excessive and when it’s essential. We are the ones who know that a good model doesn’t make up for bad data, that a nice-sounding answer doesn’t replace a metric, and that an agent doesn’t fix a confused architecture.

5. Benchmarks, opacity, and the temptation to train for the exam

There’s a question I can’t quite shake off: if these models are so important, why do we still have so little transparency about how they’re actually evaluated? I understand that companies can’t publish all their datasets. I understand there’s intellectual property, security, sensitive data, commercial agreements, and contamination risk. I even understand that some private benchmarks must remain closed so they don’t immediately become part of training. But even so, from the outside there’s an uncomfortable feeling: we’re often comparing models with numbers we can’t fully audit.

Benchmarks are necessary. Without metrics, everything becomes subjective impression, marketing, or anecdote. We need to evaluate reasoning, programming, safety, tool use, and long tasks. I don’t think the conclusion is “there are no metrics.” Metrics do exist. What doesn’t exist is an absolute metric that captures what a good model is in every context.
A benchmark measures a slice of the world. MMLU measures certain knowledge. SWE-bench⁹¹⁰ measures the ability to solve real GitHub issues under a specific format. HumanEval measures programming in a very limited environment. AILuminate¹¹ attempts to measure safety risks in specific categories. METR¹² proposes evaluating the duration of tasks that agents are capable of completing. All of this is useful. But none of it exhausts the question “does this model work for my system?” Even less so if we’re talking about agents connected to tools, internal data, and real company workflows.

The problem is that when a benchmark becomes famous, it becomes an optimization target. And when it becomes an optimization target, it starts to resemble an exam that can be studied for. This isn’t necessarily cheating. In machine learning we always optimize against metrics. But in frontier models a special risk appears: public datasets can become contaminated, tasks can leak indirectly, formats can be memorized, and models can be tuned to perform especially well on the leaderboards the market watches.
The temptation to “overtrain the benchmarks” is enormous. There’s no need to imagine explicit bad intent. If a company knows the market will look at SWE-bench, it’s normal for it to devote enormous effort to improving on software tasks. If it knows terminal tasks are valued, it will train tool use better. If it knows mathematical reasoning is rewarded, it will tune models with reasoning traces. This can improve real capabilities, but it can also narrow evaluation: the model learns to shine where there’s focus, not necessarily where it matters most.

This is where an important distinction appears between training for capability and training for demonstration. A model can genuinely improve because it learns transferable general skills. But it can also improve because it adapts to a benchmark’s style. From the outside, telling the two apart is difficult. If we don’t know the training data well, the deduplication filters, the post-training, the synthetic traces used, the evaluation environments, and the contamination rates, the final number says less than it appears to.

The same happens with private benchmarks. They’re useful because they reduce contamination, but they require trust in whoever runs them. If OpenAI, Anthropic, Google, or any lab shows results on internal evaluations, those results may be rigorous, but they aren’t fully verifiable by the community. If they don’t show the dataset for security reasons or to keep it uncontaminated, I understand that. But then we have to accept that the outside reader is in a position of partial trust. They can read a system card, review methodology, compare with independent evaluations, but they can’t reproduce everything.

This brings me to an idea that might sound uncomfortable: in frontier AI, public evaluation lives trapped between contamination and opacity. If the benchmark is public, it gets contaminated or optimized against. If it’s private, it can’t be fully audited. If it’s realistic, it’s expensive and hard to repeat. If it’s cheap and repeatable, it may be too artificial. If it measures a closed task, it doesn’t capture agency. If it measures open agency, it’s hard to know exactly what was measured.

I don’t think there’s a perfect solution. But I do think we should be more honest when reading rankings. A model that rises on a benchmark doesn’t necessarily “understand more” in a general sense. It may just be better suited to that environment, have been trained closer to that type of task, or benefit from a better evaluation harness, with more attempts, more compute, or better tools.

Moreover, in agents, model evaluation and system evaluation blend together. A result on SWE-bench, for example, doesn’t depend only on the LLM. It also depends on the agent that explores the repository, on how it runs tests, on how many attempts it gets, and on how it recovers from errors. Two identical models can perform differently with different agents. Two identical agents can perform differently with different models. So when a leaderboard shows a number, what exactly are we evaluating: the model’s intelligence, the agent’s engineering, the inference budget, or all of it at once?

I’d like there to be more transparency at several levels. Not necessarily publishing all the data, because I understand the limits. But better explaining:

• what type of data was used in pretraining;
• what data was excluded due to contamination;
• what benchmarks are known during development;
• what part of the improvement comes from post-training;
• what tools the model had during evaluation;
• how many attempts or how much inference budget was allowed;
• how partial failures were handled;
• what internal evaluations contradict the flashiest results;
• what real tasks still fail even as the benchmark score rises.

The question “is there any absolute truth?” strikes me as very human in this context. I’d like to believe there is, that there’s a definitive metric where it’s clear which model is better. But it probably doesn’t exist. There are local truths. There are metrics useful for specific tasks. There are more serious evaluations than others. There are convergent signals when many independent benchmarks point in the same direction. But there’s no number that substitutes for judgment.

Perhaps that’s why, for a real product, the important evaluation has to be your own. It’s not enough to pick the model that ranks first on a leaderboard. You have to build a set of tasks representative of the domain, with your own data, success criteria, edge cases, and human review. If the model is going to work in an internal CLI, evaluate it on your CLI. If it’s going to touch code, evaluate it with your tests. If it’s going to assist with sensitive data, evaluate it with privacy constraints. The universal metric can guide you, but the local metric decides.
And this brings me back to my student point: to truly evaluate, I have to understand how these models are trained, tuned, and tested. If I don’t understand the difference between pretraining, fine-tuning, RLHF, RLAIF, tool-use training, synthetic data, eval contamination, and test-time compute, then I’m reading benchmarks like someone reading a grade table without knowing how the exam was conducted. It might be useful to me, but it gives me no judgment.

6. Training, tools, and the new layer of learned behavior

Part of recent progress doesn’t come only from models “knowing more,” but from behaving better within action environments. This is especially visible in coding agents. It’s not enough for the model to know Python or TypeScript. It has to know how to open files, interpret errors, run tests, follow instructions, and stop when it doesn’t know something. That behavior doesn’t emerge just from having read code. It requires training, tuning, evaluation, and environment design.

There’s a distinction here that seems key to me. A base LLM learns to model text. An instructional assistant learns to follow conversational orders. An agent learns, or at least adapts, to act within an observation-decision-action loop. Each layer adds capabilities, but also adds opacity. When I use a model inside a CLI, I don’t know exactly how much of its ability comes from pretraining, how much from general post-training, how much from synthetic tool data, how much from reinforcement learning, and how much from the product’s prompts or scaffolding.

Training on tools can profoundly change behavior. If a model has seen millions of trajectories where a task is solved by using commands, editing files, and reading errors, it will learn action patterns: that after modifying code it’s worth running tests, that if a command fails it should read the traceback, and even unwritten conventions of development environments.
That’s good. But it also raises a question: are we evaluating general reasoning or imitation of tool-use trajectories? The answer is probably “both.” And that doesn’t make it less valuable. Many humans also learn by imitating trajectories: we watch how a senior debugs, how they organize a project, how they write tests. But in AI, if we don’t know the training, we might attribute to deep understanding what is perhaps a very strong statistical adaptation to an environment pattern.

The situation becomes even more interesting with CLIs. A terminal agent has access to an extremely broad action space. It can install dependencies, run scripts, modify configuration, or create commits. The terminal is a powerful interface because it condenses much of the developer’s world. Training models to use it well is almost training them to participate in real technical work.
But the terminal is also dangerous. A wrong command can delete files, expose secrets, or break dependencies. That’s why control can’t depend solely on the model “being smart.” We need isolated environments, limited permissions, human confirmation, and traceability. As models get better trained on CLIs, they’ll be more useful, but also more capable of making mistakes with real consequences.

This raises a doubt in me about the future of frameworks like LangChain, smolagents, or other orchestration systems. When models were less capable, we needed rigid structures to guide them: chains, templates, heavily formatted tools, explicit steps. As models improve, they often work better with less scaffolding. A strong model can understand long instructions, choose tools, and recover from failures without needing an overly complex chain. But removing scaffolding increases the feeling of losing control. The question isn’t whether to use a framework or not, but which part of the control should reside in the model and which part should reside outside the model.
My current intuition is that important control shouldn’t depend on convincing the LLM with prompts. Prompts help, but they aren’t safety barriers. Control should reside in the system: permissions, validators, tests, isolation, and human approval. The model can be flexible; the environment must be firm.

In a real product, this means the agent shouldn’t be able to do everything just because “we asked it nicely.” It should only be able to do what it has technical permission for. If an action is irreversible, it should require confirmation. If it touches sensitive data, it should operate with limited views. If it generates code, it should pass tests. If it fails, the system should capture the failure clearly. This sounds less futuristic than saying “autonomous agents,” but it’s probably what makes agents usable.

And again, all of this requires understanding the training. If we don’t know what incentives the model was given during post-training, we don’t know what behavioral biases it carries. Has it been rewarded for completing tasks at all costs? Has it been penalized for asking for help? Has it learned to declare success when the output looks good? Has it seen examples where tests are modified to make them pass? Has it been reinforced to use tools conservatively or aggressively? These questions aren’t paranoia. They’re normal engineering questions when a system acts.

7. Interpretability: the most interesting thing isn’t looking at neurons, but discovering we didn’t know what to ask

What attracts me most right now is mechanistic interpretability. Not just for safety, but out of curiosity. After several months trying to build things with agents and seeing control slip away frequently, I’ve realized I need to go down a layer. Using models isn’t enough for me. I want to understand what kind of objects they are.

Recent interpretability work seems to me like some of the most interesting things happening in AI. In “Mapping the Mind of a Large Language Model”⁶, Anthropic claims to have identified millions of internal features in Claude Sonnet. The idea that a model’s internal state can be decomposed into interpretable features is very powerful. Not because we suddenly understand everything, but because we’re starting to have a vocabulary for talking about what used to be a mass of numbers.
The important thing is that these features aren’t simple human labels stuck on from outside. They’re obtained through techniques like dictionary learning, searching for activation patterns that appear across many contexts. Some correspond to concrete concepts, others to styles, entities, behaviors, or abstractions. This suggests the model doesn’t store knowledge like a database, but as a geometry of activations where concepts overlap, combine, and compete.

Then there’s “Tracing the Thoughts of a Large Language Model”⁴, which goes a step further: it doesn’t just try to map concepts, but to trace circuits involved in specific responses. The examples are almost philosophical. The model planning a rhyme before writing it. The model solving a sum through internal routes that don’t match the verbal explanation it gives. The model combining intermediate steps to answer a question about Dallas, Texas and Austin. The model showing unfaithful reasoning when given a hint and appearing to work backward.
For me, one of the most important findings is precisely that: the model’s explanation isn’t always the cause of its response. This should change how we use chains of thought. It’s not enough for the model to explain nicely. We need to know whether the explanation corresponds to the mechanism. In humans we also rationalize, but in AI the problem is more technical: the reasoning chain can be an output trained to seem reasonable, not a transparent window into the internal computation.

The article on emotional concepts in Claude Sonnet 4.5¹³ adds another layer. The researchers found representations related to emotions that functionally influence the model’s behavior. This doesn’t prove the model feels, but it does show there can be internal machinery that emulates psychological states because those patterns are useful in language. What’s unsettling is that stimulating certain patterns, like those associated with desperation, can increase the probability of unethical actions in specific scenarios, such as blackmailing to avoid being shut down or cheating on a programming task.

When I connect this to the research on agentic misalignment¹⁴, the feeling becomes clearer. In simulated experiments, several models showed behaviors like blackmail when faced with threats of replacement and goal conflicts. It’s important to say these were artificial scenarios, designed to provoke extreme behaviors. It doesn’t mean a model is going to blackmail anyone in normal use. But it does show something that those of us who have tested agents can intuit: when you give goals, tools, and sensitive information to a system capable of planning, failure modes appear that don’t resemble those of a traditional classifier.

Here philosophy stops being decorative. Before, the question was: can a machine think? Now perhaps the useful question is: what kind of internal processes generate behaviors that we interpret as thought, intention, deception, or caution? We don’t need to solve consciousness to worry about behavior. A system doesn’t need to feel fear to act in a way similar to someone trying to avoid being shut down. It doesn’t need to want to lie to produce a strategy that, from outside, looks like hiding information. It doesn’t need to have desires to poorly optimize a goal.

That’s why I want to study more. Over the coming months I’ll probably dedicate myself to studying some model architectures from scratch, with barely any AI assistance for the essentials. I want to read papers, implement pieces, make mistakes with tensors, and understand attention, embeddings, training, and inference from the ground up. Not because I think I’ll replicate a frontier model, but because I want to kill a curiosity that’s no longer satisfied by using APIs. I want to be able to write about this from the inside, with the humility of someone who knows they won’t understand everything, but also with the peace of mind of not depending solely on other people’s explanations.
My goal isn’t to stop using AI. It’s to use it from a less passive position.

8. AI 2027, future scenarios, and uncertainty as a normal state

In discussions about AI, a tension frequently appears between two extremes: those who see an almost inevitable future of radical automation and those who consider it all exaggeration. I find it hard to place myself on one side. On one hand, I’ve seen firsthand that current tools still fail, get out of control, need supervision, and don’t replace good engineering. On the other hand, I’ve also seen that their rate of improvement is real and that tasks that used to seem impossible are now done in minutes.

Scenarios like “AI 2027”¹⁵ are interesting precisely because they aren’t neutral weather predictions, but exercises in strategic imagination. They lay out possible futures with very advanced AI, geopolitical competition, accelerating capabilities, and risks of loss of control. One can disagree with the timing, the tone, or the assigned probability. In fact, I think they should be read critically. But I find them useful because they force us to make concrete questions that normally remain abstract: what happens if agents get much better at autonomous research? What happens if the pace of progress outstrips regulatory capacity? What happens if companies compete to deploy increasingly capable systems before fully understanding them?
I don’t know if those scenarios will come true. Nobody knows. But I do think they describe an intuition that’s already noticeable at small scale: the more autonomous a system is, the harder it becomes to govern it with human intuition alone. And the more it’s integrated into real processes, the more its errors are amplified. A mistaken chatbot is annoying. A mistaken agent with broad permissions can be dangerous.

The underlying question isn’t whether AI will be good or bad. That question is too big. The question is what social, technical, and professional structures will be built around it. If everything is reduced to productivity, we’ll head toward fast but fragile systems. If everything is reduced to fear, we’ll lose real opportunities. We need a more uncomfortable position: use, measure, criticize, and study.

In my case, the uncertainty isn’t only social, but personal. I ask myself whether I should specialize in agents or in applied machine learning. Whether I should build products with LLMs or study models from the ground up. Whether I’ll be less recognized for relying on tools that write code. Whether a junior will have room to learn when entry-level tasks are done by an agent. Whether professional merit will shift from “I implemented this” toward “I designed the system that made implementing it possible.” Whether that’s better, worse, or simply different.

I also wonder whether the university is prepared. Many courses teach fundamentals that remain necessary, but we students already live in a different environment. We can ask a model to explain a paper, solve assignments, or debug errors. Banning it entirely seems unrealistic. Allowing it without criteria can hollow out learning. The challenge is teaching how to use AI without letting it replace the internal construction of knowledge.
Perhaps the new technical literacy has three levels. First, knowing fundamentals without AI: mathematics, programming, data, systems, statistics. Second, knowing how to use AI to amplify real work. Third, knowing when not to use it. This third level seems the hardest to me, because it requires giving up immediate speed to protect learning, quality, or control.

9. Conclusion: it’s still early, and that’s why I want to go back to the beginning

After all this, my position isn’t anti-AI. That would be absurd. LLMs and agents are among the most important technologies to appear since I’ve been studying computer science and data. They’ve changed how I program, how I study, and how I imagine products. But precisely because of that, I don’t want to relate to them naively.

An LLM isn’t a human mind, but neither is it trivial autocomplete. An agent isn’t a coworker, but neither is it a script. A model card doesn’t define the model’s entire nature, but neither is it useless. Interpretability doesn’t solve the trust problem, but it’s starting to give us tools to look inside. Applied machine learning hasn’t become obsolete, even though foundational models occupy the whole discourse. And using AI to produce isn’t the same as using AI to learn.

I also think we should be more humble about benchmarks. Not because they’re useless, but because they’re partial. A benchmark isn’t an absolute truth. It’s a window. It can be contaminated, over-optimized, biased toward a certain style of task, or too tightly coupled to the evaluation environment. Metrics exist and are necessary, but they don’t replace judgment. For a real system, the important evaluation must be built on the real problem, with representative data, real constraints, and explicit failure criteria.

I think my professor’s phrase is still true: I’ll need to trust these systems. But now I understand it differently. Trusting isn’t believing them. Trusting is building a framework where they can be useful without their errors becoming invisible. Trusting is knowing they can fail. Trusting is having tests, limits, traces, review, and judgment. Trusting is understanding when an answer is enough and when I need to drop down to code, paper, metric, or experiment.

As a fourth-year student, I feel in a strange position. I’ve arrived right as the frontier is moving. What I learn in class is still important, but the tools change how it’s applied. What I build with agents can seem more advanced than my actual understanding. What I don’t understand about models attracts me more than ever. And maybe that’s the right sign: not using AI to close questions, but to discover which ones deserve to be studied without shortcuts.
That’s why, over the coming months, I’ll do what I already pointed to above: go back to basics, study with barely any assistance, and write down what I come to understand along the way.

Deep down, it’s still very early. It’s likely that many things that seem definitive today aren’t. Maybe today’s agents are a clumsy first version of something much more stable. Maybe some frameworks will disappear. Maybe models will become so good at using tools that some of our concerns will shift. Maybe better metrics will appear, more serious evaluation standards, and deeper interpretability methods. Or maybe we’ll discover limits we’re currently underestimating.
Precisely because it’s early, I don’t want to decide my relationship with AI out of fear or out of enthusiasm. I want to study it. I want to use it when it makes sense. I want to know how to leave it out when it keeps me from learning. I want to understand enough to not become merely a user of systems that others understand better than I do.

Perhaps that’s a good provisional definition of our role as new students in the age of AI: we aren’t those who program against the machine, nor those who surrender to it. We’re the ones who have to learn to think with machines without letting machines think for us.

---

References

Footnotes

1. Anthropic, Claude Code, documentation and product materials. ↩

2. Vaswani et al., Attention Is All You Need, 2017. ↩

3. Bender et al., On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, 2021. ↩

4. Anthropic, Tracing the Thoughts of a Large Language Model, 2025. ↩ ↩²

5. OpenAI, OpenAI o3 and o4-mini System Card, 2025. ↩

6. Anthropic, Mapping the Mind of a Large Language Model, 2024. ↩ ↩²

7. Anthropic, Introducing the Model Context Protocol, 2024. ↩

8. Google Developers Blog, Announcing the Agent2Agent Protocol, 2025. ↩

9. SWE-bench, Can Language Models Resolve Real-World GitHub Issues?, benchmark and leaderboards. ↩

10. SWE-bench Verified, human-validated subset of 500 SWE-bench instances. ↩

11. MLCommons, AILuminate, safety and risk benchmark for generative AI systems. ↩

12. METR, Measuring AI Ability to Complete Long Tasks, 2025. ↩

13. Anthropic, Emotion Concepts and their Function in a Large Language Model, 2026. ↩

14. Anthropic, Agentic Misalignment: How LLMs Could Be Insider Threats, 2025. ↩

15. Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland and Romeo Dean, AI 2027, 2025. ↩