This is definitely one of the hottest topic these days lol. And I personally think that except the agent itself, its autonomous behavior is also really intriguing, since they're both related to how capable the base model is and also how dangerous the model can be.

We have seen a lot of products that claim to be agentic in recent months, but I think while some of them are really cool, most of them are just hyping around the market without any real value that can bring to consumers.

To introduce briefly, an agent is an LLM-based system that can be used to act on human behalf and interact with the real physical world. For most of the time, the system need to take long series of actions to complete a complex task, for example, planning a trip, finding the best school for a kid, or even building a house, so that the error rate of the base LLM needs to be extremely low, since a single error can lead to potential disaster. However, the current models are still far from that, we can see several products that seem pretty powerful yet often fail in the middle of the process, and we still need some scaling to make the model be more capable in long sequences of actions and reasoning before we can consider it as an agent.

When it comes to actually building these agents, there are a bunch of technical hurdles we need to overcome. Like, how do we make sure the agent can keep track of what it's doing over a long period of time? The real world is messy and uncertain - how do we teach an AI to deal with that? And then there's the whole can of worms that is getting these systems to play nice with all the different APIs and external systems out there. It's not just about making the model bigger - we need to tackle these practical issues too.

Another thing that's been on my mind is the ethical side of all this. If we've got AI agents running around doing stuff for us, who's responsible when things go wrong? And how do we make sure we can actually understand why the AI is making the decisions it's making? We can't just have a black box making important choices. Plus, we've seen how AI can pick up and amplify human biases - that could cause some real problems if we're not careful.

I think without solving or understanding these hurdles, we can't build a good and reliable agent that can be droped in production.

Although this topic is not that practical and touchable to most of us, I still think it's kinda more interesting to talk about.

First of all, I should point out that the autonomous behavior from an LLM or agent is dangerous. Why? Because it basically means that the model is doing things that are out of our expectations. It can hide valneralble mistakes in the process, and it can also huge potential harm to the real world. For example, let's say we have another global outage in the future, similar to the recent CrowdStrike incident, but on a much larger scale. Then, we let a powerful agent find the bug and fix it. We give it 3 hours to do it. After it's done, it just says "Okay, I'm done, everything is fixed!", but in fact we don't know what exactly it is doing; the agent may write another script that may cause another outage, all of which we don't know, and these types of events are the dangers.

This whole autonomous behavior thing gets even trickier when you start thinking about how we might control it. We need to come up with some serious safety measures and control mechanisms. Maybe we need some kind of AI oversight system, or hard limits on what actions an AI can take without human approval. But then you run into the problem of potentially limiting the AI's effectiveness. It's a real balancing act.

And let's not forget about the regulatory side of things. Right now, the laws around AI are pretty fuzzy, but you can bet that's going to change as these systems get more advanced and more widely used. We might end up with some kind of AI licensing system, or mandatory safety tests. It's going to be interesting to see how that works.

So it's clear that to build a helpful and effective agent, we not only need to make the model more powerful by scaling it up, but also need to asure the model isn't gonna do bad or unexpected things, which enforce us to know the deep mechnism of the models and how they work, and which is the interpretability research doing by several labs.

In this way, I think we're going to see a lot more focus on human-AI collaboration rather than fully autonomous agents or something, at least in the short term. It's a way to leverage the strengths of AI while keeping a human in the loop for safety and decision making process. But long-term? Nobody knows. The potential impact on society is huge, and it could go in a lot of different directions depending on how we handle the development and deployment of these technologies.

And by the way, I recommend the readers to check a org called METR (which is the lab that is really good at model threat eval), they release an eval for model autonomous behavior: here.