This is my attempt at opening up the next few years - what I expect to see, mainly around agents. I’ve done some reasoning on what will keep happening this year and beyond, and I also took a sneak peek into the farther future. I know I’ve done this kind of thing several times before, but I think it’s still worth redoing since AI is evolving so fast. Things look pretty different from even a year ago.

2026

For 2026, I think one of the main topics will still be agents, but they will be more personalized and truly useful agents.

Proactive agent

The whole journey has evolved like this: first we had chatbots, only used for conversations. Then people wanted them to access external data, so we got basic tool use - searching the web, finding live data. As models became more powerful, they could handle more tools and become more robust. Now, we gave them reasoning capabilities, and they became agents.

Expectations are growing. People want agents to do more personalized things, to be more useful, to know more about them. But the limitation is clear: current agents only do tasks explicitly requested by humans. They lack self-initiating ability. We say agents will help us take actions and save time, but the reality is that current agents, whether text-based, GUI-based, or combined, are slow. After you send a request like “shop some groceries for me”, you need to wait a while. That’s counter-intuitive to how we want them to save us from tedious daily work.

To improve this, agents should really be able to do tasks on its own; in another words, they have to understand how we use them, and then use that understanding to do tasks in the background without waiting for humans to initiate and intervene. They need to prepare what we want in advance. This makes me think of them as a more advanced form of autocomplete. The only difference is the scale of the task. Ordinary autocomplete, like Cursor’s Tab mode, handles lines of code across files, while task-based agents, like Manus, handle entire tasks.

To get there, they need to learn your usage patterns. For example: the agent knows you always ask it to summarize your email on Monday morning, so it automatically does that in the future; or it knows when you’re running low on groceries and handles that for you.

But we also need to make sure they’re not annoying, which means timing also matters - it shouldn’t be too intrusive or too hidden. Otherwise it’s useless. This means the UI and UX need to change. We can’t just have input-box-only interfaces anymore. Gmail’s AI Inbox from last week is a good example: it didn’t fundamentally change how you interact with Gmail, but it added AI features that actually boost productivity. AI-powered tools don’t necessarily need an obvious input box; they should be bound to the task context itself.

If this develops well, it will significantly boost people’s productivity with agents, and people will start believing in them more.

Memory

The second key piece is memory. People are giving models higher expectations, and models need to know users better to feel genuinely useful. This ties back directly to what I mentioned about proactive agents.

Currently there are a few general solutions for memory. Looking at it from a product perspective, there are basically three types:

1. Model uses a tool to store things into a memory space (ChatGPT, Gemini, Claude, Kimi, Qwen, etc.)

2. Model uses a conversation search tool to find specific topics from past chats (Claude, ChatGPT)

3. System summarizes user interactions daily, then extracts new information into detailed summarized memory (Claude)

These are pretty good, and I’ve seen promising performance from products like Claude with their memory system. But for broader general agents, it can, and should, be better. Memory isn’t just limited to basic information about us, it’s connected to our general preferences across life: shopping style, coding style, travel style, and a lot more. These affect how well an agent can complete tasks within your expectations. But it’s annoying to repeatedly mention or re-state preferences. So how products “form” these memories needs innovation too.

One approach I’ve been thinking about: hand off your apps and websites to an agent to explore first. An agent will always be better at learning your preferences than asking you to describe them, and you definitely don’t want to repeat yourself over and over; so you give the agent login access, it browses and checks though your previous orders, learns and summarizes your preferences, i.e. what do you usually order for groceries? which airline do you always prefer? The agent then summarizes these into specialized documentation. Each time it goes to that specific app or website, the relevant instructions load automatically, ensuring the model already knows what it needs to know. This doesn’t require any special model capability - it just needs the product or environment (”agent harness”) to be optimized to push what the model knows further.

The same approach would apply to many other use cases. And unlike recent skills or similar features, this doesn’t require user or model to pay extra attention. There won’t be cases where the model ignores specific preferences, because they’re loaded by default.

The above are all product-based, but we could also think from a more foundational side. Sometimes models don’t realize the importance of using user knowledge, so they just skip it. (I should note: my ideas here may be wrong since there’s no clear experimentation showing these work yet.)

1. We could rely on SAEs (sparse autoencoders) from mechanistic interpretability. Anthropic has used these in some of their research. Generally, SAEs can find activated feature points inside a model when it’s generating specific tokens. If we could use this technique to detect a model’s tendency to seek external knowledge, including user memories, then when that tendency is high, we could auto-inject relevant knowledge after that token. The model receives it and generates more useful responses.

2. We could use fewer, more specialized experts in MoE models. For example, a model with only three or four experts, each for a specific action: one for thinking/reasoning, another for tool use, and a final one for responding. Maybe one more for orchestrating which expert to use at each step.

There could be more innovation in memory on the model side.

Either way, we’re going to see a lot of surprises around proactive agents and memory. The key question now is how models can really boost productivity - because I think that’s where they can bring the most economic impact before they can truly impact society at large, they should first have huge, noticeable impact on individuals.

Trends

Also, I think there would be some continuing trends that will keep happening or starting to shift in next year or two.

Model as a product

The first continuing trend is model as a product. This has been a long-running pattern; and I think it has two slightly different side:

1. Model having unique abilities that can directly become a new product or feature (like GPT-Image, Nano Banana, Sora 2, Genie 3, etc.)

2. Model having is quite strong that people can build geneneral products around it with some engineering work (like early Manus on Claude-3.7 Sonnet, Claude Code, etc.)

Among these, I think the new Genie 3, Google’s latest world model, we got in public last week has huge potential. You can create the world, control how you “walk around” inside it - the whole thing is customizable. It’s going to be much more fun than video models like Sora. And since it can generate interactive worlds, it has potential to become one of the first reliable generative games. I didn’t play many games before, but if we get solid products based on robust world models, I might start - creating my own experiences sounds really fun lol. Some examples I saw on X:

Agent capacity

The second trend is agent capacity. Models will become more robust, that’s the clear trajectory. They’ll handle more long-tail tasks, able to do even complex tasks. They would help humans more in accelerating not just SWE work but also AI research itself; maybe even automate some of it. We’ve already seen huge potential here, both in scientific research and other areas. And we have benchmarks tracking this, like METR Time Horizo, VendingBench, and many more.

The curve is going up and will continue pretty steadily.

Model alignment

The third, and one of the most important, is model alignment. As models become more capable and people put them into more production environments, the consequences of bad intentions become catastrophic. If a model is able to help scientist in building nuclear fusion reactor, then it can help bad people build nuclear weapons; if a model is able to help companies develop madicine, then it can create bio-weapons as well; knowledge are connected anyway. I had wrote about my thoughts on this before, and there’re a lot of researches on this, but one approach I find promising is the new Claude Constitution. OpenAI’s Model Spec is similar, but more rule-based: what you should do, what you shouldn’t. The Constitution is more about teaching the model how to be good and do good things - less like rules, more like parents teaching a child (I remember Dario described it as a letter “from a deceased parent sealed until adulthood”). I think this is a promising direction, and I expect more companies to explore this kind of approach.

Human-AI interaction

The last thing is how human-AI interaction will change. Right now, we interact with AI through apps, APIs, websites - all limited to mobile and desktop. I think a really good new portal is AI glasses, because they can see what you see, hear what you hear. And they can have their own ecosystem position - they don’t need to replace phones or anything else. They can add something new: a different way to interact and co-live with AI; unlike Humane AI Pin or Rabbit r1, which tried to replace the phone and couldn’t.

Since AI glasses can sense almost everything we can, they’d be a great add-on for the proactive agents I mentioned. They could recommend things or help complete tasks based on your real-world environment. Better memory systems become important here too.

We’re already seeing some products. For example, Pickle 1 looks kinda promising - I’ve already pre-ordered, waiting to see how it goes. And it seems Google is working on something too, mentioned by Demis at Davos 2026. But these are still early.

We can focus on the previous parts for now; the glasses stuff is more a matter of hardware, software, and ecosystem catching up.

Future

I’ve written about the future many times before, but AI moves so fast that things look quite different from even a year ago. So I think it’s still worth sharing what I expect for the farther future - I have some new thoughts after seeing recent posts, interviews, and doing my own thinking.

Before I go further, I should mention Dario’s new essay, The Adolescence of Technology. It’s a serious piece that maps out the risks we’re facing and how we might address them. I have a lot of respect for how he approaches these problems - careful, concrete, not as a doomer. If you haven’t read it, I’d recommend it. What I’m writing here is more of a personal take, from someone who will live through this transition; and again, it’s all my personal thoughts, so can be incorrect.

What I want to see

The world on the other side: one where survival anxiety isn’t the default mode of human life. Where scientific progress in medicine, climate, and longevity happens way faster than before. Where people can pursue what actually matters to them, not just what pays bills.

Dario calls this “Machines of Loving Grace”. I think he’s right about what’s possible. The real question is whether we can get through the middle part without everything falling apart.

I’ve imagined this good future a lot. Robots handling physical labor. Abundance making material scarcity less relevant. People free from the constant pressure of “making a living” and able to actually live. It sounds utopian, but I don’t think it’s impossible - just hard to get to, and need a lot of efforts.

Some hard questions

If AI creates more value than you, what’s your purpose?

This will be the lived experience of a lot of people soon. Companies will do the math: AI is faster, cheaper, better. The rational move is to let people go. If that happens at scale, the whole “AI benefits humanity” thing falls apart. You can’t really benefit from something that made you economically irrelevant and gave you nothing back.

I think to prevent this, companies and society need some kind of common understanding: even if AI creates more value, we should still preserve humans in the foreseeable future. After a company takes what they need for operations, they should return that value to the workers who got replaced, which should be more like a social contract. The value came from somewhere.

This is really hard to execute. No enforcement mechanism, no clear policy, and competitive pressure pushes against it. But that’s why the journey is difficult. The tech is arriving faster than our social systems can adapt; that’s why I said, it should be us to adapt the development of theses advanced systems. We’ve almost never seen these together, and our existing frameworks aren’t built for it.

Meaning without work

Even if we solve the material side - even if displaced workers get income - there’s still the meaning problem. People don’t just want stuff. They want to matter, to be needed. Work used to provide that, even when the work itself was boring.

I’ve thought about this a lot, and there are a lot of discussions out there. In a world where AI handles most cognitive tasks, we’ll need new structures for purpose. Creative work, community, exploration, caregiving - things that matter to us even if they don’t maximize GDP. But this won’t happen automatically, we have to build it intentionally.

Maybe this sounds abstract, but it’s actually pretty concrete. What would you do if you didn’t have to work? Not vacation-mode “what would you do”, but actually, long-term, what would give your life structure and meaning? For me, I think it’s exploring unknowns, experiencing different places, maybe creating things. But a lot of people haven’t had the chance to even think about that question, because survival comes first.

The transition will force us to answer it, and I think the answer will be different for everyone, which is kinda the point. Freedom to figure out what matters to you, rather than having it dictated by economic necessity.

The transition itself

I think it’s quite obvious that this transition won’t be peaceful. I’ve said before that millions will lose jobs, and society might break down in parts. That’s what history told us, the industrial revolution caused massive suffering before things got better. This could be similar, but faster and broader.

The question is whether we can make the transition as humane as possible. Not “acceptable sacrifice for progress” - that framing has been used to justify a lot of harm historically. More like: we acknowledge it will be hard, and we try to take care of each other through it.

Why I’m still optimistic

I know the risks are huge. I’ve read a lot of doomer takes, and I get where they’re coming from. Powerful AI in the wrong hands, misaligned objectives, societal collapse, and a lot more; these are indeed real.

But there are a lot of researchers working on alignment and interpretability. Some companies (like Anthropic and DeepMind) actually taking safety and relevant problems seriously. The new Claude Constitution trying to teach models to be good, not just follow rules. People having these conversations instead of ignoring them. That matters.

I’ve thought about how to hold both things - the hopeful vision and knowing that getting there will be rough. Honestly, it comes down to something simple: I believe our world can be much better, and I want to see that happen. Maybe help build it. That’s the faith I keep.

The years ahead will be hard, maybe it will take us decades. But I keep coming back to: so what? why be afraid?

There’s still more to write, but I think this is enough for now, I’ll save the rest for a future post.

Anyway, I wish the world getting better and better in 2026 and beyond.

For some time, I have been thinking about how we could push the frontier of current reasoning models forward, not just their performance, but also other “use-cases” or “features”. So I came up with these two questions, which haven’t been mentioned by others (maybe?):

1. How could the current reasoning model paradigm affect or enhance multimodal models?

2. How could so-called “hybrid reasoning models” (possibly) work?

And around these two topics, I have some personal thoughts. I’m not aiming to be definitively correct, but just rather to share my own opinions.

1. How could the current reasoning model paradigm affect or enhance multimodal models?

People are shocked by how Gemini-2.5 Flash Image (aka Nano-Banana) performs: image generation, editing, and so much more. A lot of models today could do more than just output text, they could also create audio and image. So I’m thinking of whether we could integrate those multimodal models with the reasoning models. Instead of only thinking in text, could they also think in image and audio? If we could let a reasoning model to use multimodal content not just in its final response but also during its internal thinking, the results could be much better as well. This may sound confusing, so here’re a few examples:

• audio modality: A user is preparing a speech and wants the model to generate an audio sample. gpt‑realtime or gemini‑2.5 may help, but the quality cannot be ensured. If the model can reason about what emotions to convey at specific points, produce draft audio to simulate, and iterate, the result quality may improve. For example:

user:
	I’ll have a speech about the future of our planet earth next week at the UN. Could you please give me a sample for it?

model (CoT):
The user is asking: “...” So they want a sample speech. 
	I could give a text script, but it will not be that helpful. I guess give them an example audio would be better. It may also require some emotions. I should first get up structures and contents first.
	...
	Alright, so I am done with initial composing, let me explore how emotions should be, and I will do some drafts, then it will good to go..
	Well, I should have hope, urgency, inspiration, responsibility, solidarity, empathy, ...
	OK, so for hope, I think I should use upraising tone, just like this:
	[a piece of voice with upraising tone]
	Cool, sounds good, let’s continue:
	...
	Alright, I think I got all emotions done. Let me make a first draft:
	[full composed speech with correct emotions]
	...

model (response):
	Here’s the speech I prepared for you:
	[full final speech audio]

• image modality: Although the latest Nano‑Banana can create or edit amazing images, it still struggles in complex scenarios. This makes sense because the model outputs the image in just a single shot. If it could take some time to think and draft first, like humans do, the results might be better. For example:

user:
	Please create a cinematic image of a interstellar war. it needs to be epic.

model (CoT):
	The user is asking me to generate a image of ...
	So let me first imagine the scene: a massive fleet of sleek starships engaged in battle over a luminous nebula, laser beams and plasma torpedoes streaking across space, explosions creating bright fireballs, a giant battlecruiser in the foreground with detailed hull, background with distant stars and a planet partially visible ...
	Cool, let me create a sketch first:
	[a sketch of the scene with only few lines]
	...
	I see the planet in the image doesn’t seem real, let me change the position:
	[a final-prepared version of the image with all elements and colors]
	...
	Great, it seems perfect, let me give this to the user.
	
model (response):
	Here’s the image you want:
	[the final version of the image]

• or even video modality (I know there’s no language model that could output both text and video, but maybe it is a good idea)

  Today’s video generation models, like Veo-3, are great as well; however, they all have some huge weaknesses: short generation, not able to comply with complex story, etc. So giving the model the ability to think in audio, image and more, could enhance it greatly. For example:

user:
	please give me a short movie about space exploration.
	
model (CoT):
	The user asks for a short movie ...
	I plan to write the script first, then draw some certain frames, create some suitable BGMs. This seems good. Let me proceed with the some ideas.
	...
	Phew. I think this is amazing, user is gonna love it. So next, frames.
	For the opening, I think this would be good:
	[a sketch frame for the first part of the story]
	...
	That’s great. I’m all done. Now, music.
	[some background audios for specific frames]
	...
	Sounds good, then let me go for the final video part.
	...
	
model (response):
	Here’s the movie you want:
	[final movie]

2. How could so-called “hybrid reasoning models” (possibly) work?

So the definition for hybrid reasoning models is that the models which can either respond directly or think deeply before respond (or even able to decide when to think more on its own). And currently, there’re only a few models have such ability: Claude-3.7, Claude-4, DeepSeek-v3.1, Qwen-3, and some more (GPT-5 doesn’t count for now, because it has a router).

Claude is close-sourced, so we don’t know how their thinking toggle actually works (maybe similar to other models). For DeepSeek-v3.1 and Qwen-3’s non-thinking mode, they are just prefilled with a blank thinking block (like <think> </think>). This is a good and quick way to let the model skip thinking and respond directly, but…. well, seems like the result may not be that satisfying for Qwen team, and they separated thinking and non-thinking mode into two models soon (here).

But, what if, we let the model always think, but for different reason? Here’s what I mean:

In basic sense, we could train the model to know how to react with different setting (thinking mode on, off, or auto). In inference time, we will let the model know (like via system prompt) what user pick. And in different mode, the model will always take a look at the current setting, and decide what it needs to do on its own. This means, there will be fewer manual intervention, the model just knows what to do.

Example behavior:

• Thinking mode on:

system: ... <thinking_mode>on</thinking_mode>

user:
	...
	
model (CoT):
	Let me see. I saw the thinking mode has been set with on, which means I should take more time to continue thinking afterwards.
	The user asks ...

model (response):
	...

• Thinking mode off:

system: ... <thinking_mode>off</thinking_mode>

user:
	...
	
model (CoT):
	Hmmm... I see the thinking mode is off. This means I should start responding directly. Yes, no more thinking. Start to respond now.
	
model (response):
	...

• Thinking mode with auto:

system: ... <thinking_mode>auto</thinking_mode>

user:
	What is 1+1?
	
model (CoT):
	I see an auto thinking mode is been put. So basically I just need to decide how much I need to think. Hmmm... Let me see.
	The user asks: “What is 1+1?” This is trivial: 2. Any other things? No. Just a number is fine. Respond now.

model (response):
	2.

• or

system: ... <thinking_mode>auto</thinking_mode>

user:
	...
	
model (CoT):
	... Oh my god, this is hard. According to my setting, I guess I should take more time to think further for the message.
	The user asks ...

model (response):
	...

And we could also use RL to let the model learn such behaviors:

1. For the general “react-to-setting” behavior, we could set a verifier that look for whether model really correctly react to model’s setting, because the model may usually use similar phrasing or meaning when reacting to the setting.

2. For whether model correctly react to the setting (like stop thinking when off, continue when on), we could reward/penalize the model with its afterward behavior, like if model still insists in thinking forward when the setting is set to off, penalty would be casted.

3. For the auto thinking, I think we could use a prelabelled dataset with “easy/hard” labels, then reward on the output that correctly deal with the question under auto setting.

This may not really work, but may be a path that worth exploration. Why? Take a look at OpenAI’s o-series models and GPT-5-thinking model. Their reasoning effort are all been controlled by internal parameter called “juice” (GPT-5 even has a param called “oververbosity” which is for final response verbosity control). Also, Anthropic uses something like <max_thinking_length> to tell the model how long it should think in all. So via curated data and RL training, the model may also gain ability in thinking more adaptively and efficiently.

The past thirteen months have been great, with a lot of good things happening. We’ve made huge progress across models’ multimodal, reasoning, and agentic abilities, which are all important components on my own imaginary roadmap to capable AI systems that would have a huge impact on our species (or what people call “AGI”).

On multimodal model

This is an interesting topic. The reason for its importance is that I strongly believe letting models “feel” the world in many different ways is an important key to helping them better understand physics, the world, and the whole universe — text does not include everything in “language”; “language” is diverse, it’s much richer than text.

Currently, I think the existing tokenizer is what limits the model from moving forward. In fact, there are many simple tasks that we, as humans, would definitely not get wrong; however, even the strongest LLM currently available (i.e., o1-pro) would easily get stuck. For example:

So I think we have to come up with a real multimodal model that could get rid of those limitations of the current visual encoders/tokenizers and really understand the image. This is a SUPER basic ability that models need to have.

Beyond multimodal input, we have multimodal output. This showed up in GPT-4V back in May and the new Gemini 2. The reason I think it is also pretty cool is that it’s better than letting a model write prompts and use DALL·E or Midjourney to create images, because there are a lot of limitations with traditional text-to-image models. They sometimes get stuck with complex things, and they don’t understand what they are drawing. However, the models with true multimodal output can know what they need to generate, and humans can let them iterate on those results. What’s more, we could do more fun things with such ability, like:

Pretty cool, right? And since you can actually let the model generate or edit a picture for you, everyone can do PS work - they don’t need to actually have such expertise, which could be super convenient.

On reasoning models

This is the hottest topic from the last few months, and I’ve already written about it in last August. Up till now, we have several reasoning/thinking models in hand (o-series model, R1, Gemini Thinking models and a lot of others from research).

Thanks to RL, progress is really fast; for example, from o1 to o3 with roughly about 3 months, the model is able to solve a bunch of AGI-ARC tasks, and we could expect more crazy things in the coming months.

I think the idea of giving the model more time to respond is great. However, sometimes the model will have an overthinking problem, which is sometimes time- and compute-consuming. For example, when you ask R1 “1+1”, it will think for seconds (~100 tokens):

So that’s why I would say the model being able to control when they need to think is also another important ability, which may be the next focus for researchers. But before that, we need to make general reasoning ability (beyond math and coding) better. BTW, in the blog I wrote last year, I mentioned system 1 and system 2 thinking patterns of humans. I still think that would be useful, though directly applying that into the current models would not be feasible; we could still borrow some ideas.

Besides, another interesting thing mentioned in R1’s paper is that the model sometimes uses mixed language in its thinking process. I think as we continually scale RL and test-time compute, we would even see models generate nonsense or scrambled text, while the result will not be affected at all. I think that would be a time when we say, “Okay, RL just works.” (But this would be a total disaster for Anthropic and some AI doomers lol :P)

On agents

Besides reasoning, I think this is another term that everyone addicts to use. I still remember that last year almost all products said they had some “AI agents” stuff (and I blacklist every product that says so).

There are only a few real agents in my mind, like Project Astra from DeepMind, Operator and Deep Research from OpenAI. These tools are really the AI systems that could take reasonable actions for you.

Here, the definition I gave is that only when you have a good reasoner or your model can reason well, then you can call the tool or the system built upon your model an agent. I think that should be something that we should expect, instead of those weird and fancy tools that you click, then summarize some emails or things like that.

Although the products that claim they have agents in the past year are not acceptable to me, their idea is somehow kind of good—what they need is a better base model, i.e., o3-mini: fast, cheap, and capable.

Another core feature that would really push these agents forward should be in-thinking tool use. When o1 with tool use was released, I was concerned about whether o1’s tool use was like thinking, calling a tool, then responding directly; but now with o3, my concerns have vanished. o3’s tool use flow is like thinking, tool-using, rethinking (maybe with another few turns), then responding. In fact, I saw great benefits from this pattern in o3-mini with web browsing and Deep Research powered by fine-tuned o3. And I expect to see more agents from OpenAI and from other research labs.

Phew~ that’s all I wanna say. January is just a starting point, and we’re gonna have a wild ride in upcoming months! Just buckle up for it.

This blog is inspired by Dario’s Reflecting on Dario’s “Machines of Loving Grace”

Disclaimer: I’m not a professional in any of the domains mentioned below. My conclusions are drawn from my observations and my understanding of AI, so errors are inevitable.

I think the most inspiring term in the whole blog would be “country of geniuses in a datacenter”. I can’t think of any term better than this for describing the powerful AI.

So within my limited knowledge capacity, I would envision how our future with AI would be in two aspect:

1. Life with AI: Our life with AI, I think, would definitely be more convenient. Let’s first talk about just the models. Starting from this, we can see that a lot of things can be accompanied and assisted by AI. Because I think as AI becomes more capable, we should increasingly consider it as a companion, not just a tool. It’s like an assistant that helps us every day and knows what we know, sees what we see, hears what we hear, and helps us plan what we plan. Our way of interaction with machine is changing.

   Another aspect is robotics, which would be another part making our lives more convenient. Just look at what Elon showed us at Tesla’s “We, Robot” event. I think it was a fantastic event, and many people posted videos about Tesla’s Optimus robot from the event. And I think it’s so powerful. It can answer questions, dance, and even give you drinks. I think it’s incredibly convenient, and as its price decreases, it will become generally available in everyone’s home, just like we have televisions. And our family robots could help us do many things, i.e. houseworks, which can help us save time for doing something more meaningful.

2. Science with AI: Let’s move on to discuss how AI will change the current paradigm of scientific research. Looking back over the last century, we’ve seen numerous breakthroughs in fields such as biology, chemistry, physics, and a lot more. Notably, if you’ve been following the 2024 Nobel Prizes, you’ll have noticed that both the Physics and Chemistry awards were given for AI-related achievements. However, I believe this is just the beginning.

   This trend indicate increasing number of people recognizing how AI will impact scientific research, and wider acceptance of applying AI into science. Inevitably, I think AI will have its own profound impact on science. As I’ve written before, science with AI will be much more advanced than it was previously, and the innovation and research will continually speed up. A lot of new progresses will be made. The future of scientific research with AI integration will be fascinating, and I’m eagerly looking forward to it.

   So in this way, I very much agree with what Dario said that we’re going to shrink the research progress of the next 50 to 100 years into 5 or 10 years. I think this assumption is pretty accurate; however, since I’m just a normal person without any insights about the progress inside those companies, I cannot comment much on this. But anyway, I’m still quite optimistic about it.

I believe we will have a future where AI is seamlessly integrated into our daily lives and all other aspects of society. In this future, many global problems present today, like diseases and global warming, will possibly be addressed. The boundary between artificial and human intelligence will gradually blur, as there would be no discernible difference. Eventually, this will foster a symbiotic relationship that drives unprecedented advancements in science, technology, and social progress. As we navigate this new era together, our harmonious coexistence with AI will guide us along the path...

Good time is coming, just STAY ALIVE.

This time, OpenAI’s latest o1 family models (o1, o1-preview, and o1-mini) are indeed very powerful, with remarkably impressive performance. I think the most noteworthy points are: 1. They possess extremely strong logical reasoning abilities; 2. The models come with built-in CoT (Chain of Thoughts), requiring minimal user prompting.

I consider these two points very important because in the past, when faced with complex mathematical problems, these language models were often just “guessing answers” rather than truly reasoning step by step. But this time it’s different. OpenAI has specifically integrated RL and CoT based on GPT-4o, as well as added special “Reasoning tokens”, making the model truly “think”.

For example, when I asked o1-preview and o1-mini to calculate 279563 multiplied by 356104, they were both able to “think” first, self-reflect and correct during the thinking process, and then give the answer. Upon verification, both were correct. In the past, this task would have yielded completely incorrect results from LLMs - they would either guess or make fatal logical errors between steps. The same improvement was evident when I gave them the most challenging math problems in 2024 Gaokao in China (check result here). Additionally, I tested them with several questions from the national competitions this year, and the results were also very good. So we can see that this iteration of models performs very strongly on reasoning-related tasks. Logically rigorous and self-consistent reasoning is a necessary condition for us to reach the next level of AI, namely Agent, after all, the latter needs to be able to act on behalf of humans. We can’t allow them to make any mistakes, otherwise, it could lead to catastrophic consequences. (Read more about Agent and Model Autonomy here)

Moreover, we can notice the prompt “thought for x seconds”. For instance, o1-preview’s “thinking” time is relatively long, while mini’s “thinking” time is shorter (because the latter has been specifically fine-tuned on competitive math problems). I think the potential this brings is limitless. Now it’s “thinking” for a few seconds or minutes, in the future it will be “thinking” for months, making more complex reasoning and analysis, and obtaining more accurate and logical results.

These two phenomena remind me of System 1 and 2 that I mentioned in this article I wrote about possible future improvements in model reasoning abilities. These two concepts were originally used to describe human brain thinking, but now I see that o1 also has this characteristic. By definition, System 1 is responsible for intuitive fast thinking, like 1+1=2; System 2 is responsible for complex thinking that requires reasoning, such as complex mathematical problems, etc. More precise thinking can yield better and deeper results. This reminds me of another article I wrote, where I mentioned that if future models could reach the level of Nobel Prize winners, we could have hundreds of such AI copies form a research group and give them months to “think” and conduct research. Now it seems that models are already very strong in logical reasoning, biology, and other fields, so I believe the probability of this phenomenon occurring is very high. I’m looking forward to seeing AI assist humans in developing important drugs, discovering new materials, and even proving mathematical theorems in the future.

The future of humanity looks bright at the moment.

Frankly speaking, I don’t think that the LLMs can’t reasoning; in fact, I think there can be three possible ways to help them get this ability:

1. Scaling is all we need: We could literally do nothing to the current LLM, what we need to do is just to continue scaling (compute, data, and model size), and just let the model learn and understand the underlying logic pattern and syntax inside the training data, as the model becoming more and more complex during scaling; then we just wait for the “miracles” to happen, but it won’t happened till the end of the possibly exponential scaling curve;

2. Human’s thinking pattern is all we need: We could try to apply the System 1 and 2 thinking to the current LLM: System 1 thinks fast and intuitively, best for quick decision, similar to the current LLMs; while System 2 thinks slowly and deliberately, which can be the perfect system for LLMs to solve all those kind of complex and reasoning-required tasks.

3. Tree search is all we need: We could implement the tree search into the current LLM, and we have seen deepseek-prover-v1.5 succeeded with MCTS and AlphaProof-2 succeeded with hybrid approach with tree search usage; both of these achieve great results, which means it is effective for model’s complex problem solving ability. (since I’m not quite familiar with this, I really recommend “The Bitter-er Lesson” by Aidan Mclau, it can be really helpful to get the importance and potential of tree search!)

For agent stuff:

We have seen a lot of products that claim to be agentic in recent months, but I think while some of them are really cool, most of them are just hyping around the market without any real value that can bring to consumers. To introduce briefly, an agent is an LLM-based system that can be used to act on human behalf and interact with the real physical world. For most of the time, the system need to take long series of actions to complete a complex task, for example, planning a trip, finding the best school for a kid, or even building a house, so that the error rate of the base LLM needs to be extremely low, since a single error can lead to potential disaster. However, the current models are still far from that, we can see several products that seem pretty powerful yet often fail in the middle of the process, and we still need some scaling to make the model be more capable in long sequences of actions and reasoning before we can consider it as an agent. When it comes to actually building these agents, there are a bunch of technical hurdles we need to overcome. Like, how do we make sure the agent can keep track of what it’s doing over a long period of time? The real world is messy and uncertain - how do we teach an AI to deal with that? And then there’s the whole can of worms that is getting these systems to play nice with all the different APIs and external systems out there. It’s not just about making the model bigger - we need to tackle these practical issues too. Another thing that’s been on my mind is the ethical side of all this. If we’ve got AI agents running around doing stuff for us, who’s responsible when things go wrong? And how do we make sure we can actually understand why the AI is making the decisions it’s making? We can’t just have a black box making important choices. Plus, we’ve seen how AI can pick up and amplify human biases - that could cause some real problems if we’re not careful. I think without solving or understanding these hurdles, we can’t build a good and reliable agent that can be droped in production.

For model autonomous behavior:

Although this topic is not that practical and touchable to most of us, I still think it’s kinda more interesting to talk about. First of all, I should point out that the autonomous behavior from an LLM or agent is dangerous. Why? Because it basically means that the model is doing things that are out of our expectations. It can hide valneralble mistakes in the process, and it can also huge potential harm to the real world. For example, let’s say we have another global outage in the future, similar to the recent CrowdStrike incident, but on a much larger scale. Then, we let a powerful agent find the bug and fix it. We give it 3 hours to do it. After it’s done, it just says “Okay, I’m done, everything is fixed!”, but in fact we don’t know what exactly it is doing; the agent may write another script that may cause another outage, all of which we don’t know, and these types of events are the dangers. This whole autonomous behavior thing gets even trickier when you start thinking about how we might control it. We need to come up with some serious safety measures and control mechanisms. Maybe we need some kind of AI oversight system, or hard limits on what actions an AI can take without human approval. But then you run into the problem of potentially limiting the AI’s effectiveness. It’s a real balancing act. And let’s not forget about the regulatory side of things. Right now, the laws around AI are pretty fuzzy, but you can bet that’s going to change as these systems get more advanced and more widely used. We might end up with some kind of AI licensing system, or mandatory safety tests. It’s going to be interesting to see how that works. So it’s clear that to build a helpful and effective agent, we not only need to make the model more powerful by scaling it up, but also need to asure the model isn’t gonna do bad or unexpected things, which enforce us to know the deep mechnism of the models and how they work, and which is the interpretability research doing by several labs. In this way, I think we’re going to see a lot more focus on human-AI collaboration rather than fully autonomous agents or something, at least in the short term. It’s a way to leverage the strengths of AI while keeping a human in the loop for safety and decision making process. But long-term? Nobody knows. The potential impact on society is huge, and it could go in a lot of different directions depending on how we handle the development and deployment of these technologies. And by the way, I recommend the readers to check a org called METR (which is the lab that is really good at model threat eval), they release an eval for model autonomous behavior: here.

Frankly speaking, while the idea of having AI agents to handle complex tasks for us seems really cool, we’ve got a long way to go before we can truly rely on them. It gonna take a lot of careful thought and development to get it right.

I believe there’s still significant scaling potential for LLM training in terms of computational power. This requires simultaneous efforts in both funding and resources, and the current trend looks promising. While we should be cautious about computational scaling turning into an arms race between companies, I think competition is exactly what we need right now. The key is to focus on training efficiency and avoid the paperclip effect. Otherwise, even if we pour all of humanity’s resources into it, we won’t see significant results. This could end up having a catastrophic impact on the global ecosystem, completely contradicting our vision of building an AGI system that benefits all of humanity. To be honest, even with high training efficiency, model training is incredibly expensive. A couple of months ago, Microsoft and OpenAI announced plans to invest $100 billion in building a massive computational center. In an interview last month, Anthropic CEO Dario mentioned that their current investments are sufficient for training the next generation of models, but he’s unsure about next year. He predicts that model training next year could potentially cost tens to hundreds of billions of dollars.

Some might argue that instead of focusing so much on scaling up computational power to improve model capabilities, we should research more efficient model architectures. However, I think you need to ensure short-term research results with enormous potential and feasibility. Otherwise, for the big players, you’re essentially gambling - and the reality is they can’t afford to gamble. Once you fall behind, it’s very difficult to catch up. Now it’s just a matter of seeing who will have the last laugh in this long run.

I think it should be us to adapt the development of the AI system.

Currently, one of the greatest concern is about unemployment. My opinion is that general deployment of advanced models would make a lot of people lose their jobs, for example, those who do repetitive text-based work (transcriptionist, proofreader, etc.). This would cause great negative impact to our society. But we can try to slow down the speed of unemployment rate by bring more job opportunities. I suppose in the near future, we would need more people who are adept in guiding the AI systems to do something that they’re not so good at, or trying to bring them into some specific domains (Chemistry, Biology, etc.). So it comes to my next point, I believe we would get something really great when combining the most frontier models with different domains. And I would say AI + Science ≥ Science. I believe that we might have models that are as knowledgeable and creative as top scientists by 2025-26, including Nobel Prize winners or heads of research labs; for example, if we had a million copies of such AI models, they can collaborate with each others, and just work like a research group. This unlike human scientists who definitely would get tired, AI systems would not, you can just turn the server on and let them do research in the background. The research group they form would have greater efficiency comparing to human one, while also free human scientists from heavy (unnecessary maybe?) work. And more importantly, may accelerate the rate of scientific discoveries significantly. We can actually use AI to help us solve some of the most challenging or unsolved problems in the world, such as climate change, cancer, etc. I think it would not only be a great way to let human society adapt AI development, but also largely accelerate the development of human society. Except of this, I think we should also consider the existential crisis that AI would possibly bring. As those AI models become more advanced in areas like creativity, reasoning, and emotional intelligence, some of us may question what makes us truly unique or special as a species. Besides, AI raises profound questions about the nature of consciousness, intelligence, and what it means to be a sentient being. What’s more, AI’s potential to solve many of our problems might lead some to question the meaning and purpose of our existence if our traditional roles are diminished.

I think these are complex and nuanced questions without simple answers; and only time would give us the answer.

I think the security is very important to the future AI development. I have argued with another person for quite a long time. He thinks that the current AI systems are not capable enough to have threat to us, while I think we should take precautions, we should get full prepared for any kind of thing that could happen in the future. I mean, I didn’t believe the “catastrophic consequences” shown in the Terminator in the past, but for today, I would fear bad things that would come along due to the rapid development. I really believe that stuff like alignment, security post-training is the lifeguard of humanity. They can set a guardrail for the model capacity, for the “monster” inside; like locking the model in a cage and we, human, can study it from outside w/out letting it harm us. But this does not mean that I’m denying or rejecting the development of AI systems, I think the current ones are great; gpt-4o, claude-3-opus, and upcoming llama-3-400b are all quite awesome. I just want them to be more secure, while being capable, which means finding a perfect balance between these two. And OpenAI just announced that new Security Committee will be formed, I hope it works ;) By the way, I really love Anthropic; I love their models, I love their research, especially the recent one about model interpretability, I think the ability to “enable” the features in the model really pave a new way for model capacity discovery. I was surprised when I first read the blog.

On AGI

It’s a pretty abstract concept. I mean, how “general” is “general”? Even ASL classification by Anthropic is kinda fuzz. I think, to achieve “AGI” does not necessarily mean that we need to have an AI system that exceeds human in all domains. We just need one that exceeds average human ability in most domains. But before that, I think we should make sure the AI system can really understand:

• Riddles

• Jokes

• Memes

• Idioms

• Stuff related to special cultural context

Although these seem not so important, I think they can reflect the model’s basic and crucial language ability. They are the system that predicts the next words, so unless they really understand the pattern under words, they won’t be able to nail the points I mentioned. And if the model has strong capacity in next-word-prediction or really understands the underlying mystery between the words, it would possibly be really capable overall. (I learnt it from a podcast from Ilya hahaha) But then again, what would possibly happen if an AI system really exceeds humans in not just average but all domains? Will we be replaced? Only time will tell. The only thing we can do is to get fully prepared, and make sure the future frontier will reach a balance.

I hope AGI can really benefits humanity in the future.