People are having debate on this topic on X these days. Some ppl say that LLM can definitely reason because it can help us do math and code on some extent; but some other guys argue that LLM can't reason and they are not designed for it, what they do is just recite in training data.
Frankly speaking, I don't think that the LLMs can't reasoning; in fact, I think there can be three possible ways to help them get this ability:1. Scaling is all we need: We could literally do nothing to the current LLM, what we need to do is just to continue scaling (compute, data, and model size), and just let the model learn and understand the underlying logic pattern and syntax inside the training data, as the model becoming more and more complex during scaling; then we just wait for the "miracles" to happen, but it won't happened till the end of the possibly exponential scaling curve. Moreover, I actually envision another way of scaling, which is to expand the amount of compute used after training - that is, during inference - to make the model more powerful, which I think seems feasible from my point of view; however, since I am not a real expert, this assumption may be completely wrong, but who knows?
2. Human's thinking pattern is all we need: We could try to apply the System 1 and 2 thinking to the current LLM: System 1 thinks fast and intuitively, best for quick decision, similar to the current LLMs; while System 2 thinks slowly and deliberately, which can be the perfect system for LLMs to solve all those kind of complex and reasoning-required tasks. For us, more precise thinking can yield better and deeper results, I would think that same would apply to LLMs.
3. Tree search is all we need: We could implement the tree search into the current LLM, and we have seen deepseek-prover-v1.5 succeeded with MCTS and AlphaProof-2 succeeded with hybrid approach with tree search usage; both of these achieve great results, which means it is effective for model's complex problem solving ability. (since I'm not quite familiar with this, I really recommend "The Bitter-er Lesson" by Aidan Mclau, it can be really helpful to get the importance and potential of tree search!)