[Reach]

I agree with your sentiments here. Our mind is our body, and our cognitive processes are fundamentally limited by our neurological architecture.

An AI will always be limited by its architecture as well. I don't doubt that AI could eventually become far more intelligent than we are, but the idea of an essentially unlimited intelligence seems impossible.

I suppose if we want to theorycraft here, the only way you could achieve an essentially unlimited artificial intelligence would be if the AI could expand its own architecture indefinitely. While it seems possible at face value for a computer to continue to build on itself if it were intelligent enough to do so, you face fairly simple restrictions on what is possible via resources required to build the materials the AI is composed of, as well as power sources required to run the AI and the ever increasing demands on construction and maintenance of the super AI.

The resource demands of such a super AI could quickly become absurd and unsustainable, without the ability to extract more resources from the far reaches of space, which may or may not be viable. I'm imagining the scenario where an ASI continues to expand until it engulfs an entire planet, an interconnected network with hordes of worker AI continually maintaining and constructing additions to the ASI. Eventually the ASI would have to reach an equilibrium where it simply can't expand itself anymore based on the confines of the system it has built itself in. Exactly how intelligent that AI would be is anyones guess, but it would still have fundamental limitations.

I'll stand on the pessimist end of things as well and say it won't happen any time soon. AI advancements will probably creep in continual increments with most of them having practical applications that will help human life, but I don't think we're going to see a zero to hero ASI any time soon.

One reason I think people erroneously overestimate the potential of AI is related to how we've traditionally seen Moore's law, but computational efficiency can only continue to double for so long. In the coming decade or two we're going to see efficiency plateau, as the maximal efficiency of computer parts is limited by the laws of physics. At some point, systems will have to become larger and not simply just more efficient. This will place practical restrictions on ASI.