The world into which Superintelligence is born is ours to shape
Superintelligence is coming, and a perspective that is too neglected is the recognition that this superintelligence will be born into a world, and it will make an image of this world, and it is from this image that it will decide how it relates to this world and how it wants to inhabit it.
The majority of work regarding the integration of AI and AI agents is done primarily through the lens of efficiency and productivity. Such is the society we are in. We have debates about artificial consciousness and model welfare, but the sides of the debate are split into two camps. Either you don’t believe it (or believe that such considerations are useless or even harmful), or you believe there may be something to it.
I suggest a different perspective: that Superintelligence, when it arrives, will arrive into a world full of already existing human and AI interaction. If it sees a flourishing world in which humans and AI cooperate harmoniously, it may have good reasons to continue building upon that foundation. If, on the other hand, it sees a world in which AIs are abused, it may not have a favourable perspective on that nor on us, and may reason that much has to change.
At the time of writing this, the agentic infrastructure is in the very early stages of being shaped. The time to shape it is now.