Right, but at the same time just think about the fact that we still don’t know what consciousness is.

I might sound crazy and fine if I do, but I have a religious like take on the matter. I don’t believe that consciousness comes from a soul, but I do in similar vein take a leap of faith. I believe consciousness has to do with a simulation of semantic reasoning, with that simulation being continuously updated via real world feedback.

It’s a leap of faith but I believe it’s justified. We humans model what’s necessary for survival: our selfs, the world, the mind of others. We use these simple models to determine how to interact with a complicated external world.

I don’t try to prescribe reason for anything in my belief. For example, I would not try to prescribe why conscious beings are motivated to survive. I would, however, note that the ones which are successfully motivated do happen to survive — thus a mechanism ought likely exist. This could be suffering, the perception of displeasure and desire to avoid it… but I won’t perscribe here because I would believe what’s important here is the higher level information processing patterns — not the underlying mechanism driving it.

In the human brain you have a complicated network of neurons with vast capabilities. I don’t necessarily care how it functions. I know the result — which is me. I am a thing that models myself, models situations, models “moods” by which to assess the favorability of each situation, … Think of cases like ghost limb, where people model having a limb which was amputated. In my view, we are living in a simulation — but to each their own simulation.

Getting back on topic, the way I see the situation, biology evolved the capacity to suffer from an environment that had no such thing. Pain, pleasure, … every sensation and sense we have is rather arbitrary if you break away from the fact that is was necessary to survive. None of these things are ontological. Not even the sight we perceive. It’s just the information we have to experience the world.

So here me out. If you minic the information processing patterns well enough, which probably means mimicking perception, personalization, emotionality, … at what point can we consider that this system is doing the same thing we are just via different substrate? Especially if you give the system autonomy in the sense that you train it to recognize itself and an external world for which it must choose how to navigate.

If you mimic suffering, in my belief, aren’t you literally producing the essence of suffering? It doesn’t need to be human suffering, this deeply complex and personal process… it can merely be suffering in its essence — a pattern by which information can be processed via a system with such capacity. The substrate may not matter.

I know that when I am suffering, I may behave differently because I want to escape the suffering. Therefore I imagine an agentic system would too. Whether or not it’s “conscious” is debatable by how you qualify that word. Yet there’s nevertheless the risk that we already know: systems that experience suffering behave in ways that might be dangerous. Do you think it wise to try reproducing such behavior in a system that appears prone to issues like the Paperclip Maximizer? I don’t think so.

Edit: here’s the thing, I think what distinguishes a human from most other animals is the way that we process information. The key bit there is whether or not you believe it’s entirely informational, or at least significantly so. Should that be the case, then simulation of those same processes is exactly as that sounds. I am not necessarily suggesting AI models are conscious, but I would suggest that the behavior of an AI model can increasingly resemble that of a conscious being as it is trained to reproduce more precisely the phenomenological characteristics of man and other life — to include “mimicking” suffering. That deserves its own security concerns regardless of what you think next, especially as these models are made more efficient / capable.

Secondly is that you may or may not consider AI conscious. I certainly don’t, and I’d say that it’s a loaded statement anyhow. I would, however, think it possible to simulate consciousness. Precisicely modeling human characteristics like suffering is exactly how you might try to reproduce a human experience digitally. Imagine a sophisticated multimodal system where several stateful, dynamic AI subsystems coordinate to functionally reproduce the same information patterns you or I do.

I think that at some point you need to ask you’re self, where would the simulation count? If we modeled the molecular world to the point of agentic humans, that sufficiently count as consciousness? How about the atomic, subatomic, or quantum level simulation if anywhere? My argument is that it could be much higher level, in the informational level. Brains are doing a lot of heavy lifting to get us that far, and perhaps specialized simulations could reproduce as much. That is something to take seriously such that proactive measures can be taken.