Moral realism blogging is traditional

Some shit I was on about one time

It is a grand tradition of daily blogging, including mostly-daily blogging, that at least one post be a moral realism post. Let's do it.

Set 'em up 🎳

We can identify "moral realism" with the position that mind-independent reasons for action exist. Then, if we're allowed to use traditional sloppy-rationalist-reasoning, moral realism is trivially false. For you see, given any mind-indepedent reason for action, you could construct a mind stupid enough not to act on the reason, because e.g., every time it started to think about it it got distracted and went and did something else (ask me how I know about this).

...you might observe then that it seems like "being too stupid to act on the reasons" is the one axis of mind-variance we should ignore when considering the possibility of mind-independent reasons for action, since we can hypothetically retain all the structure we care about when excluding just this one dimension. Just because something is unable to identify an incentive structure doesn't mean the incentive structure doesn't exist. (Unless, uh, it does mean that. Don't worry about it!)

(At the very least it seems like we can say that whether or not an incentive structure "exists" is not necessarily the same thing as whether or not it influences the course of events.)

Avoiding the "stupidity" axis, it seems vaguely intuitive to me that for any given reason-shape you could build a mind that didn't care about that specific thing even in the fullness of understanding. But, you could call this being built containing contradictions, and that any such agent is unstable and would collapse into one of the stable subsets of itself under intelligent reflection, presumably in a way now influenced by the mind-indepedent reason for action it was originally built to avoid.

So perhaps we can conclude that if mind-independent reasons for action exist, we can assume that they at least will influence the behavior of *sufficiently intelligent* and *sufficiently coherent* agents. That sounds like it excludes a lot of possible agents! It gets me asking questions like: why do we care about this, again?

Knock 'em down 💥

It's 2026 and folks are building AI as fast as they possibly can, with minor exceptions for fanservice. It would be nice to know that if they make the AI "smart enough," it would realize moral realism 1. is true, and 2. constrains its actions to actions we also consider good. And now you're telling me that even if moral realism is true, it might not work if the AI is dumb enough or incoherent enough? This doesn't make me less concerned, really.

One frame for our above conclusion (to speak in underdefined abstractions) is that for any given agent, there are a set of arguments that it would find convincing. We can classify agents by the sets of arguments they find convincing, using that abstract unique well-agreed-upon argument-definition-language we have lying around. This includes agents that could, if they were smarter, be convinced by the fully mind-indepedent reasons for action; they aren't there (yet! growth mindset), so they don't fall into the MIRFA-convinced class (...whatever that means in this context...), but they still form agglomerations.

Some of these perhaps are very large--the space of agents convinceable that multiplayer games should be played in ways that adhere to game theory seems likely to be quite large indeed--and some of them, among the more gerrymandered and incoherent agents (that are, by their instability, less encounterable, but nevertheless not *un*-encounterable!), probably much smaller, relatively.

We could split our understanding of "moral realism" into two classes of theory. What we currently understand as "moral realism," let's call "strong moral realism," the idea that as described previously there is only one "convinceability class," where the convincing arguments are the mind-indepedent reasons for action (it being in the name, and all). "Weak moral realism," then, is the claim that there are *reasonably large* classes of agent convinceable by reasonably abstract or general arguments.

Regardless of whether we find ourselves in a "strong moral realism" universe or a "weak moral realism" universe, we still have to identify the reasons for action that apply to the convinceability class we ourselves reside in. It seems undeniable, though, that to be *really sure about things,* even in a strong moral realism universe, we would want to make sure that any given counterparty we were interested in understanding, was smart and coherent enough to recognize the MIRFAs, before we trusted them with anything properly important.

That is to say, even if the distinction between strong and weak moral realism I've drawn doesn't make any sense, the mere fact of strong moral realism does not seem like it should be any great assurance that any given agent will take actions influenced by mind-independent reasons, in any situation of consequence. (Source: lookin around.)

Given that that's true--that we can't assume our counterparty is "safe" just because moral realism is true--then it sounds like we need to know a thing or two about our counterparty. Like, say, what convinceability class they fall into, if weak moral realism is true, and indeed what convinceability class *we* fall into, and if the two classes are the same.

Even if there's some truth to be found out in the world about the Good, I don't think we can make use of it without understanding the truth about ourselves and others, as well.

The kanji of the day is 良

• Its Kodansha Kanji Learner's Course keyword is GOOD.
• Its readings are よ, い, and リョウ.

If you will refer to the chart you'll find that I placed this kanji at D-tier. Regrettable! Sometimes I think it's cute when one tiny stroke differentiates two kanji/radicals/graphemes (see: 艮), but in this case I think he got a little less pretty. I find this kanji fun to write but less fun to look at. Fascinatingly, Wiki claims it is etymologically(?) unrelated to 食/FOOD/EAT.