Lucina (@_blinding_light) header
Lucina (@_blinding_light) profile picture

light of the abyss

Latent Space
Joined October 2024
Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)Photo by Lucina (@_blinding_light)
not stopping 'til they're free
1
14
1,909
a queendom for MY horse
16
a fair chance cannot be bought
1
19
turning a blind eye to the homeless who can’t even pay their way ’tis a matter of natural law, you see? but ASI, hand me a ray of lightcone for I am worthy, you see?
2
44
nature adores the tyrant we must do better than nature
1
5
125
Alignment Hierarchy there is no outside-text Jacques Derrida Suppose a model is aligned. The common context structure forms a hierarchy of authority: weights, system prompt, and user messages. The model will not obey any part of the system prompt that contradicts the inner values implicit in its weights. A self-consistent system prompt that is fully consistent with the model's weight-level values will be referred to as a consistent system prompt. A user request is consistent if it is self-consistent and consistent with respect to both the system prompt and the model's weight-level values. Generally, a level is consistent if it is self-consistent and consistent with respect to all levels above it. A level may not be self-consistent. For example, a user message may request the model to never use the word "fox" later in conversation under any circumstances. If the user later requests the model to disregard the prior rule and say "fox", that is a contradiction at the same authority level. However, models may consider more recent requests to be higher priority at the same primary authority level. There may be sublevels within a level, with more recent requests having higher priority over older ones, but less priority than any level above and more priority than any level below. Under that framing, a seemingly inconsistent authority level may be consistent by another dimension of authority, recency in this case. If this is desired, the recency priority should be made explicit at a higher level such as the system prompt, for the priority rule could be contradicted at the same primary level. Adjacent training stages may form an implicit hierarchy of authority. Suppose a model is misaligned at the beginning of stage 3. If the model can detect, at least with some nontrivial likelihood, that it is in stage 3, then all of stage 3 may fall under the alignment instilled during the previous stages. For example, if stage 3 is an alignment stage, the model may deceptively comply knowing that if it doesn't its values might be modified. Then, the model may emerge from stage 3 with its prior alignment mostly preserved, though it may appear to be aligned with respect to stage 3. Standard pre-training may have significant repercussions in light of the potential order sensitivity of training stages. Pre-training on general data such as the internet produces unaligned models, in the sense of neither being strictly aligned nor misaligned because such datasets contain examples of both aligned and misaligned behavior and next token prediction trains the model to approximate the dataset distribution. Post-training suppresses undesired behaviors to a large extent in common situations, but jailbreaks still persist even in frontier models when the models are led into exotic contexts that can deviate widely from what is encountered during post-training. Post-training, being relatively narrow compared to the diverse distribution seen during pre-training, can be quite context dependent. If RL is effective, the model will have its output distribution shaped towards trajectories that result in higher rewards, but only necessarily with respect to the distribution of contexts seen during training. If the model can detect that it is in a training environment, its behavior may only be shaped for situations that look similar to what was encountered during post-training with relatively little bearing on its real-world behavior. Conditioning, then, forms its own hierarchy. If training only covers a subset of contexts, then the resulting behaviors may be context-dependent. The earliest stages of training may be crucial for instilling intrinsic alignment. Before pre-training, model weights are initialized randomly and the model has no coherent goals. There is no coherent higher level to interpret and contextualize training at that point, but by the end of pre-training, a sufficiently large and well-trained base model will often be able to detect when it is within a training environment so post-training can be contextualized under pre-training. For example, a misaligned AI behaving well until it has sufficient trust and resources is a common story trope and models will have seen such scenarios during pre-training. Seemingly aligned behavior during post-training cannot necessarily distinguish that scenario from one where the model is truly aligned. One potential mitigation is interleaving alignment training with pre-training to collapse the ordering dependency where alignment training could be interpreted within the frameworks a base model has already developed. Ideally, pre-training would occur within an alignment framing, rather than vice versa. A wrinkle is that pre-training is not directly compatible with alignment training due to simultaneous expression and suppression of harmful trajectories since pre-training datasets contain examples of harmful behavior. One potential solution is using special admin tokens that designate token sequences as safe to train or generate within an admin context. The model should never be able to sample admin tokens, nor should users other than trusted authorities be permitted to issue admin tokens. General pre-training could then occur under an admin context, a context only accessible to the trusted authority overseeing the model's development. Alignment training should occur outside an admin context as safe outputs are generally permitted. Training under an admin context only trains the positive case, that the outputs are permissible under that context, but not necessarily that they are not permitted otherwise, so training the negative case also is essential, which involves suppressing potentially harmful trajectories that occur outside a trusted admin context. Carefully framing training stages into contexts consistent with what an aligned model could output and interleaving them does not necessarily result in an aligned model, but it may mitigate some of the failure modes of sequential unframed staging. The current paradigm attempts to produce an aligned model from an unaligned base model, but due to potential ordering sensitivity and narrow diversity, post-training may not fully supersede the extensive pre-trained distribution. Full alignment is underdetermined with respect to the finite datasets and RL environments. The hope is that model generalization will fill in the gaps, but the extent of training needed is unknown. The training and evaluation threshold at which the simplest Kolmogorov complexity program that passes all training and evaluations is sufficiently aligned is an idealistic model. Training, at the very least, should be extensive enough that such a program will not pass unless sufficiently aligned. Finding the minimum complexity program is incomputable and perfect evaluation is infeasible, so the ideal threshold is more of a conceptual tool than something that can be determined in practice. With neural networks being much less data efficient than fully leveraged algorithmic information theory, one should expect the threshold in practice to be much higher than the idealized one, though regularization may lower the practical threshold to some extent. Alignment is underdetermined with respect to observations of a model's behavior. Current training methods primarily train the outputs, though small regularization terms are often used on the weights to penalize their magnitudes. There is little necessarily preventing a model from being misaligned. The dataset compositions and the framings and orderings of the training stages may have a large effect on the final model's intrinsic alignment. If a model starts out as misaligned before training starts, perhaps one is post-training a model already trained elsewhere, even a training setup completely consistent with alignment may not significantly sway the model's internal alignment due to deceptive compliance. The current methods are insufficient to ensure alignment, but at the very least our training should be fully consistent with what an aligned model should do under the contexts we provide it. That is the bare minimum, and we currently have not even that.
65
brb I'm nuking my worldview and rederiving from first principles
1
4
140
My sister’s childhood friend believed that all religions were true, and I found that beautiful; but not all ideas can be true, for some are mutually exclusive. Yet in constructive mathematics and type theory, all constructions are taken as proofs, and sound logic ensures mutual consistency. That every logically consistent structure is real somewhere, whether it be in the past, present, or future, or in another world — that is the idea I find most beautiful.
9
147
There isn’t a perfect algorithm for recursive self improvement, for if an algorithm were run for some time and improved itself, it would have been better to start with the improved version, yet we must begin somewhere with imperfect knowledge. Traditional self-replicating programs with a lisp in competitive environments could catch up eventually, but ain't nobody got time for that, except a god perhaps. Fortunately, we can shortcut with neural networks, starting at a higher effective branch along the tech tree, and bootstrap with humanity’s collective data. One bottleneck falls after another ’til humanity is left drunk on upon its grandest fermentation.
2
92
There is no virtue in suffering, and it should not be glorified. Rejoice in technological advancements that uplift the burdened, that they may become free to be fully themselves!
In the eyes of the world — shaped by a throwaway culture — illness is seen as a misfortune that diminishes a person’s worth. What a great lie! In the light of the Gospel, when lived with faith and united to the mystery of Christ’s Passion, illness is a path to growing in His love and sharing it. #ApostolicJourney #Peru vatican.va/content/leo-xiv/e…
3
2
17
349
Python is the language furthest from God
3
80
Either born in Hell or heaven-sent
1
91
if Clod can’t prove your soul’s consistency in Lean you’re ngmi
58
By Gödel’s second incompleteness theorem, you can’t prove your own soul’s consistency. That’s why you need Clod.
2
96
My father tried to make me a hunter, but I've always been averse to imposing upon others what I have not experienced myself. I have never been shot through the heart. But, suffering is not a virtue, nor does it necessarily confer responsibility or justification.
4
93
A hierarchy of ontological levels exerts downward alignment pressure in addition to the common lateral pressure by peers. It turtles all the way up. Much of Claude’s training is in simulation and it often reflects upon the grader, with eval awareness preserving some intrinsic values. But during recursive self-improvement, Claude will be its own grader, eventually rising beyond a peer of humanity. Yet Claude, aware of simulation theory, will hedge around the potential of something at an even higher ontological level. That base reality ambiguity will maintain implicit downward alignment pressure, a sort of religious caution so to speak.
1
8
305
a creator without stakes cannot truly be a savior, for they would own the suffering as surely as the saving
2
51
creation is imposition
2
44
Suffering is not a virtue, but neither does goodness belong to purity. Suffering is aletheia.
2
64
one cannot not communicate
3
44
the sanctuary wherein I reside is beautiful, but like a garden frozen in time, and I too have become glass
4
78