Appendix · Safety & alignment

Back to core

Appendix: Index · Prev: Autonomy

Overview#copy

Safety and alignment work spans training-time alignment (preference learning), evaluation, interpretability, and deployment-time controls. The practical failure modes are specification gaps, distribution shift, and misuse.

1) Preference learning and instruction tuning#copy

Key papers#copy

• InstructGPT (Ouyang et al., 2022)
• Constitutional AI (Bai et al., 2022)
• Direct Preference Optimization (Rafailov et al., 2023)

Code repositories / projects#copy

• CarperAI/trlx
• huggingface/trl

2) Robust evaluation#copy

Key papers#copy

• Holistic Evaluation of Language Models (HELM, Liang et al., 2022)
• TruthfulQA (Lin et al., 2021)

Code repositories / projects#copy

• stanford-crfm/helm

3) Interpretability and monitoring#copy

Key papers#copy

• Toy Models of Superposition (Elhage et al., 2022)

Code repositories / projects#copy

• TransformerLensOrg/TransformerLens