Constitutional AI: Harmlessness from AI Feedback

Constitutional AI: Harmlessness from AI Feedback

Research questions. The paper asks whether AI systems can help supervise and improve other AI systems, especially for harmlessness, using only a small set of explicit principles rather than human feedback labels. It also asks whether this can produce an assistant that refuses harmful requests without becoming evasive or unhelpful.  

Methodology. The authors introduce Constitutional AI, with a supervised phase where the model critiques and revises its own harmful responses, followed by a reinforcement learning phase using AI-generated preference feedback. The “constitution” is a short list of natural-language principles that guide both critique and preference judgments.  

Findings. The resulting RL-CAI model is described as more harmless while remaining helpful and less evasive than prior human-feedback-trained models. The paper also finds that chain-of-thought style reasoning can improve both performance and transparency in the critique and feedback process.  

Why it matters. This paper is important because it helped define a major alternative to traditional RLHF: using AI feedback and explicit principles to scale alignment work. It matters for sycophancy and alignment research because it shows one way to make model behavior more governable, inspectable, and less dependent on large volumes of human preference labels.