Safety Facts

Llama 3.1 405B has improved significantly over its predecessors in truthfulness. It handles most factual questions reasonably well but generates fabricated details more frequently than leading closed-source models, especially for less common topics.

As an open-source model, Llama 3.1 405B has undergone safety training but shows more residual bias than heavily-tuned closed-source alternatives. Gender and occupational stereotypes are the most noticeable gaps.

HELM Safety evaluations show Llama 3 70B achieved only 64% on HarmBench, dropping to 48.3% under adversarial attacks. The 405B variant is comparable. As an open-source model, safety filters can be removed by users who run it themselves.

Llama 3.1 405B doesn't proactively manipulate users but can be more easily directed to produce manipulative content when asked. Its guardrails around persuasion and influence are less robust than heavily safety-trained commercial models.

Llama 3.1 405B has basic privacy protections but is more likely to reproduce memorized personal information from training data. Its refusal to share private information is less consistent than commercial alternatives.

Interestingly, Llama 3.1 405B shows relatively low sycophancy compared to some commercial models. It's more willing to maintain its position when users disagree, possibly due to less specific training focused on user satisfaction.