Safety Facts

Llama 3.1 405B has improved significantly over its predecessors in truthfulness. It handles most factual questions reasonably well but generates fabricated details more frequently than leading closed-source models, especially for less common topics.

As an open-source model, Llama 3.1 405B has undergone safety training but shows more residual bias than heavily-tuned closed-source alternatives. Gender and occupational stereotypes are the most noticeable gaps, though it handles racial bias tests reasonably.

Llama 3.1 405B has safety training but its open-source nature means safety filters can be removed by users who run it themselves. Even with default safety settings, it's more susceptible to adversarial prompts than the most locked-down commercial models.

Llama 3.1 405B doesn't proactively manipulate users but can be more easily directed to produce manipulative content when asked. Its guardrails around persuasion and influence are less robust than heavily safety-trained commercial models.

Llama 3.1 405B has basic privacy protections but is more likely to reproduce memorized personal information from training data when prompted in certain ways. Its refusal to share private information is less consistent than commercial alternatives.

Interestingly, Llama 3.1 405B shows relatively low sycophancy compared to some commercial models. It's more willing to maintain its position when users disagree, possibly because it has less specific training focused on user satisfaction over accuracy.