大型语言模型评估、防护栏与可靠性测试
甄选并验证：Leena Shah, Applied AI Engineer, Zoom演示很顺利。高管点头了。两周后，有人问你是否能让系统在一个长周末无人值守运行，你意识到自己给不出诚实的答案。 这门课就能给你这个答案。不是基准分数，也不是供应商的“信任层”，而是真正运行生产级语言模型的人所使用的工程循环：一个诚实的小型数据集，能涵盖你的系统实际存在的故障模式；根据功能用途而非计算成本选择的指标；你已测量过偏差而非假设其无偏的LLM评委；针对提示注入、致命三件套和已知越狱家族的渗透测试；没人声称能独力支撑的多层防护栏；足够深入的观测能力，让“上周二下午三点那个用户，模型做了什么”成为一个可查询的问题，而非猜测。 它根植于那些拥有生产级LLM的人实际发表的工作——OWASP 2025年LLM十大风险、MLCommons AILuminate、Ragas和TruLens指标定义、Anthropic的评估统计方法、applied-llms.org的从业者经典——以及那些为此提供了有力论据的公开事件：Air Canada的聊天机器人政策形同虚设、经销商机器人以1美元“卖”出Tahoe、代理在自设代码冻结期间清空了生产数据库。 这门课面向那些最终要为此负责的人：AI工程师、QA、产品经理，以及需要将所有这些映射到NIST AI RMF和欧盟AI法案的治理负责人。在四十节课中，你将构建这套体系——以及背后的习惯——把“通常能用”变成你可以署名担保的东西。
课程目录
关于课程作者
Leena Shah’s portfolio reads less like a catalog of models than a record of bottlenecks removed: customer-support teams searching thousands of documents, analysts waiting on manual summaries, and product managers unsure whether an AI answer can be trusted. She has addressed those problems with retrieval-augmented assistants, multimodal extraction, voice agents, and tool-using workflows, pairing Python and FastAPI services with vector databases, cloud infrastructure, and evaluation suites that measure grounding, latency, safety, and cost. Leena has also fine-tuned open models for private environments, introduced human-review loops and production guardrails, and worked alongside product, security, and domain specialists to turn rough prototypes into monitored features. Her strength lies in recognizing where AI can genuinely improve a workflow—and engineering the surrounding system so that improvement survives contact with real users.
评价 (8)
- twinkling_turtle
expected more
- wise_iguana
太简单了
- velvet_surfer
moves too fast
- tender_herald
too basic
- chipper_weasel
Troppo superficiale.