# Academic Researcher Portfolio # Theme: Dark Academia (scholarly aesthetic) meta: name: Dr. Sarah Williams title: Machine Learning Researcher location: Cambridge, MA timezone: UTC-5 contact: email: s.williams@mit.edu website: https://sarahwilliams.ai github: swilliams-ml linkedin: sarahwilliamsml bio: | I study how machines learn to reason about language. Assistant Professor at MIT CSAIL. My research focuses on interpretability and alignment in large language models. Previously: Research Scientist at DeepMind, PhD at Stanford. about: short: ML researcher studying interpretability and alignment in large language models long: | My research sits at the intersection of natural language processing and interpretability. I'm driven by a simple question: how do we build AI systems that we can understand and trust? Large language models are remarkable, but they remain largely opaque. My lab develops techniques to peer inside these systems—understanding what they've learned, how they reason, and when they might fail. I believe interpretability isn't just an academic exercise. As AI systems become more powerful, understanding them becomes a safety imperative. When I'm not in the lab, I enjoy hiking in the White Mountains, playing classical piano, and collecting first editions of philosophy books. settings: color_scheme: light enable_hotkeys: false animate: none layout: homepage_style: hero project_layout: case-study show_experiments: true show_timeline: true sections: experience: - company: MIT CSAIL role: Assistant Professor date: start: 2023-09 end: present description: Computer Science and Artificial Intelligence Laboratory highlights: - Leading the Interpretable AI Lab with 8 PhD students and 3 postdocs - Teaching graduate courses on ML interpretability and NLP - Principal Investigator on $2.5M NSF grant for alignment research - company: DeepMind role: Research Scientist date: start: 2020-06 end: 2023-08 description: Language Team highlights: - Co-led development of constitutional AI approaches for Gemini - Published 12 papers at top venues (NeurIPS, ICML, ACL) - Developed novel probing techniques for understanding LLM internals - Mentored 6 research interns, 4 now in PhD programs - company: Stanford NLP Group role: PhD Researcher date: start: 2015-09 end: 2020-05 description: Advised by Prof. Christopher Manning highlights: - Dissertation on compositional generalization in neural networks - Stanford Graduate Fellowship recipient - Best Paper Award at EMNLP 2019 # Simple projects (backwards compatible) projects: - name: Interpretability Toolkit description: Open-source library for probing and visualizing transformer internals. Used by 500+ researchers. url: https://github.com/swilliams-ml/interp-toolkit tags: [Python, PyTorch, Research] featured: true - name: LLM Reasoning Benchmark description: Comprehensive benchmark for evaluating logical reasoning in language models. Adopted by major labs. url: https://github.com/swilliams-ml/llm-reasoning tags: [Benchmark, Evaluation, NLP] featured: true - name: Attention Atlas description: Interactive visualization of attention patterns across transformer layers. url: https://attention-atlas.ai tags: [Visualization, Interpretability, Web] skills: Research: [Natural Language Processing, Interpretability, Alignment, Reasoning] Technical: [Python, PyTorch, JAX, Transformers, Distributed Training] Methods: [Probing, Mechanistic Interpretability, Causal Analysis, Behavioral Testing] writing: - title: "Towards Faithful Chain-of-Thought Reasoning" url: https://arxiv.org/abs/2024.12345 date: 2024-10 publication: NeurIPS 2024 featured: true - title: "Probing the Depths: A Survey of Interpretability Methods for LLMs" url: https://arxiv.org/abs/2024.11111 date: 2024-06 publication: ACL 2024 (Best Paper Nominee) featured: true - title: "Do Language Models Learn Compositionality? Evidence from Synthetic Languages" url: https://arxiv.org/abs/2023.99999 date: 2023-12 publication: ICML 2024 - title: "Constitutional AI Without Constitutional Constraints" url: https://arxiv.org/abs/2023.88888 date: 2023-07 publication: NeurIPS 2023 - title: "The Geometry of Meaning in Language Models" url: https://arxiv.org/abs/2022.77777 date: 2022-10 publication: EMNLP 2022 education: - institution: Stanford University degree: Ph.D. Computer Science date: start: 2015-09 end: 2020-05 description: Dissertation on compositional generalization. Advised by Christopher Manning. - institution: UC Berkeley degree: B.S. Computer Science & Mathematics date: start: 2011-09 end: 2015-05 description: Summa Cum Laude. Regents' and Chancellor's Scholar. # Rich projects (research case studies) projects: - id: interp-toolkit title: Interpretability Toolkit subtitle: Understanding transformer internals featured: true thumbnail: /images/projects/interp-toolkit-thumb.png hero: /images/projects/interp-toolkit-hero.png meta: year: 2024 role: Principal Investigator timeline: Ongoing tech: [Python, PyTorch, Transformers, Visualization] links: github: https://github.com/swilliams-ml/interp-toolkit demo: https://interp-toolkit.ai/demo sections: - type: overview content: | The Interpretability Toolkit is an open-source library for understanding what happens inside transformer-based language models. It provides state-of-the-art probing techniques, visualization tools, and analysis frameworks used by researchers worldwide. The goal: make interpretability research accessible to anyone working with large language models. - type: code title: Probing a Model's Internal Representations language: python code: | from interp_toolkit import Prober, AttentionVisualizer from transformers import AutoModel # Load any transformer model model = AutoModel.from_pretrained("gpt2-large") prober = Prober(model) # Analyze what the model knows about syntax results = prober.probe_for( task="dependency_parsing", layer_range=(8, 12), examples=syntax_dataset ) # Visualize attention patterns viz = AttentionVisualizer(model) viz.plot_attention_flow("The cat sat on the mat") - type: metrics data: - label: GitHub Stars value: "1.8k" - label: Monthly Downloads value: "25k" - label: Research Papers Using value: "150+" - label: Contributors value: "32" - type: image src: /images/projects/interp-toolkit-viz.png caption: Attention flow visualization across layers alt: Multi-layer attention visualization - type: outcomes content: | ## Impact on Research The toolkit has been cited in 150+ papers and is used by research teams at: - Anthropic (alignment research) - Google DeepMind (mechanistic interpretability) - EleutherAI (open-source LLM research) - Stanford HAI (human-AI interaction) Several key findings from papers using this toolkit: - Attention heads specialize for specific syntactic relations - Model depth correlates with semantic abstraction - Early layers encode surface features, late layers encode meaning - id: llm-reasoning title: LLM Reasoning Benchmark subtitle: Evaluating logical reasoning in language models featured: true thumbnail: /images/projects/reasoning-thumb.png meta: year: 2023 role: Lead Researcher timeline: 18 months tech: [Python, Evaluation, Benchmark Design] links: github: https://github.com/swilliams-ml/llm-reasoning demo: https://reasoning-benchmark.ai sections: - type: overview content: | Can language models actually reason, or are they sophisticated pattern matchers? This benchmark was designed to answer that question rigorously. We created a comprehensive evaluation framework covering deductive reasoning, inductive reasoning, abductive reasoning, and mathematical problem-solving. - type: code title: Benchmark Categories language: yaml code: | categories: deductive: - syllogistic_reasoning - propositional_logic - first_order_logic inductive: - pattern_recognition - rule_learning - analogical_reasoning abductive: - best_explanation - causal_inference mathematical: - arithmetic_reasoning - algebraic_manipulation - proof_verification - type: metrics data: - label: Test Cases value: "15,000+" - label: Categories value: "12" - label: Models Evaluated value: "40+" - label: Research Labs Using value: "25+" - type: outcomes content: | ## Key Findings Our benchmark revealed several important patterns: 1. **Scale ≠ Reasoning**: Larger models don't consistently reason better. GPT-4 fails on many problems that require systematic deduction. 2. **Training Data Leakage**: Models often "solve" problems by recognizing similar examples from training, not by genuine reasoning. 3. **Fragile Compositionality**: Models struggle to combine reasoning steps they can perform individually. The benchmark is now used by OpenAI, Anthropic, and Google for model evaluation during development. # Experiments (smaller research projects) experiments: - title: Attention Atlas description: Interactive web visualization of attention patterns across transformer layers and heads. demo: https://attention-atlas.ai tags: [Visualization, D3.js, Research] image: /images/experiments/attention-atlas.png - title: Synthetic Language Generator description: Tool for creating controlled synthetic languages to test compositional generalization. github: https://github.com/swilliams-ml/synth-lang tags: [Python, NLP, Testing] - title: Chain-of-Thought Analyzer description: Scripts for analyzing and categorizing chain-of-thought reasoning patterns in LLM outputs. github: https://github.com/swilliams-ml/cot-analyzer tags: [Python, Analysis, LLMs] # Academic timeline timeline: - year: 2024 title: NSF CAREER Award description: Received $2.5M grant for research on interpretable AI systems. image: /images/timeline/nsf-award.png - year: 2023 title: Joined MIT as Assistant Professor description: Started the Interpretable AI Lab at MIT CSAIL. - year: 2022 title: NeurIPS Outstanding Paper description: '"Constitutional AI Without Constitutional Constraints" recognized for impact.' - year: 2020 title: Joined DeepMind description: Research Scientist on the Language team, working on model interpretability. - year: 2020 title: PhD from Stanford description: Defended dissertation on compositional generalization in neural networks. - year: 2019 title: Best Paper at EMNLP description: '"The Geometry of Meaning in Language Models" received best paper award.' - year: 2015 title: Started PhD at Stanford description: Joined the Stanford NLP Group, advised by Christopher Manning. # Testimonials from collaborators testimonials: - quote: "Sarah's work on interpretability has fundamentally shaped how we think about understanding large language models. Her toolkit is essential infrastructure for the field." author: Chris Olah company: Anthropic role: Research Lead - quote: "One of the most rigorous researchers I've worked with. Sarah's benchmark for reasoning evaluation has become the gold standard." author: Percy Liang company: Stanford HAI role: Director - quote: "Sarah's dissertation was one of the most influential I've advised. She asks the right questions and follows them wherever they lead." author: Christopher Manning company: Stanford University role: PhD Advisor theme: dark-academia # Page content customization pages: projects: title: Research & Publications description: Scholarly work exploring interpretability, alignment, and reasoning in large language models. about: title: About Me description: Machine learning researcher at MIT CSAIL experiments: title: Laboratory description: Smaller research projects and exploratory investigations. writing: title: Publications description: Peer-reviewed papers and preprints on interpretability and alignment.