|
Hi! I am a forth-year PhD student at Fudan University, advised by Prof. Xipeng Qiu. I received my bachelor's degree from Sun Yat-sen University, advised by Prof. Hanjiang Lai. I am currently interning at Shanghai AI Laboratory as a LLM researcher (2023.5-Now). Previously, I worked as a software development engineer intern at ByteDance (2021.3-2021.8). My research interests include post-training of large language models, including alignment (truthfullness and safety), reinforcement learning for LLMs (reasoning), multimodal large models (vision and speech), interpretability (mechanistic interpretability), and LLM-based agent systems (RAG and task-oriented dialogue systems). Reach out to me over email: chengqy2019@foxmail.com. CV / Google Scholar / Github / X (twitter) / OpenMOSS |
|
|
|
Full list of papers can be found at Google Scholar (*: Equal contribution) |
|
We ask the question "Can AI assistants know what they don't know and express them through natural language?" To answer this question, we construct a model-specific "I don't know" (Idk) dataset for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets. Then we design a series of truthfulness alignment methods to align the assistant with its corresponding Idk dataset and observe whether it can refuse to answer its unknown questions after alignment. |
|
We introduce HalluQA, a benchmark with 450 adversarial questions to assess hallucinations in Chinese large language models, covering diverse domains and cultural contexts. It targets imitative falsehoods and factual errors, built using GLM-130B and ChatGPT, and evaluated automatically with GPT-4. Testing 24 models like ERNIE-Bot and Qwen, 18 scored below 50% non-hallucination rates, showing its difficulty. We analyze hallucination types, causes, and prioritization for different models. |
|
We propose Contrastive Learning of sentence embeddings from AI Feedback (CLAIF) to enhance contrastive learning in NLP. Unlike typical methods struggling with sample pair quality due to language's discrete nature, CLAIF uses AI feedback from large language models to create fine-grained similarity scores for sample pairs. Combining this with human feedback, it improves supervised contrastive learning. Experiments show CLAIF outperforms other methods on semantic textual similarity and transfer learning tasks. |
|
We introduce Unified Active Retrieval (UAR) to improve Retrieval-Augmented Generation (RAG) by addressing inefficiencies in existing active retrieval methods. UAR uses four orthogonal criteria as simple classification tasks for better retrieval timing, with minimal extra cost. Supported by UAR-Criteria, it handles diverse instructions effectively. Experiments show UAR outperforms others in accuracy and downstream tasks. |
|
We propose an interactive evaluation framework for Task-Oriented Dialogue (TOD) systems to address the policy mismatch in current methods, where user utterances don’t align with varied system responses. Our approach uses a goal-oriented user simulator built on pre-trained models to generate dialogues, introducing sentence- and session-level scores for fluency and coherence. Experiments show RL-based TOD systems trained with our simulator achieve 98% inform and success rates on MultiWOZ, with the new scores enhancing response quality assessment. |
|
We present the Safe Inputs but Unsafe Output (SIUO) challenge to assess cross-modality safety in Artificial General Intelligence (AGI), where individually safe modalities can combine to produce unsafe outputs. Unlike prior studies on single-modality risks, SIUO targets complex interactions across 9 safety domains, including self-harm and privacy violations. Our benchmark tests reveal significant vulnerabilities in models like GPT-4V and LLaVA, highlighting their limitations in handling real-world scenarios safely. |
|
Sparse dictionary learning has been a rapidly growing technique in mechanistic interpretability to attack superposition and extract more human-understandable features from model activations. We ask a further question based on the extracted more monosemantic features: How do we recognize circuits connecting the enormous amount of dictionary features? We propose a circuit discovery framework alternative to activation patching. |
|
|
|
We introduce SpeechGPT 2.0-preview, our first human-like real-time interaction system, trained on millions of hours of Chinese speech data. This end-to-end model offers low-latency, natural responses with human-like expressions, supporting interruptions and multi-style emotional control. It excels in role-playing, vocal talents like poetry and dialects, and integrates text capabilities for tool use and searches. Currently, it supports only Chinese, with no English training yet. |
|
We analyze OpenAI's o1, an AI milestone excelling in reasoning tasks, driven by reinforcement learning. Unlike alternatives like knowledge distillation, limited by teacher models, our roadmap focuses on four key components: policy initialization for human-like reasoning, reward design for effective guidance, search for high-quality solutions, and learning to enhance performance with more data and parameters. These elements highlight how learning and search power o1's success, influencing LLM development. |
|
Mainly responsible for data synthesis in the post-training stage, including multi-turn dialogue, safety alignment, honest alignment. MOSS is a conversational language model like ChatGPT. It is capable of following users' instructions to perform various natural language tasks including question answering, generating text, summarzing text, generating code, etc. MOSS is also able to challenge incorrect premises, and reject inappropriate requests. Here is a brief introduction to MOSS. |
|
Reviewer / Program Committee Member
|
|
Design and source code from Jon Barron's website |