Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning

Created2024-07-26|Updated2024-07-26

|Post Views:

第一作者：Zhaorui Yang

作者单位：浙江大学

发表时间：2024/5

发表期刊：ACL 2024

关键内容：Self-Distillation Fine-Tuning (SDFT)，用模型生成的数据来对模型进行训练，弥补数据集与LLM分布的不同而导致微调带来的灾难性遗忘的问题。

参考文献：Yang Z, Liu Q, Pang T, et al. Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning[J]. arXiv preprint arXiv:2402.13669, 2024.

Author: Wenliang Liang

Link: https://lwl1751.github.io/2024/07/26/Self-Distillation-Bridges-Distribution-Gap-in-Language-Model-Fine-Tuning/

大模型文献笔记

2024-07-06

Unifying Large Language Models and Knowledge Graphs: A Roadmap

2024-07-07

Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding

2024-07-08

Ontology-enhanced Prompt-tuning for Few-shot Learning

2024-07-20

ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models

2024-07-26

Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization

2024-07-26

REASONING ON GRAPHS: FAITHFUL AND INTERPRETABLE LARGE LANGUAGE MODEL REASONING

Loading the Database