Introduction to gpt-4 Fine-Tuning Experiments

The article presents a detailed study conducted by a group of friends fascinated by language models like gpt-4. They investigated the potential of fine-tuning gpt-3.5 using the OpenAI Fine-Tuning API for two practical tasks: reliable output formatting and custom tone.

• The goal was to substantiate the claim that fine-tuning with around 100 data points can lead to significant improvements.
• The experiments focused on assessing the performance, cost, and latency implications of fine-tuning gpt-3.5 for specialized tasks.

Methodology for gpt-4 Fine-Tuning Tasks

The study explored two specific use cases mentioned in the OpenAI API release note:

• Reliable Output Formatting: Fine-tuning gpt-4 to consistently format responses, such as for code completion or API calls.
• Custom Tone: Honing the qualitative tone of gpt-4 output to better fit a brand's voice, like a rude customer service agent.

Results and Findings with gpt-4 Fine-Tuning

The experiments yielded promising results for both tasks:

• Reliable Output Formatting: At 50-100 data points, the fine-tuned gpt-3.5 model achieved a 96% improvement in formatting accuracy compared to the base model, while maintaining answer correctness.
• Custom Tone: In a double-blind study, the fine-tuned gpt-3.5 model with 1000 data points outperformed both base gpt-3.5 and gpt-4 in exhibiting a rude tone for customer service scenarios.

Cost and Latency Considerations for gpt-4 Fine-Tuning

The article discusses the cost and latency advantages of fine-tuning gpt-4:

• Cost: Fine-tuning can lead to savings by reducing the need for expensive models like gpt-4, fewer input tokens, and less complex architectures.
• Latency: Surprisingly, the fine-tuned gpt-3.5 models exhibited 3.6 to 3.76 times faster inference speed compared to the base gpt-3.5 model.

Unstable Behaviors and Challenges with gpt-4 Fine-Tuning

The study also highlighted some unstable behaviors and challenges encountered during fine-tuning:

• Non-deterministic training and evaluation runs, with some models not converging properly.
• Catastrophic forgetting observed at 1000 examples and temperature = 1 for the custom tone task.

Future Exploration and Unanswered Questions

The authors discuss several unanswered questions and potential avenues for future exploration:

• Scaling laws and fine-tuning for other use cases like RAG contextualization, personalization, and traditional NLP tasks.
• Sweeping hyperparameters like temperature, epochs, and data mix for fine-tuning.
• Exploring boundaries like catastrophic forgetting, secondary-order effects, and non-determinism.
• Fine-tuning open-source models and evaluating different fine-tuning methods.

Appendices and Additional Resources

The article includes several appendices providing detailed information on:

• Data and metrics used for output formatting and custom tone tasks.
• Latency experiment details and methodology.
• A lightweight repository for distilling gpt-4 to gpt-3.5.
• Examples of emotional damage caused by language models during fine-tuning.