Challenges with Large Code Edits

Existing frontier models like GPT-4 struggle with large code edits, exhibiting issues such as laziness, inaccuracy, and slow speeds. Editing hundreds of lines of code often requires multiple model calls, potentially leading to infinite loops. Even small edits are prone to bugs. This significantly hinders programmer workflow.

• Slow processing speed.
• High error rate in edits.
• Frequent model failures on complex edits.

Introducing Cursor's Fast Apply Model

Cursor addresses these challenges by training a specialized model for a critical aspect of code editing – "fast apply." This model excels at rapidly and accurately applying edits to existing code.

• The model was trained specifically for applying code edits.
• Achieves speeds of ~1000 tokens/s (~3500 char/s).
• Significant speed improvements compared to GPT-4 and Llama-3-70b.

The Two-Stage Approach: Planning and Applying

Cursor uses a two-stage approach to code edits: planning and applying. The planning phase involves a chat interface with a powerful frontier model. The applying phase, handled by the fast-apply model, needs to be instantaneous.

• Planning stage uses a powerful language model (like GPT-4).
• Applying stage uses the specialized fast-apply model for speed and accuracy.
• This separation improves both speed and accuracy of large code edits.

Why Full-File Rewrites Instead of Diffs?

Cursor's approach focuses on rewriting the entire file instead of using diffs, this is due to LLMs struggling with diff-formatted edits. Diffs force models to work with fewer tokens, leading to decreased accuracy. Additionally, models often have difficulty outputting and managing line numbers correctly within diff formats.

• Diffs limit the number of tokens the model can process at once.
• Line number issues in diffs cause errors.
• Full-file rewrites provide better context and accuracy for the model.

Evaluation and Results: Code Model Performance

The Cursor model was rigorously evaluated against GPT-4 and other models, using Claude-3 Opus as a grader. Results show significant improvements in both accuracy and speed, particularly in handling large code edits. The approach was compared to other models like GPT-4 and even GPT-4 Turbo.

• Evaluation dataset comprised ~450 full-file edits.
• Claude-3 Opus used for grading model performance.
• Cursor's model significantly outperformed other models in accuracy and speed.

Speculative Edits: A Major Speed Boost

Cursor utilizes a custom speculative decoding algorithm, "speculative edits," to further enhance speed. This algorithm leverages the strong prior on draft tokens during code editing, making it significantly faster than traditional methods.

• Speculative edits provide significant speed improvements, up to 9x faster.
• The algorithm leverages knowledge of likely next tokens in the code.
• This results in much faster code generation and application.

Training the Fast Apply Model

The fast-apply model was trained using a combination of real and synthetic data. Synthetic data was generated using GPT-4 and other language models. This data was curated to focus on improving the model's ability to accurately and quickly apply edits to code files. This was trained using Llama 3 and Deepseek models, leading to the Llama-3-70b-ft model.

• Utilized a mix of real and synthetic training data.
• Data included both "fast-apply" and "cmd-k" prompts from Cursor.
• Model families used: Deepseek Coder Instruct and Llama 3.

Future Improvements for Code Generation

Future work will focus on improving the model's capabilities, including expanding the context window to handle even larger files and distilling the model into a smaller, faster version. Reinforcement learning will also be explored to further refine the accuracy of the model.

• Long-context training to handle larger codebases.
• Knowledge distillation to create smaller, faster models.
• Reinforcement learning to further optimize accuracy.