Introduction to Minoru

Minoru is an open-source application that converts PDFs into machine-readable formats such as Markdown, JSON, etc. It originated as a byproduct of the InternLM model's pre-training process and has since evolved into a standalone application.

Minoru's Capabilities

Converts PDFs to Markdown, JSON, and other formats.
Removes headers, footers, footnotes, and page numbers while maintaining semantic continuity.
Outputs text in a human-readable order from multi-column documents.
Retains the original document structure (titles, paragraphs, lists).
Extracts images, image captions, tables, and table captions.
Automatically recognizes and converts formulas to LaTeX.
Automatically recognizes and converts tables to LaTeX.
Automatically detects and enables OCR for corrupted PDFs.
Supports CPU and GPU environments (requires a good CPU with ~32GB RAM).
Supports Windows, Linux, and Mac platforms.

Minoru's Development Background

Minoru was developed to address the challenges of symbol conversion in scientific literature during the InternLM model's pre-training. The creators found it difficult to sift through PDF files, particularly technical and research papers, to extract scientific symbols and equations.

Installation and Setup

Cloning the Repository

The Minoru repository can be cloned using Git. The link to the repository is provided in the video description.

git clone [repository link]

Creating a Conda Environment

A Conda environment with Python 3.10 is recommended. The specific commands are available in the video description.

Installing Prerequisites

Prerequisites like magic-pdf need to be installed. The specific commands are in the video description.

# Example (commands from video description)
conda create -n minoru python=3.10
conda activate minoru
pip install -r requirements.txt

Downloading the Model

The Minoru model needs to be downloaded using Git LFS. Commands for this are in the video description.

# Example (commands from video description)
git lfs install
git lfs pull

Configuring the JSON File

The magic-pdf.template.json file needs to be edited to point to the downloaded model directory and specify CPU or CUDA (GPU) usage.

Usage

Running Minoru

To run Minoru, use the following command, replacing [PDF filename] with the path to your PDF file:

magic-pdf [PDF filename]

Output

Minoru outputs the extracted data into an "output" directory, containing files in various formats (e.g., Markdown, JSON).

Performance

The video demonstrates Minoru's performance on both Chinese and English PDF files. While the initial run might take time, subsequent runs are faster, especially when the model is loaded in GPU memory.

Sponsorship

The video is sponsored by Mast Compute, which provides affordable GPU rental services. A link and a 50% discount coupon code are provided in the video description.

Conclusion

Minoru is a promising open-source tool for PDF conversion, offering strong capabilities and a user-friendly experience. While still under development, its features and performance make it a valuable tool for researchers and developers.

Discover content by category