Working with documents in different formats is a common challenge when building AI applications. Whether you're processing PDFs, Word documents, or HTML files, extracting clean, structured text can be surprisingly difficult. Docling is a Python library that makes this process straightforward.
This guide walks you through the essentials of using Docling to process documents, with a focus on practical examples and best practices you can apply immediately.
Why Docling?
Docling solves common document processing problems in a unified way. It provides multi-format support that works seamlessly with PDFs, Word documents, PowerPoint presentations, HTML, and more. The library includes OCR capabilities that can extract text even from scanned documents and images, making it versatile for various document types.
What sets Docling apart is its smart chunking feature that breaks documents into meaningful pieces while preserving context, rather than arbitrarily splitting text. The output is clean and structured, whether you need markdown or plain text format. Best of all, Docling offers a simple, intuitive API that's easy to get started with, even for developers new to document processing.
Getting Started
First, install Docling:
pip install doclingBasic Usage
Converting a Document
The simplest way to use Docling is with the DocumentConverter:
from docling.document_converter import DocumentConverter
# Create a converter
converter = DocumentConverter()
# Convert a document
result = converter.convert("document.pdf")
document = result.document
# Export to markdown
markdown_text = document.export_to_markdown()
print(markdown_text)That's it! Docling automatically detects the file format and processes it accordingly.
Working with Different File Sources
Docling can process both local files and remote URLs:
from docling.document_converter import DocumentConverter
import requests
import tempfile
converter = DocumentConverter()
# Local file
result = converter.convert("/path/to/document.pdf")
# Remote URL - download first
url = "https://example.com/document.pdf"
response = requests.get(url)
with tempfile.NamedTemporaryFile(suffix='.pdf', delete=False) as tmp:
tmp.write(response.content)
tmp_path = tmp.name
result = converter.convert(tmp_path)
# Don't forget to clean up
import os
os.unlink(tmp_path)What Formats Are Supported?
Docling works with many common file formats out of the box. It handles PDF files, including scanned documents using OCR technology. Microsoft Office formats like Word (.docx) and PowerPoint (.pptx) are fully supported, as are web formats such as HTML. You can also process Markdown files, plain text documents, and even image files (.webp, .webp) using its built-in OCR capabilities.
The DocumentConverter automatically detects the file format and applies the appropriate processing method, so you don't need to worry about specifying the type explicitly.
Chunking Documents
For many AI applications, you need to split documents into smaller pieces ("chunks"). Docling's HybridChunker makes this smart and easy.
Basic Chunking
from docling.document_converter import DocumentConverter
from docling.chunking import HybridChunker
from transformers import AutoTokenizer
# Convert document
converter = DocumentConverter()
result = converter.convert("document.pdf")
document = result.document
# Set up chunker with your tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
chunker = HybridChunker(
tokenizer=tokenizer,
max_tokens=512 # Maximum tokens per chunk
)
# Get chunks
chunks = list(chunker.chunk(document))
# Process chunks
for i, chunk in enumerate(chunks):
print(f"Chunk {i}: {chunk.text[:100]}...")Why Use HybridChunker?
The HybridChunker provides intelligent document splitting that goes beyond simple character or word counts. It preserves natural document structures like paragraphs and sections, ensuring you never get chunks that awkwardly cut off mid-sentence. This is particularly important for maintaining semantic meaning in your text.
- Preserves natural document structures like paragraphs and sections
- Token-aware chunking that respects embedding model limits
- Configurable chunk sizes based on your specific needs
- Preserves metadata tracking where each chunk originated
Working with Metadata
Docling extracts useful metadata from documents:
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
result = converter.convert("document.pdf")
document = result.document
# Access document metadata
if hasattr(document, 'meta'):
print(f"Document metadata: {document.meta}")
# Chunk with metadata
from docling.chunking import HybridChunker
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
chunker = HybridChunker(tokenizer=tokenizer, max_tokens=512)
for chunk in chunker.chunk(document):
print(f"Text: {chunk.text[:50]}...")
if hasattr(chunk, 'meta'):
print(f"Metadata: {chunk.meta}")Putting It All Together
Here's a complete example that processes a document and prepares it for use in an AI application:
from docling.document_converter import DocumentConverter
from docling.chunking import HybridChunker
from transformers import AutoTokenizer
def process_document(file_path, max_tokens=512):
"""Process a document and return chunks ready for embedding."""
# Step 1: Convert document
converter = DocumentConverter()
result = converter.convert(file_path)
document = result.document
# Step 2: Set up chunker
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
chunker = HybridChunker(
tokenizer=tokenizer,
max_tokens=max_tokens
)
# Step 3: Get chunks
chunks = []
for i, chunk in enumerate(chunker.chunk(document)):
chunks.append({
'text': chunk.text,
'index': i,
'source': file_path
})
return chunks
# Usage
chunks = process_document("my_document.pdf")
print(f"Created {len(chunks)} chunks")
for chunk in chunks:
print(f"Chunk {chunk['index']}: {chunk['text'][:50]}...")Practical Tips
Processing Multiple Documents
import os
from pathlib import Path
def process_folder(folder_path, max_tokens=512):
"""Process all PDFs in a folder."""
all_chunks = []
for file_path in Path(folder_path).glob("*.pdf"):
try:
chunks = process_document(str(file_path), max_tokens)
all_chunks.extend(chunks)
print(f"Processed {file_path.name}: {len(chunks)} chunks")
except Exception as e:
print(f"Failed to process {file_path.name}: {e}")
return all_chunksHAQQ AI kostenlos testen
Erleben Sie KI-gestützte juristische Recherche und Entwurf
Exporting to Different Formats
Docling can export documents to various formats:
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
result = converter.convert("document.pdf")
document = result.document
# Export to markdown
markdown = document.export_to_markdown()
with open("output.md", "w", encoding="utf-8") as f:
f.write(markdown)
# Export to plain text
text = document.export_to_text()
with open("output.txt", "w", encoding="utf-8") as f:
f.write(text)Handling Errors Gracefully
def safe_process_document(file_path):
"""Process document with error handling."""
try:
converter = DocumentConverter()
result = converter.convert(file_path)
return result.document
except FileNotFoundError:
print(f"File not found: {file_path}")
return None
except Exception as e:
print(f"Error processing {file_path}: {e}")
return NoneBuilding a Document Search System
One of the most common use cases for Docling is building document search systems powered by AI. By combining Docling's document processing with embedding models, you can create powerful semantic search capabilities.
from docling.document_converter import DocumentConverter
from docling.chunking import HybridChunker
from transformers import AutoTokenizer, AutoModel
import torch
# Process and embed documents
def build_document_index(file_paths):
converter = DocumentConverter()
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
chunker = HybridChunker(tokenizer=tokenizer, max_tokens=512)
document_index = []
for file_path in file_paths:
# Convert and chunk
result = converter.convert(file_path)
chunks = list(chunker.chunk(result.document))
# Create embeddings for each chunk
for chunk in chunks:
inputs = tokenizer(chunk.text, return_tensors="pt",
truncation=True, max_length=512)
with torch.no_grad():
embedding = model(**inputs).last_hidden_state.mean(dim=1)
document_index.append({
'text': chunk.text,
'embedding': embedding,
'source': file_path
})
return document_indexPerformance Tips
Choose the Right Chunk Size
Match your chunk size to your embedding model:
# For most sentence transformers (512 token limit)
chunker = HybridChunker(tokenizer=tokenizer, max_tokens=512)
# For models with larger context windows
chunker = HybridChunker(tokenizer=tokenizer, max_tokens=2048)Process Files in Parallel
from concurrent.futures import ThreadPoolExecutor
def process_multiple_files(file_paths, max_workers=4):
with ThreadPoolExecutor(max_workers=max_workers) as executor:
results = list(executor.map(process_document, file_paths))
return resultsConclusion
Docling makes document processing straightforward by providing a simple API that lets you convert any document with just a few lines of code. Its smart chunking capabilities break documents into meaningful pieces that preserve context and structure, making it ideal for AI applications.
Whether you're building a search system, a chatbot, a document analysis tool, or any AI application that needs to work with documents, Docling provides the foundation you need.
The library's combination of ease of use and powerful features makes it an excellent choice for both prototyping and production applications. With multi-format support for PDFs, Word documents, HTML, and more, plus built-in OCR for scanned documents, Docling handles the complexity of document processing so you don't have to.
Resources
You can find the Docling project on GitHub where you'll find the source code and additional documentation. For working with transformer models and tokenizers, check out the Hugging Face Transformers documentation. The Docling documentation provides more detailed information about advanced features and configuration options.
