You want to filter or curate URLs before extraction
You need targeted extraction with query and chunks_per_source
Key difference: The main distinction is control, with Extract you choose exactly which URLs to extract from, while Search with include_raw_content extracts from all search results.
Extract Depth
The extract_depth parameter controls extraction comprehensiveness:
extract_depth=advanced provides better accuracy but increases latency and
cost. Use basic for simple content.
Advanced Filtering Strategies
Beyond query-based filtering, consider these approaches for curating URLs before extraction:
Strategy
When to use
Re-ranking
Use dedicated re-ranking models for precision
LLM-based
Let an LLM assess relevance before extraction
Clustering
Group similar documents, extract from clusters
Domain-based
Filter by trusted domains before extracting
Score-based
Filter search results by relevance score
Example: Score-based filtering
import asynciofrom tavily import AsyncTavilyClienttavily_client = AsyncTavilyClient(api_key="tvly-YOUR_API_KEY")async def filtered_extraction(): # Search first response = await tavily_client.search( query="AI healthcare applications", search_depth="advanced", max_results=20 ) # Filter by relevance score (>0.5) relevant_urls = [ result['url'] for result in response.get('results', []) if result.get('score', 0) > 0.5 ] # Extract from filtered URLs with targeted query extracted_data = await tavily_client.extract( urls=relevant_urls, query="machine learning diagnostic tools", chunks_per_source=3, extract_depth="advanced" ) return extracted_dataasyncio.run(filtered_extraction())
Integration with Search
Optimal workflow
Search to discover relevant URLs
Filter by relevance score, domain, or content snippet
Re-rank if needed using specialized models
Extract from top-ranked sources with query and chunks_per_source
Validate extracted content quality
Process for your RAG or AI application
Example end-to-end pipeline
Extract accepts up to 20 URLs in one request. This example keeps the first 20 unique URLs selected from the search results and submits them together.
async def content_pipeline(topic): # 1. Search with sub-queries queries = generate_subqueries(topic) responses = await asyncio.gather( *[tavily_client.search(**q) for q in queries] ) # 2. Filter and aggregate urls = [] for response in responses: urls.extend([ r['url'] for r in response['results'] if r['score'] > 0.5 ]) # 3. Deduplicate while preserving the selected order urls = list(dict.fromkeys(urls))[:20] if not urls: return {"results": [], "failed_results": []} # 4. Extract once, retaining successes and per-URL failures return await tavily_client.extract(urls=urls, extract_depth="advanced")
The function returns one Extract response dictionary: results contains successful extractions and failed_results contains per-URL failures. If no URLs qualify, both lists are empty and no Extract request is sent. Authentication and network errors raise exceptions that the caller must handle.
Use Session Tracking for Multi-Step Workflows
When an agent issues several Tavily calls to answer a single user task — for example, retrieving sources, then extracting full content from a subset, then running follow-up searches — pass a consistent session_id across all related calls.If your agent serves multiple end-users behind a single API key, also pass a stable human_id per user. For security, Tavily hashes human IDs before processing or storing them.See the SDK references or the API HTTP headers for how to set these.
Summary
Use query and chunks_per_source for targeted, focused extraction
Choose Extract API when you need control over which URLs to extract from
Filter URLs before extraction using scores, re-ranking, or domain trust
Choose appropriate extract_depth based on content complexity
Batch up to 20 URLs in each Extract request
Implement error handling to manage failed extractions gracefully
Validate extracted content before downstream processing
Optimize costs by extracting only necessary content with chunks_per_source
Use session_id and human_id to link related calls across multi-step agent workflows
Start with query and chunks_per_source for targeted extraction. Filter URLs strategically, extract with appropriate depth, and handle errors gracefully for production-ready pipelines.
Assistant
Responses are generated using AI and may contain mistakes.