APIBatch API Guide - Visual Learning Edition

Batch API Guide - Visual Learning Edition

This guide provides detailed explanations and diagrams for working with batch compilations in the bloqr-backend API. Perfect for visual learners!

Batch API Guide - Visual Learning Edition

📚 A comprehensive visual guide to using the Batch Compilation API

This guide provides detailed explanations and diagrams for working with batch compilations in the bloqr-backend API. Perfect for visual learners!

Table of Contents

Overview

The Batch API allows you to compile multiple filter lists in a single request. Behind the scenes, it uses Cloudflare Queues for reliable, scalable processing.

Key Benefits

Architecture Diagrams

High-Level System Architecture

Queue Processing Pipeline

Batch Types

Synchronous vs Asynchronous Comparison

When to Use Each Type

API Endpoints

Endpoint Overview

Request Structure Diagram

Request/Response Flow

Synchronous Batch Flow (Detailed)

Asynchronous Batch Flow (Detailed)

Priority Queue Routing

Code Examples

Example 1: Simple Synchronous Batch

Scenario: Compile 3 filter lists and get immediate results

// JavaScript/TypeScript example
const batchRequest = {
    requests: [
        {
            id: 'adguard-dns',
            configuration: {
                name: 'AdGuard DNS Filter',
                sources: [
                    {
                        source: 'https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt',
                        transformations: ['RemoveComments', 'Validate']
                    }
                ],
                transformations: ['Deduplicate', 'RemoveEmptyLines']
            },
            benchmark: true
        },
        {
            id: 'easylist',
            configuration: {
                name: 'EasyList',
                sources: [
                    {
                        source: 'https://easylist.to/easylist/easylist.txt',
                        transformations: ['RemoveComments', 'Compress']
                    }
                ],
                transformations: ['Deduplicate']
            }
        },
        {
            id: 'custom-rules',
            configuration: {
                name: 'Custom Rules',
                sources: [
                    { source: 'my-custom-rules' }
                ]
            },
            preFetchedContent: {
                'my-custom-rules': '||ads.example.com^
||tracking.example.com^'
            }
        }
    ]
};

// Send synchronous batch request
const response = await fetch('https://api.bloqr.dev/compile/batch', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify(batchRequest)
});

const results = await response.json();

// Process results
console.log('Batch compilation complete!');
results.results.forEach(result => {
    console.log(`${result.id}: ${result.ruleCount} rules`);
    console.log(`Compilation time: ${result.metrics?.totalDurationMs}ms`);
});
Expected Response: 
{
    "success": true,
    "results": [
        {
            "id": "adguard-dns",
            "success": true,
            "rules": ["||ads.com^", "||tracker.net^", "..."],
            "ruleCount": 45234,
            "metrics": {
                "totalDurationMs": 2341,
                "sourceCount": 1,
                "transformationMetrics": [...]
            },
            "compiledAt": "2026-01-14T07:30:15.123Z"
        },
        {
            "id": "easylist",
            "success": true,
            "rules": ["||ad.example.com^", "..."],
            "ruleCount": 67891,
            "metrics": {
                "totalDurationMs": 3567
            },
            "compiledAt": "2026-01-14T07:30:16.234Z"
        },
        {
            "id": "custom-rules",
            "success": true,
            "rules": ["||ads.example.com^", "||tracking.example.com^"],
            "ruleCount": 2,
            "metrics": {
                "totalDurationMs": 45
            },
            "compiledAt": "2026-01-14T07:30:15.456Z"
        }
    ]
}

Example 2: Asynchronous Batch with Polling

Scenario: Queue 10 large filter lists for background processing

// JavaScript/TypeScript example with async/await
async function compileBatchAsync() {
    // Step 1: Queue the batch
    const batchRequest = {
        requests: [
            // ... 10 compilation requests
            { id: 'list-1', configuration: { /* ... */ } },
            { id: 'list-2', configuration: { /* ... */ } },
            { id: 'list-3', configuration: { /* ... */ } },
            // ... up to list-10
        ]
    };
    
    const queueResponse = await fetch(
        'https://api.bloqr.dev/compile/batch/async',
        {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify(batchRequest)
        }
    );
    
    const queueData = await queueResponse.json();
    console.log('Batch queued:', queueData.requestId);
    
    // Step 2: Poll for results
    const requestId = queueData.requestId;
    let results = null;
    let attempts = 0;
    const maxAttempts = 10;
    
    while (!results && attempts < maxAttempts) {
        // Wait 30 seconds between polls
        await new Promise(resolve => setTimeout(resolve, 30000));
        
        const statusResponse = await fetch(
            `https://api.bloqr.dev/queue/results/${requestId}`
        );
        
        const statusData = await statusResponse.json();
        
        if (statusData.status === 'completed') {
            results = statusData.results;
            console.log('Batch complete! Got results for', results.length, 'items');
        } else if (statusData.status === 'failed') {
            throw new Error('Batch compilation failed: ' + statusData.error);
        } else {
            console.log('Still processing... attempt', ++attempts);
        }
    }
    
    if (!results) {
        throw new Error('Timeout waiting for results');
    }
    
    return results;
}

// Usage
try {
    const results = await compileBatchAsync();
    results.forEach(result => {
        console.log(`${result.id}: ${result.ruleCount} rules`);
    });
} catch (error) {
    console.error('Batch compilation error:', error);
}

Example 3: Python with Requests Library

import requests
import time
from typing import List, Dict

BASE_URL = 'https://api.bloqr.dev'

def compile_batch_async(requests_data: List[Dict]) -> List[Dict]:
    """
    Compile multiple filter lists asynchronously
    
    Args:
        requests_data: List of compilation requests (max 10)
    
    Returns:
        List of compilation results
    """
    
    # Step 1: Queue the batch
    response = requests.post(
        f'{BASE_URL}/compile/batch/async',
        json={'requests': requests_data}
    )
    response.raise_for_status()
    
    queue_data = response.json()
    request_id = queue_data['requestId']
    print(f'📬 Batch queued: {request_id}')
    print(f'⚡ Priority: {queue_data["priority"]}')
    
    # Step 2: Poll for results
    max_attempts = 20
    poll_interval = 30  # seconds
    
    for attempt in range(max_attempts):
        print(f'⏳ Checking status (attempt {attempt + 1}/{max_attempts})...')
        
        response = requests.get(f'{BASE_URL}/queue/results/{request_id}')
        response.raise_for_status()
        
        data = response.json()
        
        if data.get('status') == 'completed':
            print('✅ Batch compilation complete!')
            return data['results']
        elif data.get('status') == 'failed':
            raise Exception(f'Batch failed: {data.get("error")}')
        else:
            if attempt < max_attempts - 1:
                print(f'⌛ Still processing, waiting {poll_interval} seconds...')
                time.sleep(poll_interval)
    
    raise TimeoutError('Timeout waiting for batch completion')

# Example usage
if __name__ == '__main__':
    batch_requests = [
        {
            'id': 'adguard',
            'configuration': {
                'name': 'AdGuard DNS',
                'sources': [
                    {
                        'source': 'https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt'
                    }
                ],
                'transformations': ['Deduplicate', 'RemoveEmptyLines']
            },
            'benchmark': True
        },
        {
            'id': 'easylist',
            'configuration': {
                'name': 'EasyList',
                'sources': [
                    {
                        'source': 'https://easylist.to/easylist/easylist.txt'
                    }
                ],
                'transformations': ['Deduplicate']
            }
        }
    ]
    
    try:
        results = compile_batch_async(batch_requests)
        
        print('
📊 Results Summary:')
        for result in results:
            print(f"  {result['id']}: {result['ruleCount']} rules")
            print(f"    Time: {result['metrics']['totalDurationMs']}ms")
    
    except Exception as e:
        print(f'❌ Error: {e}')

Example 4: cURL Commands

# Example: Synchronous batch compilation
curl -X POST https://api.bloqr.dev/compile/batch \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {
        "id": "test-1",
        "configuration": {
          "name": "Test List 1",
          "sources": [
            {
              "source": "my-rules-1"
            }
          ]
        },
        "preFetchedContent": {
          "my-rules-1": "||ads.com^
||tracker.net^"
        }
      },
      {
        "id": "test-2",
        "configuration": {
          "name": "Test List 2",
          "sources": [
            {
              "source": "my-rules-2"
            }
          ]
        },
        "preFetchedContent": {
          "my-rules-2": "||spam.org^
||malware.biz^"
        }
      }
    ]
  }'

# Example: Asynchronous batch compilation

# Step 1: Queue the batch
curl -X POST https://api.bloqr.dev/compile/batch/async \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {
        "id": "large-list-1",
        "configuration": {
          "name": "Large Filter List",
          "sources": [
            {
              "source": "https://example.com/large-list.txt"
            }
          ],
          "transformations": ["Deduplicate", "Compress"]
        }
      }
    ]
  }'

# Response will include a requestId, e.g.:
# {
#   "success": true,
#   "requestId": "req-1704931200000-abc123",
#   "priority": "standard"
# }

# Step 2: Check status (wait 30 seconds, then run this)
curl https://api.bloqr.dev/queue/results/req-1704931200000-abc123

# If still processing, you'll get:
# {
#   "success": true,
#   "status": "processing"
# }

# When complete, you'll get full results:
# {
#   "success": true,
#   "status": "completed",
#   "results": [...]
# }

Best Practices

Batch Size Optimization

Error Handling Strategy

Caching Strategy

Performance Tips

Troubleshooting

Common Issues and Solutions

Debugging Workflow

Queue Status Monitoring

Quick Reference

API Endpoints Summary

EndpointMethodPurposeReturns
/compile/batchPOSTSynchronous batch compilationImmediate results
/compile/batch/asyncPOSTAsynchronous batch compilationRequest ID
/queue/results/:idGETGet async resultsResults or status
/queue/statsGETQueue statisticsMetrics

Request Limits

Decision Matrix

Cancelling Async Jobs

A queued compilation job can be cancelled while it is still pending:

DELETE /api/queue/cancel/{requestId}
Authorization: Bearer <token>
StatusMeaning
200Job cancelled successfully
401Authentication required
404Job not found (may have already expired)
409Job already completed, failed, or cancelled

Once a job has been picked up by the queue consumer it cannot be cancelled retroactively.

Need Help?

Last updated: 2026-01-14

Was this page helpful?