Blob Storage Export

The Blob Storage Export feature continuously exports your XeroML data to cloud storage — enabling large-scale data analysis, long-term retention, and integration with data warehouses like BigQuery, Snowflake, or Redshift.

Supported Destinations

Provider	Service
AWS	S3
Google Cloud	Cloud Storage (GCS)
Microsoft Azure	Blob Storage

What Gets Exported

• Traces (with all attributes)
• Observations (spans, generations, events)
• Scores
• Sessions

Data is exported in JSONL format, partitioned by date.

Setup

AWS S3

1. Navigate to Project Settings → Exports
2. Click Add Export Destination → S3
3. Enter your bucket name and region
4. Configure IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

5. Enter the IAM role ARN or access key credentials
6. Set the export schedule (hourly, daily)
7. Click Save

Google Cloud Storage

1. Create a service account with Storage Object Creator role on your bucket
2. Download the service account JSON key
3. In XeroML: Project Settings → Exports → Add GCS
4. Upload the service account key and enter the bucket name

Azure Blob Storage

1. Create a storage account and container
2. Generate a SAS token with read/write permissions
3. In XeroML: Project Settings → Exports → Add Azure Blob
4. Enter the connection string and container name

Export Format

Data is exported as JSONL files, partitioned by date and data type:

s3://your-bucket/xeroml-export/
  traces/
    2025-01-15.jsonl
    2025-01-16.jsonl
  observations/
    2025-01-15.jsonl
  scores/
    2025-01-15.jsonl

Each line is a valid JSON object representing one record.

Incremental Exports

By default, XeroML exports only new or updated records since the last export run. This minimizes storage and processing costs for large projects.

Loading into a Data Warehouse

BigQuery:

bq load --source_format=NEWLINE_DELIMITED_JSON \
  my-dataset.xeroml_traces \
  gs://your-bucket/xeroml-export/traces/2025-01-15.jsonl

Snowflake:

COPY INTO xeroml_traces
FROM @my_s3_stage/xeroml-export/traces/
FILE_FORMAT = (TYPE = 'JSON');