Loading Property Account Information

Overview

Property accounts in Pie are represented by the Parcel model — one record per (tax_year, account) pair. Parcel records are the source of truth for property data and serve as the foundation for creating Appeals. All parcel data is loaded via CSV import using a configurable schema mapping system.

Prerequisites

1. Schema Profile

Before importing any CSV, a ParcelSchemaProfile must exist and be active. The profile defines how your vendor's CSV column names map to Pie's canonical fields.

Where: Django admin → Parcel Column Import Manager

Steps:

1. Click Add Parcel Schema Profile.
2. Set a name (e.g., esri-default), vendor (esri), and mark Active.
3. Under Column Mappings, add a ParcelFieldMap row for each CSV column:
   • Source Field — exact CSV column header as it appears in the file
   • Canonical Field — the Pie field to map to (see canonical fields below)
   • Transform — how the raw value should be parsed (see transforms below)
   • Required — check at minimum for account
4. Save.

Canonical Fields:

Transforms:

CSV Preparation

• Encode as UTF-8.
• Include a header row; column names must exactly match the source_field values in your schema profile.
• Every row must have a non-blank account value — rows with blank accounts are silently skipped.
• Each import is scoped to a single tax year; the (tax_year, account) pair is unique.

Import Methods

Method A — Azure Blob URL (Recommended for Large Files)

This method queues a background job that downloads, validates, and applies the CSV without risk of HTTP timeout.

1. Upload the CSV to Azure Blob Storage (container doc).
2. Copy the blob URL (e.g., https://caliperdocuments.blob.core.windows.net/doc/imports/Parcels_2026.csv).
3. In the admin, go to Data Tools.
4. Under Parcels.csv Import (Azure Blob URL), paste the URL, enter the tax year, and select the schema profile.
5. Click Import from Blob.
6. A ParcelImportJob with action BLOB_IMPORT is queued. The background worker picks it up and runs the full pipeline (ingest → validate → apply).

Method B — Admin Upload (Small to Medium Files)

This method uploads the file directly through the admin interface and queues background jobs for each pipeline phase.

1. Go to admin → Validate Importations → Add.
2. Enter tax year, select schema profile, optionally enter a source name, and upload the CSV file.
3. Save. This creates a ParcelImportBatch in UPLOADED status with all rows stored as ParcelRawRow records.
4. On the batch detail page, click Queue Validation to normalize and validate all rows (transitions to VALIDATED).
5. After validation, click Queue Apply to upsert Parcel records (transitions to APPLIED).

Method C — Management Commands (CLI / Automated)

Use for scripted or automated imports (e.g., CI pipelines, scheduled tasks).

Full workflow in one command:

python manage.py import_parcels \
  --file /path/to/Parcels_2026.csv \
  --tax-year 2026 \
  --profile esri-default \
  --apply

Step by step:

# 1. Ingest and validate only
python manage.py import_parcels --file /path/to/Parcels_2026.csv --tax-year 2026 --profile esri-default

# 2. Apply a validated batch by ID
python manage.py apply_import --batch <BATCH_ID>

# 3. Re-validate an existing batch
python manage.py validate_import --batch <BATCH_ID>

Import from Azure Blob via CLI:

python manage.py import_parcels_blob \
  --url https://caliperdocuments.blob.core.windows.net/doc/imports/Parcels_2026.csv \
  --tax-year 2026 \
  --profile esri-default \
  --apply

Force re-import of a duplicate file:

python manage.py import_parcels --file /path/to/Parcels_2026.csv --tax-year 2026 --profile esri-default --apply --force

The Import Pipeline

Every import — regardless of method — passes through three phases:

1. Ingest

   • Reads the CSV (UTF-8).
   • Computes a SHA256 hash of the file for deduplication.
   • Stores each row as a ParcelRawRow record (raw field = original CSV row as JSON).
   • Skips rows with a blank account value.
   • Creates a ParcelImportBatch record.
   • Rejects the upload if a batch with the same (tax_year, sha256) already exists (unless --force).

2. Validate / Normalize

   • Applies the schema profile's ParcelFieldMap transforms to each raw row.
   • Requires at least account; any field marked required must be present and parseable.
   • Writes normalized JSON, errors list, and valid flag back to each ParcelRawRow.
   • Updates batch counters: row_count_valid, row_count_invalid.
   • Invalid rows are not applied but remain visible for inspection.

3. Apply / Upsert

   • For each valid row, upserts a Parcel record by (tax_year, account).
   • If assessor_total_assessed_value is absent, it is auto-computed from residential + commercial + agricultural assessed values.
   • Sets current_import_batch and raw_last_seen pointers on the Parcel.
   • Re-applying the same batch is idempotent — existing records are updated in place.

Monitoring Progress

• Data Tools (admin) → Parcel Import Jobs table shows the most recent 10 jobs with status and row counters. Refresh the page to poll for updates.
• Validate Importations → individual batch detail page shows Latest Job Progress with a progress percentage.
• Parcel Import Jobs admin list shows full job history including error messages.

Job statuses: QUEUED → RUNNING → SUCCESS / FAILED

Batch statuses: UPLOADED → VALIDATED → APPLYING → APPLIED / FAILED / DUPLICATE

Verifying the Import

1. Go to admin → Property Account Search.
2. Search by account number or owner name to confirm records were created.
3. Open a parcel record to review all canonical fields, assessed values, and the import batch pointer.

Deduplication Behavior

Troubleshooting

Invalid rows / missing data:

• On the batch detail page, review Top Errors and Invalid Rows (sample).
• Go to admin → Parcel Row Import Manager (superusers) to inspect the raw, normalized, and errors fields for individual rows.
• Common causes:
  • source_field in the profile does not exactly match the CSV column header (including whitespace or non-breaking spaces).
  • Wrong transform selected (e.g., date applied to a decimal column).
  • Required field is blank in some rows.

Job stuck in RUNNING:

• Check that the parcel_import_worker management command is running (it is a long-running daemon process).
• The worker polls for queued jobs every 5 seconds by default.

Duplicate rejection:

• If the import was partially applied and must be re-run, use the --force flag via CLI, or Dump Parcel Records (Data Tools) to wipe all parcel data and start fresh.
• Note: Dump Parcel Records is blocked if any Appeals exist — Appeals must be removed first.

Key Models Reference

Key Code Locations