Parcel Schema Profiles

This section is a required prerequisite for Loading Property Account Information.

What Is a ParcelSchemaProfile?

A ParcelSchemaProfile is a named, reusable translation table that tells Pie how to read a vendor's CSV export. Because different county assessors and GIS platforms (such as Esri/ArcGIS) use their own column naming conventions, Pie does not assume a fixed column layout. Instead, every import requires a profile that explicitly maps each CSV column header to one of Pie's canonical internal fields.

Without a profile, Pie cannot interpret any parcel data - the import pipeline will refuse to proceed.

Why This Matters for Esri / ArcGIS Exports

Property assessment data exported from Esri/ArcGIS (or any GIS-based assessor platform) typically has column names that reflect the platform's own data model rather than any universal standard. For example, a situs address column might be called SiteAddress, SitusAddr, PropertyAddress, or something else entirely depending on how the jurisdiction has configured their system.

A ParcelSchemaProfile bridges this gap:

• It records the exact column names present in the vendor export.
• It specifies how each value should be parsed or normalized (for example, trimming whitespace, parsing a decimal currency field, converting a date from epoch milliseconds).
• It marks which columns are required for a row to be considered valid.

Profiles are reusable across tax years - once configured for a given assessor extract layout, the same profile can be used for every annual import as long as the column structure does not change.

Creating a Profile

Step 1 - Open the admin page

Go to Django admin -> Parcel Column Import Manager.

This section is visible to superusers only.

Step 2 - Add a new profile

Click Add Parcel Schema Profile and fill in:

Step 3 - Add column mappings

Under Column Mappings, add one row per CSV column you want to import. Each row has:

Tip: After saving, use the Auto-fill Friendly Labels action to populate any blank friendly labels automatically from the canonical field name.

Step 4 - Save

Save the profile. It is immediately available for use in any import method (UI or CLI).

Importing and Exporting Profiles via JSON

Profiles can be exported to a JSON file and re-imported on another environment (for example, from local to production, or to back up a configuration). This avoids manual re-entry.

Export

1. Go to Data Tools.
2. Under the schema profile export section, select the profile and click Export.
3. A .json file is downloaded in the pie.parcel-schema-profile format.

Import

1. Go to Data Tools.
2. Under the schema profile import section, upload the .json file.
3. Pie validates the file and upserts the profile by name:
   • If the profile name does not exist, it is created.
   • If it already exists, its vendor/description/active fields are updated and all existing column mappings are replaced with the imported ones.

Warning: Importing a profile with an existing name will delete and replace all of its current column mappings. This is intentional - the import is designed to be idempotent.

The esri-default Profile (Reference)

The esri-default profile is the standard mapping for Esri/ArcGIS property assessment exports. Its column mappings are:

Note that assessor_total_assessed_value is not imported directly - it is auto-computed by the pipeline from the sum of residential, commercial, and agricultural assessed values if not explicitly provided.

The JSON representation of this profile (suitable for import) follows the pie.parcel-schema-profile format:

{
  "format": "pie.parcel-schema-profile",
  "format_version": 1,
  "profile": {
    "name": "esri-default",
    "vendor": "esri",
    "description": "Default ESRI mapping",
    "active": true,
    "field_maps": [
      { "source_field": "Account", "canonical_field": "account", "transform": "strip", "required": true, "friendly_label": "Account" },
      { "source_field": "AgriculturalAssessedValue", "canonical_field": "ag_assessed_value", "transform": "decimal", "required": false, "friendly_label": "Ag Assessed Value" },
      { "source_field": "AgriculturalImprovementValue", "canonical_field": "ag_improvement_value", "transform": "decimal", "required": false, "friendly_label": "Ag Improvement Value" },
      { "source_field": "AgriculturalLandValue", "canonical_field": "ag_land_value", "transform": "decimal", "required": false, "friendly_label": "Ag Land Value" },
      { "source_field": "CommercialAssessedValue", "canonical_field": "com_assessed_value", "transform": "decimal", "required": false, "friendly_label": "Com Assessed Value" },
      { "source_field": "CommercialImprovementValue", "canonical_field": "com_improvement_value", "transform": "decimal", "required": false, "friendly_label": "Com Improvement Value" },
      { "source_field": "CommercialLandValue", "canonical_field": "com_land_value", "transform": "decimal", "required": false, "friendly_label": "Com Land Value" },
      { "source_field": "MailingAddress", "canonical_field": "mailing_address", "transform": "strip", "required": false, "friendly_label": "Mailing Address" },
      { "source_field": "Municipality", "canonical_field": "municipality", "transform": "strip", "required": false, "friendly_label": "Municipality" },
      { "source_field": "Owner", "canonical_field": "owner_name", "transform": "strip", "required": false, "friendly_label": "Owner Name" },
      { "source_field": "Parcel_ID", "canonical_field": "parcel_id", "transform": "strip", "required": false, "friendly_label": "Parcel Id" },
      { "source_field": "ParcelReport", "canonical_field": "parcel_report_url", "transform": "strip", "required": false, "friendly_label": "Parcel Report Url" },
      { "source_field": "PrevOwner1", "canonical_field": "prev_owner_1", "transform": "strip", "required": false, "friendly_label": "Prev Owner 1" },
      { "source_field": "ResidentialAssessedValue", "canonical_field": "res_assessed_value", "transform": "decimal", "required": false, "friendly_label": "Res Assessed Value" },
      { "source_field": "ResidentialImprovementValue", "canonical_field": "res_improvement_value", "transform": "decimal", "required": false, "friendly_label": "Res Improvement Value" },
      { "source_field": "ResidentialLandValue", "canonical_field": "res_land_value", "transform": "decimal", "required": false, "friendly_label": "Res Land Value" },
      { "source_field": "SalePrice1", "canonical_field": "sale_price_1", "transform": "decimal", "required": false, "friendly_label": "Sale Price 1" },
      { "source_field": "SiteAddress", "canonical_field": "situs_address", "transform": "strip", "required": false, "friendly_label": "Situs Address" },
      { "source_field": "SitusZip", "canonical_field": "situs_zip", "transform": "strip", "required": false, "friendly_label": "Situs Zip" },
      { "source_field": "TotalMarketValue", "canonical_field": "assessor_total_market_value", "transform": "decimal", "required": false, "friendly_label": "Assessor Total Market Value" }
    ]
  },
  "compatibility": {
    "dropped_legacy_mappings": []
  }
}

Canonical Fields Reference

These are all the fields Pie can store on a Parcel record. Every column mapping must point to one of these.

assessor_total_assessed_value is automatically computed as res_assessed_value + com_assessed_value + ag_assessed_value if it is not mapped or the mapped value is blank.

Transforms Reference

Tips for Working With Esri / ArcGIS Exports

1. Open the CSV in a text editor first. Confirm the exact header row. Esri exports sometimes include non-breaking spaces (\u00a0) in column names - Pie strips these automatically during ingest, but confirming headers upfront prevents confusion.

2. Match source_field exactly. Source field matching is case-sensitive and whitespace-sensitive. If the CSV header is Parcel_ID (with underscore), the source_field must be Parcel_ID, not ParcelID or parcel_id.

3. Use decimal for all currency/value columns. Esri exports often format numbers with commas (1,234,567.00). The decimal transform handles this; identity will not parse it as a number.

4. Date columns from feature services use epoch ms. If you are exporting directly from an Esri feature service (REST API) rather than a file geodatabase export, date values may be integer epoch milliseconds (for example, 1737936000000). Use date_epoch_ms for those columns. Standard file exports typically use YYYY-MM-DD strings; use date for those.

5. Only map the columns you have. You do not need to map every canonical field. Unmapped canonical fields will simply be blank on the resulting Parcel record. Map only what your assessor extract actually provides.

6. One profile per extract layout. If different counties use different column names, create a separate profile for each. Profiles are cheap to create and the name makes it clear which profile to use for which source.

7. Re-use across tax years. A profile does not encode a tax year. The same esri-default profile works for 2025, 2026, and beyond as long as the column structure has not changed.

Key Code Locations

Data Tools (/admin/data-tools/)

What Is Data Tools?

Data Tools is the central admin page for bulk data operations in Pie. It provides export and import controls for every major dataset, parcel-specific import tooling, schema profile management, and a full-instance migration system. All actions on this page require Django admin staff access — every route is wrapped in admin.site.admin_view().

Page Layout

The page is divided into the following sections, in order:

1. Parcel Background Jobs — conditionally shown; lists the 10 most recent parcel import jobs with status and row counters. Refresh the page to update.
2. Dataset Cards — side-by-side export/import cards for every standard dataset.
3. Parcel Schema Export / Import — JSON transfer of a ParcelSchemaProfile between environments.
4. Migration Ledger Export / Import — full-instance data migration via a deterministic JSON snapshot.
5. Database Manifest Export — export-only manifest of every Pie model/table.
6. Audit Logs Export — export-only full audit log.
7. Parameter Manifest Export — export-only field-level schema manifest.
8. Parcel Sections (bottom of page):
   • Parcels.csv Import (Upload File)
   • Parcels.csv Import (Azure Blob URL)
   • Dump Parcel Records

Parcel Background Jobs

When any parcel import job is active or has run recently, a status table appears at the top of the page showing:

Progress does not auto-refresh — reload the page to get the latest status.

Standard Dataset Cards

For each dataset below, there is an Export card (downloads a CSV immediately) and an Import card (uploads a CSV and applies it). Datasets with extra fields require additional form inputs before import.

Export-only datasets (no import card):

Parcel Schema Export / Import

Transfers a ParcelSchemaProfile (column mappings) between Pie environments as a JSON file.

Export

• Select a profile from the dropdown and click Download ParcelSchemaProfile.json.
• Produces a pie.parcel-schema-profile JSON file containing the profile settings and all column mappings.
• URL: POST /admin/data-tools/export/parcel-schema/

Import

• Upload a previously exported .json file and click Import ParcelSchemaProfile.json.
• Matches by profile name:
  • If the name does not exist, a new profile is created.
  • If the name already exists, vendor/description/active are updated and all existing column mappings are deleted and replaced with the imported ones.
• URL: POST /admin/data-tools/import/parcel-schema/

See parcel_schema_profile.md for the full reference on schema profiles.

Migration Ledger Export / Import

The Migration Ledger is a full-instance data snapshot in JSON format, designed for moving an entire Pie configuration from one environment to another (e.g., local → staging → production).

Export

• Click Download MigrationLedger.json.
• Produces a timestamped MigrationLedger_YYYYMMDD_HHMMSS.json file.
• Excludes: audit log records, parcel staging tables (ParcelRawRow, ParcelImportBatch, ParcelImportJob), and document file bytes (document rows include metadata only, not the binary file content).
• URL: GET /admin/data-tools/export/migration-ledger/

Import

• Upload a MigrationLedger.json file and click Import MigrationLedger.json.
• The import is:
  • Dependency-aware — processes models in foreign key dependency order.
  • Primary-key-preserving — records are created with their original IDs.
  • Many-to-many aware — M2M links are applied after all base rows are written.
  • Idempotent — rows are created or updated, not duplicated.
• After import, Django messages report: models processed, created, updated, skipped, and error counts. Warnings and row errors (up to 8 each) are shown inline.
• URL: POST /admin/data-tools/import/migration-ledger/

Parcels.csv Import (Upload File)

For importing a local parcel CSV directly through the browser.

Form fields:

• Tax year — the assessment year for this import (e.g., 2026)
• Schema profile — the ParcelSchemaProfile that maps the CSV column headers to canonical Parcel fields
• File — the .csv file to upload

What happens on submit:

1. The file is uploaded and a ParcelImportBatch is created.
2. A background ParcelImportJob with action APPLY is queued (validate + apply runs automatically).
3. The parcel_import_worker process must be running to pick up the job.

URL: POST /admin/data-tools/import/parcels/

For large files that may cause HTTP timeouts, use the Azure Blob URL method below instead.

Parcels.csv Import (Azure Blob URL)

For importing a parcel CSV that has been pre-uploaded to Azure Blob Storage. This method runs entirely in the background and avoids browser timeout issues.

Steps shown on the page:

1. Upload the CSV to the doc container in Azure Storage.
2. Copy the blob URL (starts with https://caliperdocuments.blob.core.windows.net/doc/).
3. Fill in the form and click Import from Blob.
4. Monitor progress in the Parcel Background Jobs section by refreshing the page.
5. For detailed status and row counts, go to Parcel Teleporter (Start Import) (the ParcelImportBatch admin list).

Form fields:

• Blob URL — full HTTPS URL to the blob (e.g., https://caliperdocuments.blob.core.windows.net/doc/imports/Parcels_2026.csv)
• Tax year — the assessment year
• Schema profile — the active ParcelSchemaProfile to use

What happens on submit:

1. URL is validated (must be https://, must include container/blob path).
2. A ParcelImportJob with action BLOB_IMPORT is queued.
3. The worker downloads the file from blob storage, runs ingest → validate → apply automatically.
4. Job status is stored in the session and shown at the top of the page on next load.

URL: POST /admin/data-tools/import/parcels-blob/

Dump Parcel Records

Wipes all parcel data from the database — Parcel, ParcelImportBatch, and ParcelRawRow records.

• Clicking Dump Parcel Records opens a confirmation page (/admin/data-tools/dump-parcels/) that shows current record counts.
• The confirmation page requires typing delete before the action proceeds.
• Blocked if any Appeals exist — the dump cannot run while Appeal records are present.
• On confirmation, queues a ParcelImportJob with action DUMP_PARCELS and logs an audit entry.
• The actual deletion is executed by the parcel_import_worker background process.

This is irreversible. Use only when preparing to re-import a clean parcel dataset from scratch.

Access Control

Every URL under /admin/data-tools/ is wrapped in admin.site.admin_view(), which enforces Django admin staff authentication. Non-staff users are redirected to the admin login page.

URL Reference

Key Code Locations

Parcel Column Import Manager (/admin/appeals/parcelschemaprofile/)

What Is This Page?

The Parcel Column Import Manager is the admin section for creating and editing ParcelSchemaProfile records. A schema profile is the translation table that tells Pie how to read a vendor's CSV export — mapping each source column header to one of Pie's canonical parcel fields and specifying how the raw value should be parsed.

Every parcel import requires a profile. This page is where profiles are built and maintained.

Access: Visible and accessible to superusers only. Staff users cannot see this section in the navigation even if they have admin access. This is enforced by get_model_perms() in ParcelSchemaProfileAdmin.

List View

URL: /admin/appeals/parcelschemaprofile/

Columns

Filters

• Vendor — filter by data platform
• Active — filter by active/inactive status

Search

Searches across name and description fields.

Action: Auto-fill Friendly Labels

Selecting one or more profiles and running Auto-fill Friendly Labels (leave existing values untouched) fills any blank friendly_label values on all column mappings for the selected profiles. Labels are generated by title-casing the canonical field name (e.g., res_assessed_value → Res Assessed Value). Mappings that already have a label are not touched.

This is a cosmetic operation — friendly labels are displayed in the UI but never used by the import pipeline.

Detail / Edit View

URL: /admin/appeals/parcelschemaprofile/<id>/change/

Profile Fields

Column Mappings Inline

Below the profile fields is the Column Mappings tabular inline. Each row maps one CSV column to one canonical Parcel field.

Rows are ordered by profile, then source field alphabetically.

Canonical Fields

The complete list of fields a column mapping can target:

assessor_total_assessed_value is automatically computed by the pipeline as res_assessed_value + com_assessed_value + ag_assessed_value if it is not present in the CSV or the mapped value is blank. You do not need to map it unless the source file provides it directly.

Transforms

Constraints and Validation

• Name is unique across all profiles — two profiles cannot share the same name.
• Source field is unique within a profile — a given CSV column can only be mapped once per profile. Attempting to add a duplicate source_field on the same profile raises a validation error.
• The import pipeline only applies mappings to columns it finds in the CSV header. Mappings for columns not present in the file are silently skipped (unless the mapping is marked Required, in which case the row is marked invalid).
• Mappings for legacy canonical fields (objectid, situs_name) that have been removed from the Parcel model are silently dropped during JSON export and import. They cannot be added via the admin dropdown as they are not in the CANONICAL_FIELDS list.

Relationship to Imports

When an import runs (via Data Tools, Validate Importations, or CLI), the selected profile is loaded by _load_field_map() in the pipeline service. The pipeline:

1. Reads the ParcelFieldMap rows for the selected profile.
2. For each CSV row, looks up each column header in the field map.
3. Applies the specified transform to the raw value.
4. Stores the result under the canonical field name in the normalized JSON.
5. Checks required fields — marks the row invalid if any are missing or unparseable.

The profile is stored as a foreign key on the ParcelImportBatch record (schema_profile), so it is always possible to see which profile was used for a given import.

Export and Import via JSON

Profiles can be transferred between environments without manual re-entry using the JSON export/import on Data Tools.

• Export: Data Tools → Parcel Schema Export → select profile → Download ParcelSchemaProfile.json
• Import: Data Tools → Parcel Schema Import → upload .json file

The import matches by profile name and replaces all column mappings atomically. See parcel_schema_profile.md for the full JSON format reference.

Key Code Locations

Parcel Import Jobs (/admin/appeals/parcelimportjob/)

What Is This Page?

Parcel Import Jobs is the admin section for monitoring and inspecting background jobs in the parcel import pipeline. Every validate, apply, blob import, dump, and raw-row purge operation runs as a ParcelImportJob record. This page is the authoritative history of all background parcel operations — what ran, when, who queued it, and what happened.

Jobs are created (queued) by:

• Data Tools — Parcels.csv Import (file upload or Azure Blob URL)
• Data Tools — Dump Parcel Records
• Validate Importations — Queue Validation / Queue Apply buttons and bulk actions
• Parcel Row Import Manager — Queue background purge action

The parcel_import_worker management command is the process that picks up and executes queued jobs. Nothing runs unless the worker is running.

Access: Viewable by all staff users. No staff user can add or manually edit a job — the admin is entirely read-only. Jobs are created only through the queue system.

List View

URL: /admin/appeals/parcelimportjob/

Jobs are ordered newest first (-created_at).

Columns

Filters

• Action — filter by job type
• Status — filter by lifecycle state

Search

Searches by job ID or batch ID.

Action Types

Job Statuses

Status transitions are always: QUEUED → RUNNING → SUCCESS or FAILED. There is no retry — a failed job remains failed. To retry, a new job must be queued (e.g., by clicking Queue Validation or Queue Apply again on the batch).

Detail View

URL: /admin/appeals/parcelimportjob/<id>/change/

All fields are read-only. No field can be edited through the admin.

Fields

Payload Contents by Action

Job Deduplication

The queue system prevents redundant jobs from piling up. When a job is queued via queue_parcel_job():

• For batch-linked jobs (VALIDATE, APPLY): if a job with the same action and batch is already QUEUED or RUNNING, the existing job is returned and no new record is created.
• For non-batch jobs (DUMP_PARCELS, PURGE_RAW_ROWS): deduplication is by dedupe_key in the payload. If a job with the same action, no batch, and matching dedupe_key is already active, the existing job is returned.

This means clicking Queue Validation twice for the same batch will not create two jobs — the second click is silently a no-op.

The Worker Process

Jobs do not execute automatically — they require the parcel_import_worker management command to be running as a long-lived daemon process.

python manage.py parcel_import_worker

How the worker operates:

1. Polls for QUEUED jobs every 5 seconds (default; configurable with --sleep N).
2. Claims the oldest queued job using SELECT FOR UPDATE SKIP LOCKED to prevent concurrent workers from claiming the same job.
3. Transitions the job to RUNNING and records started_at.
4. Executes the action.
5. On success: sets status to SUCCESS, progress_message = Completed, records finished_at.
6. On exception: sets status to FAILED, writes the exception to last_error, records finished_at.

Progress updates are written to the job record every 250 rows or every 2 seconds (throttled by the ProgressTracker in the pipeline service). Refreshing the list view or detail view shows the latest progress.

Single-run mode (useful for testing or one-off manual execution):

python manage.py parcel_import_worker --once

What to Check When a Job Fails

1. Open the failed job's detail view.
2. Read Last Error — this is the raw exception message from the worker.
3. Common failure causes:

4. To retry after fixing the root cause, re-queue a new job from the batch detail page or Data Tools — do not attempt to edit the failed job record.

Relationship to Other Pages

Key Code Locations

Parcel Row Import Manager (/admin/appeals/parcelrawrow/)

What Is This Page?

The Parcel Row Import Manager is the admin section for inspecting individual ParcelRawRow records — one record per CSV row per import batch. Every row ingested from a parcel CSV is stored here in its original and normalized forms, along with any validation errors produced during the validate phase.

This page is primarily a diagnostic tool. When an import has invalid rows, this is where you come to see exactly what was in the source data, what the pipeline produced after applying transforms, and which specific validation errors caused a row to be rejected.

Access: Visible to superusers only. Regular staff users cannot see this section in the navigation. This is enforced by get_model_perms() in ParcelRawRowAdmin.

List View

URL: /admin/appeals/parcelrawrow/

Rows are ordered by row_number (ascending) within each batch.

Columns

Filters

• Valid — show only valid or only invalid rows
• Import Batch — filter to all rows for a specific batch

Search

Searches by normalized_key (account value).

Action: Queue Background Purge

Selecting rows and running Queue background purge for selected import batch(es) queues a PURGE_RAW_ROWS job for the import batches represented in the selection.

Important behavior:

• The purge operates at the batch level, not the row level. Selecting any row from a batch targets all raw rows for that entire batch — not just the selected rows.
• Deduplication prevents a second purge job from being queued if one is already QUEUED or RUNNING for the same batch set.
• If a matching job already exists, the action reports the existing job ID and takes no further action.
• The default Django Delete selected action is removed — raw rows cannot be deleted individually through the admin. Deletion must go through the background purge job.

Detail View

URL: /admin/appeals/parcelrawrow/<id>/change/

All fields are read-only. No field can be edited.

Fields

How Raw Rows Are Created

During the ingest phase of the pipeline:

1. The CSV is read line by line.
2. Each row is stored as a ParcelRawRow with raw = the original CSV row dict and valid = False.
3. Rows with a blank account field are skipped entirely — they produce no ParcelRawRow record.
4. Rows are bulk-created in batches for efficiency.

At this point normalized, errors, and normalized_key are all unpopulated.

How Rows Are Validated

During the validate phase:

1. The pipeline loads the schema profile's field maps.
2. For each raw row, it applies the transform for each mapped column.
3. The result is stored in normalized as a canonical-field-keyed JSON object.
4. Any field marked required that is missing or unparseable is recorded in errors.
5. valid is set to True if errors is empty, False otherwise.
6. normalized_key is set to the account value extracted from the normalized output.

Reading the Raw and Normalized Fields

raw example

This is exactly what was in the CSV:

{
  "Account": "  1234-5678  ",
  "Owner": "SMITH JOHN",
  "SiteAddress": "123 MAIN ST",
  "ResidentialAssessedValue": "142,500.00",
  "TotalMarketValue": "189,000.00"
}

normalized example

After applying the schema profile transforms (strip, strip, strip, decimal, decimal):

{
  "account": "1234-5678",
  "owner_name": "SMITH JOHN",
  "situs_address": "123 MAIN ST",
  "res_assessed_value": "142500.00",
  "assessor_total_market_value": "189000.00"
}

errors example for an invalid row

[
  "Required field 'account' is missing or blank."
]

or for a transform failure:

[
  "Field 'res_assessed_value': cannot convert 'N/A' to decimal."
]

Raw Row Retention

Raw rows are retained indefinitely after import — they are not automatically deleted when a batch is applied. This is intentional: they serve as the permanent audit record of exactly what source data was imported.

To free database space after a batch has been applied and verified, use the Queue background purge action on this page, or the same action available from the batch detail page in Validate Importations. The PURGE_RAW_ROWS job deletes all ParcelRawRow records for the targeted batches without affecting the Parcel records that were created from them.

Relationship to Other Pages

Key Code Locations

Validate Importations (/admin/appeals/parcelimportbatch/)

What Is This Page?

Validate Importations is the admin section for managing ParcelImportBatch records. A batch represents one uploaded CSV file — it tracks the file itself, which tax year and schema profile it belongs to, its current pipeline status, and all row counts and errors produced during validation.

This page is the step-by-step path for importing parcels when uploading a file directly through the admin (as opposed to using the Azure Blob URL method on Data Tools). It is also where you monitor validation results and apply batches after reviewing them.

Access: Available to all staff users with standard Django admin permissions on the ParcelImportBatch model.

List View

URL: /admin/appeals/parcelimportbatch/

Batches are ordered newest first (-uploaded_at).

Columns

Filters

• Tax Year — filter to a specific assessment year
• Status — filter by pipeline state
• Schema Profile — filter by which profile was used

Search

Searches by original filename and file hash (SHA256).

Bulk Actions

Batch Statuses

Adding a New Batch (Upload)

URL: /admin/appeals/parcelimportbatch/add/

This is how you upload a parcel CSV directly through the admin. The upload form has the following fields:

On save:

• The file's SHA256 hash is computed from the uploaded bytes.
• If a batch with the same (tax_year, file_hash_sha256) already exists, the form raises a validation error and the upload is rejected. The error message includes the ID of the existing duplicate batch.
• If accepted, the batch is saved with status UPLOADED and the file is stored under media/parcel_imports/<year>/.
• The uploaded_by field is set to the current user automatically.
• Raw rows are not created at this point — ingest runs when the first job (validate or apply) processes the batch.

After saving, proceed to the batch detail page to queue background jobs.

Batch Detail View

URL: /admin/appeals/parcelimportbatch/<id>/change/

Editable Fields (on existing batches)

Once a batch is saved, only schema_profile, source_name, and uploaded_file remain editable through the form. All status, count, and diagnostic fields are read-only.

Read-Only Fields

Background Actions (inline buttons)

Two buttons appear in the Background Actions read-only field:

• Queue Validation — queues a VALIDATE job for this batch. If a validate or apply job is already QUEUED or RUNNING for this batch, reports the existing job ID and does nothing.
• Queue Apply — queues an APPLY job for this batch. The apply worker will auto-validate if needed before applying.

Both buttons redirect back to the batch detail page after queuing.

Latest Job Progress

Displays the most recently created job for this batch:

<Action> — <Status> | <current>/<total> (<percent>%) | <message>

Examples:

• Validate — Running | 1500/8000 (18%) | Validating
• Apply — Success | 8000/8000 (100%) | Completed
• Apply — Failed | 450/8000 (5%) | Failed

This field does not auto-refresh — reload the page to see updated progress.

Top Errors (sample)

Aggregates error messages from the first 200 invalid ParcelRawRow records, counts occurrences of each distinct message, and shows the top 20 most frequent errors in descending order.

Format: <count> × <error message>

Example:

142 × Required field 'account' is missing or blank.
37 × Field 'res_assessed_value': cannot convert 'N/A' to decimal.
12 × Field 'sale_date_1': unrecognized date format '00/00/0000'.

Use this to quickly identify systematic problems — a transform misconfiguration, a missing column mapping, or a data quality issue in the source file.

Invalid Rows (sample)

Shows the first 20 invalid rows with their row number and error list:

Row 14: ["Field 'res_assessed_value': cannot convert 'N/A' to decimal."]
Row 23: ["Required field 'account' is missing or blank."]

For deeper inspection of individual rows (including the full raw and normalized JSON), navigate to Parcel Row Import Manager and filter by this batch.

Typical Workflow

1. Upload — Add a new batch via the Add page. Select tax year, schema profile, and file. Save.
2. Queue Validation — On the batch detail page, click Queue Validation. The parcel_import_worker will pick this up and run the validate phase.
3. Review — Reload the page to check Latest Job Progress, Row Count Valid / Invalid, Top Errors, and Invalid Rows (sample).
4. Decide — If invalid rows are acceptable (e.g., a known subset of unmappable records), proceed to apply. If errors indicate a schema or data problem, fix the profile or re-export the CSV and upload again.
5. Queue Apply — Click Queue Apply. The worker upserts all valid rows to Parcel records.
6. Confirm — Verify in Property Account Search that parcel records exist and look correct.

Deduplication

A batch is rejected at upload time if a ParcelImportBatch record already exists with the same (tax_year, file_hash_sha256). The error message identifies the conflicting batch by ID.

To import the same file again legitimately:

• Use the CLI with --force to bypass the hash check.
• Or modify the file (even a single whitespace change will produce a different hash).
• Or dump all parcel records first (if starting fresh).

Re-applying an already-applied batch is idempotent — the same valid rows simply update existing Parcel records in place.

Relationship to Other Pages

Key Code Locations

Login, User Management, and Permissions

Login Entry Points

Pie has two distinct login paths depending on the user's role.

Selector Page (/login/)

The root login selector at /login/ presents two buttons:

• Local Login → /accounts/login/?next=/portal/ — for portal (non-admin) users authenticating with a username and password.
• Admin Login → /admin/login/?next=/admin/ — for staff and superusers accessing the Django admin.

Already-authenticated users are automatically redirected to their appropriate destination (/admin/ or /portal/) without seeing the selector.

Local / Portal Login (/accounts/login/)

Used by portal-only users. Renders the allauth local login form with:

• Username and password fields
• Local User Sign In button
• Continue with Microsoft button (only shown if Microsoft SSO is configured)
• Forgot password link

After successful login, RoleAwareAccountAdapter.get_login_redirect_url() sends admin-capable users to /admin/ and all others to /portal/.

Admin Login (/admin/login/)

Used by staff and superusers. Renders the Django admin login form with:

• Username and password fields
• Remember Me checkbox
• Admin User Sign In button
• Continue with Microsoft button (only shown if Microsoft SSO is configured)

Remember Me behavior:

• Checked → session persists for SESSION_COOKIE_AGE (default Django value: 2 weeks)
• Unchecked → session expires when the browser is closed

If an authenticated user who does not meet admin access requirements hits this URL, they are redirected to /portal/.

Microsoft SSO (/accounts/microsoft/login/)

Both login forms show a Continue with Microsoft button when Microsoft SSO is configured. Clicking it initiates an OAuth2 flow via the allauth Microsoft social provider.

How it works:

1. The user is redirected to Microsoft's login page.
2. Microsoft authenticates the user and returns an email address.
3. Pie's MicrosoftSocialAccountAdapter intercepts the callback and checks:
   • The email returned by Microsoft is non-empty.
   • A Django user with that email already exists in Pie's database.
   • That user's account is active (is_active = True).
4. If all checks pass, the Microsoft account is linked to the existing Django user and the login proceeds.
5. If any check fails, the user is redirected back to /login/ with an error message.

Microsoft SSO does not create new users. The Django user account must exist first. Self-signup is disabled (is_open_for_signup returns False).

The next parameter controls post-login destination:

• Portal-bound: /accounts/microsoft/login/?process=login&next=%2Fportal%2F
• Admin-bound: /accounts/microsoft/login/?process=login&next=%2Fadmin%2F

Setting Up Microsoft SSO for a User

Step 1 — Verify environment variables are set

Microsoft SSO is only active when both of the following environment variables are set on the server (Azure Web App Application Settings):

The Continue with Microsoft button will not appear on login pages unless both MICROSOFT_CLIENT_ID and MICROSOFT_CLIENT_SECRET are set.

Step 2 — Create the Django user account first

A Pie user account must exist before Microsoft SSO can be used. Create the user in the Django admin:

1. Go to Django admin → Authentication and Authorization → Users → Add User.
2. Set a username (can be anything — the email address is recommended for clarity).
3. Set a temporary password (the user will never need to use it if they always log in via Microsoft).
4. Click Save and continue editing.
5. Fill in the user's Email address — this must exactly match the email in their Microsoft account (case-insensitive).
6. Set Active = checked.
7. Assign the appropriate staff status and group (see Roles and Permissions below).
8. Save.

Step 3 — Link the Microsoft account (automatic on first sign-in)

The user's Microsoft account is linked to their Pie account automatically on their first successful Microsoft SSO login. No manual linking step is required — MicrosoftSocialAccountAdapter.pre_social_login() calls sociallogin.connect(request, user) on every login, which creates or updates the SocialAccount link.

Step 4 — Verify the link (optional)

Linked social accounts can be inspected in the Django admin under Social Accounts → Social accounts. Each linked account shows the provider (microsoft), the associated Django user, and the last login timestamp.

User Roles and Permissions

Pie uses three distinct access tiers, controlled by Django's built-in is_superuser, is_staff, and groups flags.

Superuser

• is_superuser = True
• Full, unrestricted access to all of Django admin — all models, all actions.
• Bypasses all group checks.
• Always passes user_can_access_admin().
• Can access all admin sections hidden from regular staff (e.g., Parcel Column Import Manager, Parcel Row Import Manager, certain raw import pipeline models).

When to use: System administrators and developers only.

Staff (Admin Portal User)

• is_staff = True AND member of the AdminPortal group
• Can access Django admin (/admin/).
• Subject to Django's standard object-level permissions — only sees and can edit models their permissions allow.
• Cannot access superuser-only sections.

When to use: Board of equalization staff who use the admin to manage appeals, hearings, meetings, and documents.

Access rule: Both conditions must be true. is_staff = True alone is not sufficient — the user must also be in the AdminPortal group. The group name is configured via the ADMIN_PORTAL_GROUP_NAME environment variable (defaults to AdminPortal).

Portal User

• is_staff = False, is_superuser = False
• No access to Django admin.
• Can log in at /accounts/login/ and access /portal/ only.
• Sees appeal status and related information for their own appeals.

When to use: Property owners or representatives who filed an appeal and need read-only portal access.

Creating and Managing Users

Create a new user

1. Django admin → Authentication and Authorization → Users → Add User.
2. Enter username and password. Click Save and continue editing.
3. Fill in:
   • Email — required for Microsoft SSO; must match their Microsoft account email exactly.
   • First name / Last name — optional but recommended.
   • Active — must be checked for the user to be able to log in.
   • Staff status — check this for admin users.
   • Superuser status — check only for system administrators.
4. Under Groups, add the user to AdminPortal for admin access.
5. Under User permissions, add individual Django model permissions if needed (usually handled through groups instead).
6. Save.

Deactivate a user

Set Active = unchecked. The user cannot log in via password or Microsoft SSO. Their data and audit history are preserved.

Reset a password

In the user's admin edit page, use the Change password link. The self-service password reset at /accounts/password/reset/ is currently out of service — staff must reset passwords manually through the admin.

Delete a user

Deleting a user is permanent and will break any foreign key references to that user. Deactivating is preferred.

Managing Groups and Permissions

Groups in Django are collections of permissions assigned to users collectively. In Pie, the AdminPortal group is the primary mechanism for granting admin access to staff users.

Create or edit the AdminPortal group

1. Django admin → Authentication and Authorization → Groups.
2. Click AdminPortal (or Add Group if it does not exist yet).
3. Set the group Name to AdminPortal (must match the ADMIN_PORTAL_GROUP_NAME setting exactly).
4. Under Permissions, add the Django model permissions the group's members should have (e.g., appeals | appeal | Can add appeal, appeals | appeal | Can change appeal, etc.).
5. Save.

Add a user to the AdminPortal group

On the user's edit page, scroll to Groups and move AdminPortal to the Chosen groups list. Save.

Or from the Group page, users can be viewed but not assigned directly — use the User edit page for group assignment.

Additional groups

You can create additional groups for more granular permission segmentation (e.g., a read-only staff group). Assign those groups to users in the same way. Pie does not enforce any specific secondary group names beyond AdminPortal for admin gate access.

Access Policy Summary

The admin access check is enforced by user_can_access_admin() in appeals/auth_roles.py, which patches admin.site.has_permission globally via enforce_admin_site_access_policy().

Audit Logging

All login activity is recorded automatically:

• Successful login — logged by audit_user_login() in appeals/signals.py. Records user ID, username, email, IP address, path, and group membership at login time.
• Failed login attempt — logged by audit_user_login_failed(). Records the attempted username and IP. Passwords are never captured.
• Logout — logged by audit_user_logout().

Audit records are viewable in Django admin → Audit Logs, or exportable via Data Tools → Audit Logs Export.

Key Code Locations