Planning a Whole-State Property Export on a Daily Allowance

Flat-rate plans meter exports by a daily allowance rather than by the record, so a whole state is a question of scheduling, not of price. How long a state takes depends on two numbers, its record total and your plan’s daily allowance, and for most states and plans it is more than one day. The work goes better with a plan: which areas to export on which day, how to handle counties bigger than a day’s allowance, and how to pick up where you left off.

This guide works through that plan. The plan figures below are computed from the current pricing, and the state and county figures from the live record totals, so they stay correct as either changes.

How the daily allowance works

• Each paid plan allows a number of records per day. The day is a UTC day, and the allowance refills at midnight UTC. Allowance you don’t use does not roll over.
• An export is one area: a whole state, a county, a city or a ZIP code, optionally limited to one list such as absentee owners.
• You choose how many records to take, up to what is left today. A request larger than the remaining allowance is capped at what remains.
• One build runs at a time. Queue the next area once the current one is ready.
• Large exports arrive in parts of up to 50,000 records, each a complete CSV with a header row.

Over a week or a month, the current plans add up to:

The arithmetic: days per state

The minimum number of days for a state is its record total divided by your daily allowance, rounded up. That is a lower bound. You reach it only if every day’s allowance is used in full, which depends on how well the areas you export pack into each day.

Each state page, for example California, repeats this calculation for that state and lists its largest counties.

Why you export a state area by area

Unless a state’s total fits inside what is left of today’s allowance, requesting the whole state at once returns only part of it. Requesting the state again the next day does not continue where the first export stopped: an export takes up to the requested number of records from the area, with no guarantee of which ones or in what order. Two partial exports of the same area can overlap and can both miss the same records.

The reliable approach is to export areas that fit completely within what remains of the day, and to combine them afterwards. Counties are the natural unit: every property belongs to exactly one county, so a set of complete county exports adds up to the whole state with no overlap.

Counties bigger than a day

The largest counties can hold more records than the smaller plans allow in a day. Those counties need to be split one level further down, into cities or ZIP codes, until each piece fits:

In the largest state file, Texas, the 5 biggest counties compare with each plan’s daily allowance like this. A county that fits shows the number of days it uses on its own; one that does not fit in a single request needs splitting into cities or ZIP codes.

Cities or ZIPs?

• Cities are fewer and larger, so there is less to manage. A city area follows the city name in the property address, which does not always line up with county boundaries, so check that the city totals add up to the county total and fill any gap with ZIPs.
• ZIP codes cover everything and are small enough to pack tightly into days. They also cross county lines, so a ZIP export can include parcels from a neighbouring county. That is harmless as long as you deduplicate when you combine files, as shown below.

Either way, the export builder shows each area’s property count when you search for it. Note the counts down as you plan; they are the input to the schedule in the next step.

Step 1: Build a manifest

Before exporting anything, list every area you will need in a small CSV: the area, its record count, and a status column. The example rows below show the format; the counts come from the builder.

area_kind,area_key,label,records,status,exported_on
county,48201,Harris County,,todo,
county,48113,Dallas County,,todo,
county,48029,Bexar County,,todo,

Leave the records column blank until you have the count for each area, and replace any county too large for your plan with its cities or ZIP codes. The sum of the records column should be at least the state total from the data by state page; more is fine if ZIP areas overlap.

Step 2: Pack areas into days

With counts in hand, assign areas to days so that each day’s exports fit within the allowance and as little as possible is left unused. This is a bin-packing problem. The first-fit decreasing heuristic (largest areas first, each into the first day with room) is simple and in practice gets close to the lower bound:

"""Pack a state's counties into export days for one daily allowance.

Input: areas.csv with columns area_kind, area_key, label, records
       (records = the area's property count, as shown in the export builder).
Output: schedule.csv, one row per area with the day it is planned for.
"""
import csv
import math
import sys

ALLOWANCE = int(sys.argv[1])          # e.g. 15000 for a 15,000-a-day plan

with open("areas.csv", newline="") as f:
    areas = [dict(r, records=int(r["records"])) for r in csv.DictReader(f)]

too_big = [a for a in areas if a["records"] > ALLOWANCE]
fits = sorted((a for a in areas if a["records"] <= ALLOWANCE),
              key=lambda a: a["records"], reverse=True)

# First-fit decreasing: put each area on the first day with room left.
days: list[list[dict]] = []
room: list[int] = []
for a in fits:
    for i, left in enumerate(room):
        if a["records"] <= left:
            days[i].append(a)
            room[i] -= a["records"]
            break
    else:
        days.append([a])
        room.append(ALLOWANCE - a["records"])

with open("schedule.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["day", "area_kind", "area_key", "label", "records", "status"])
    for d, group in enumerate(days, start=1):
        for a in group:
            w.writerow([d, a["area_kind"], a["area_key"], a["label"], a["records"], "todo"])

total = sum(a["records"] for a in areas)
print(f"{len(days)} days for {len(fits)} areas that fit")
print(f"lower bound for the whole state: {math.ceil(total / ALLOWANCE)} days")
for a in too_big:
    print(f"split needed: {a['label']} ({a['records']:,} records) -> cities or ZIPs")

Run it with your plan’s daily allowance, for example python pack_days.py 15000. The script prints the number of days its schedule needs next to the theoretical minimum, and lists any area that still needs splitting. If the gap between the two is large, splitting the biggest remaining counties into ZIPs usually closes it, because smaller pieces pack more tightly.

Step 3: Export day by day

1. Open schedule.csv and take today’s areas, largest first.
2. For each area, request exactly its record count. Wait for the build to finish before queuing the next.
3. Download every part into a folder named after the area, for example exports/48201/.
4. Verify the row count, then set the area’s status to done and note the date.

# After downloading an area's parts into exports/<area_key>/, check the row
# count against the schedule before marking the area done.
area=48201
expected=$(awk -F, -v k="$area" '$3 == k { print $5 }' schedule.csv)
actual=0
for f in exports/$area/*.csv; do
  n=$(($(wc -l < "$f") - 1))       # minus the header line
  actual=$((actual + n))
done
echo "$area expected=$expected actual=$actual"

If the count is short, the most likely cause is that the area did not fit in what was left of the day and the export was capped. Do not top it up with a second partial export. Mark the area redo and export it in full on a later day.

Step 4: Resuming after a gap

Because each area is either complete or not, resuming is simple: the manifest is the state of the job. After a missed day, a week away, or a plan change, start again from the first area whose status is not done. Nothing already downloaded needs to be repeated.

Two situations change the plan:

• You change plans. Re-run the packing script with the new allowance on the remaining areas only. A larger allowance may also let you merge ZIP-level pieces back into whole counties.
• The monthly data refresh lands mid-job. Records are refreshed monthly, so a long export can span two editions. Record the export date for every area. For most uses a mix of adjacent months is fine. If you need a single consistent edition, re-export the areas done before the refresh, smallest first, and plan long jobs to start soon after a refresh.

Step 5: Combine and deduplicate

When every area is done, combine the parts into one file. Deduplicate on Property ID, the column that identifies each property, which removes the overlaps from ZIP and city areas. DuckDB does this in a single statement without loading a database:

-- DuckDB: combine every downloaded part into one statewide file, keeping one
-- row per property. ZIP and city areas can cross county lines, so overlaps
-- between areas are expected and removed here. Headers contain spaces, so
-- they are quoted.
COPY (
  SELECT * EXCLUDE (filename)
  FROM read_csv('exports/*/*.csv', header = true, all_varchar = true, filename = true)
  QUALIFY row_number() OVER (PARTITION BY "Property ID" ORDER BY filename) = 1
) TO 'state_full.csv' (HEADER);

-- Records per county in the result, to compare with your schedule.
SELECT "County", count(*) AS n
FROM read_csv('state_full.csv', header = true, all_varchar = true)
GROUP BY 1 ORDER BY 1;

Compare the per-county counts with your manifest. The file names each county in its County column rather than by code, so match them to the label in your manifest. A county that comes out noticeably short points to an area that was missed or capped. From here, the PostgreSQL loading guide covers putting the combined file into a database, and the DuckDB guide covers filtering it in place.

Do you need the whole state?

A full-state file is the right choice for market-wide analysis, building your own database, or matching against a large internal dataset. For outreach it often is not. Exports can be limited to one list, such as absentee owners, free-and-clear or high equity properties, so a list export covers far fewer records than the whole area and fits in fewer days. Check the area’s count for that list in the export builder before you plan around it.

If the number of days on your plan is the obstacle, compare plans on the pricing section; every plan includes every column, so the only difference is how fast the file arrives. The flat-rate vs per-record guide works through what a statewide build costs under each model.