Per-Tile Quality (Uncertainty) API

Separating marker scatter from vote support in the Planet Four catalog

Every marking in the v3.1 catalog is a cluster of citizen-science markings averaged into one object. The catalog records, for each, the standard deviations of the markings that were averaged — x_std, y_std, angle_std, plus distance_std/spread_std (fans) or radius1_std/radius2_std (blotches) — together with the vote count n_votes. These are the raw material for a per-tile quality measure.

A raw standard deviation conflates two different causes of unreliability: markers genuinely disagreeing (scattered votes) and weak support (few votes). This API keeps the two on separate, never-mixed axes. Vote counts never divide the scatter numbers. In particular, a standard error of the mean — SEM, the standard deviation divided by √n — shrinks as votes accumulate, so using it would quietly fold vote support back into the scatter axis and re-mix the two things we are trying to keep apart. Reporting raw scatter instead keeps contested tiles (many votes, real disagreement) visible rather than averaging them away.

import numpy as np, pandas as pd
from p4tools import io, stats
pd.set_option('display.max_columns', None)

Layer 1 — per-marking enrichment

add_uncertainty_columns infers the kind (fan vs blotch) from the columns and adds four derived columns:

column	unit	meaning
`pos_std`	px	`hypot(x_std, y_std)` — positional scatter of the cluster
`size_cv`	—	mean relative std (a coefficient of variation): fans use `distance`/`spread`, blotches `radius_1`/`radius_2`
`angle_usable`	bool	always True for fans; False for near-circular blotches (`radius_2/radius_1 > 0.8`), where orientation is ill-defined
`scatter_ok`	bool	`n_votes >= min_votes` — a std from a handful of votes is itself noise

fu = stats.add_uncertainty_columns(io.get_fan_catalog("v3.1"))
fu[["tile_id","n_votes","pos_std","size_cv","angle_usable","scatter_ok"]].head()

	tile_id	n_votes	pos_std	size_cv	angle_usable	scatter_ok
0	APF00006nk	11.0	5.519993	0.249842	True	True
1	APF00006nk	10.0	4.979427	0.305487	True	True
2	APF00006nk	10.0	3.649387	0.337209	True	True
3	APF00006nk	8.0	4.086334	0.226463	True	True
4	APF00006nk	8.0	5.117829	0.247728	True	True

For blotches, angle_usable drops the near-circular ones whose fitted orientation is meaningless:

bu = stats.add_uncertainty_columns(io.get_blotch_catalog("v3.1"))
bu.angle_usable.value_counts()

angle_usable
True     240905
False    186499
Name: count, dtype: int64

Layer 2 — per-tile aggregation

tile_quality returns one row per tile_id, with columns organised into named groups:

support — how much evidence backs this tile: n_markings, n_fans, n_blotches, votes_median, votes_min, votes_total. Pure vote axis.
scatter — how much the markers disagreed: angle_scatter, pos_scatter, size_scatter, computed only from markings that pass the vote gate (scatter_ok, i.e. n_votes >= min_votes), so weak support can never leak in as apparent disagreement. n_scatter_markings reports how many survived; scatter is NaN if none did.
ranks — each raw metric as its cap-wide percentile position (0–100; 50 = median tile). scatter_rank is the median of the three scatter metrics’ individual ranks, and support_rank the same on the vote axis — so incommensurate units (deg, px, —) become comparable and combinable.

Why ranks, concretely. A tile with angle_scatter 15° (p80), pos_scatter 3 px (p40), size_scatter 0.10 (p55) gets scatter_rank = median(80, 40, 55) = 55 — slightly worse than the median tile. Another at (p20, p95, p92) gets median = 92 — very scattered, driven by position and size while its angles agree. Raw units could not be combined this way ((15 + 3 + 0.10)/3 is meaningless).

tq = stats.tile_quality("both", version="v3.1")
tq.head()

	n_markings	n_fans	n_blotches	votes_median	votes_min	votes_total	angle_scatter	pos_scatter	size_scatter	n_scatter_markings	scatter_rank	support_rank
tile_id
APF0000001	4	3	1	17.0	7.0	60.0	5.329304	4.432020	0.306949	4	30.809548	57.756535
APF0000002	2	0	2	27.0	11.0	54.0	23.466506	9.144525	0.406551	2	79.314403	54.660899
APF0000004	4	0	4	10.5	5.0	42.0	19.672716	5.126946	0.221274	4	22.280351	47.259435
APF0000005	5	0	5	33.0	7.0	143.0	26.156416	6.272975	0.551231	5	83.691334	81.310665
APF0000006	11	4	7	21.0	16.0	246.0	22.180734	5.149757	0.345502	11	60.195479	91.661240

Tiles where no marking clears the vote gate have NaN scatter (and scatter_rank), reported honestly rather than imputed:

tq[["n_markings","n_scatter_markings","scatter_rank","support_rank"]].describe().round(2)

	n_markings	n_scatter_markings	scatter_rank	support_rank
count	64494.00	64494.00	56579.00	64494.00
mean	10.74	7.27	48.93	50.35
std	17.12	12.76	22.00	27.61
min	1.00	0.00	0.10	6.17
25%	2.00	1.00	32.53	26.25
50%	6.00	3.00	49.12	50.02
75%	13.00	9.00	65.19	74.31
max	280.00	211.00	99.77	99.98

Layer 3 — opt-in classification

This layer is opt-in: tile_quality already gives you the full, continuous picture (the support_rank/scatter_rank axes), and you only call classify_tile_quality if you additionally want discrete labels. It is a separate step because collapsing two continuous ranks into four buckets discards information and bakes in a threshold choice (where “high” starts, default the 50th percentile). Many analyses are better served by the raw ranks — e.g. weighting by scatter_rank — so the labels are offered, never imposed.

classify_tile_quality splits the two rank axes at a percentile threshold (default 50) into four classes:

`quality_class`	support	scatter	reading
`consistent`	high	low	reliable — best QC tier
`contested`	high	high	many votes, genuine disagreement — scientifically interesting; exactly the class a single conflated score would hide
`sparse`	low	low	few votes, but they agree
`noisy`	low	high	least reliable

Tiles with no scatter estimate (scatter_rank NaN) are left <NA> rather than guessed.

cls = stats.classify_tile_quality(tq)
cls.quality_class.value_counts(dropna=False)

quality_class
consistent    17194
contested     15294
noisy         12192
sparse        11899
<NA>           7915
Name: count, dtype: int64

The contested tiles — high vote support and high disagreement — are the ones worth a human look:

cls[cls.quality_class == "contested"].sort_values("support_rank", ascending=False).head()

	n_markings	n_fans	n_blotches	votes_median	votes_min	votes_total	angle_scatter	pos_scatter	size_scatter	n_scatter_markings	scatter_rank	support_rank	quality_class
tile_id
APF00007kk	179	0	179	9.0	3.0	1624.0	24.614313	5.460791	0.325892	150	53.760229	99.827116	contested
APF00007to	160	0	160	5.0	3.0	1119.0	24.034693	5.931912	0.349409	92	61.369059	99.627872	contested
APF00007t3	148	0	148	7.0	3.0	1090.0	33.661004	5.174341	0.331405	121	55.626646	99.568952	contested
APF00007mk	150	0	150	7.0	3.0	1058.0	32.251943	4.964730	0.317002	104	50.826278	99.567402	contested
APF00007tz	135	0	135	9.0	4.0	1347.0	29.321577	4.536079	0.322249	124	52.574277	99.495302	contested

Using it downstream

Because the output is tile_id-indexed, it joins straight onto any tile-level analysis — e.g. to weight or filter coverage numbers by tile reliability:

reliable = cls.index[cls.quality_class == "consistent"]
print(f"{len(reliable):,} consistent tiles of {len(cls):,} "
      f"({100*len(reliable)/len(cls):.1f}%)")
# cov = io.get_tile_coords().merge(cls, on="tile_id")  # then subset / weight

17,194 consistent tiles of 64,494 (26.7%)