catalog
catalog
PDS Catalog: index of all instruments and products from the PDS archive.
Built from the MillionConcepts pdr-tests repository, this module provides a local DuckDB database cataloging all known PDS instruments and their data products, following the mission/instrument/product access scheme.
Usage: from planetarypy.catalog import build_catalog, list_missions
# Build the catalog (clones pdr-tests repo, parses definitions, populates DB)
build_catalog()
# Query the catalog
list_missions()
list_instruments("cassini")
list_products("cassini", "iss")
example_products("cassini", "iss", "edr_sat")
# Review ambiguous mission mappings
ambiguous_mappings()Classes
| Name | Description |
|---|---|
| BatchFetchResult | Outcome of a single product in a :func:fetch_products batch. |
| OfflineError | Raised by :func:fetch_products when the preflight internet check fails. |
BatchFetchResult
catalog.BatchFetchResult(product_id, downloaded, exception)Outcome of a single product in a :func:fetch_products batch.
Attributes
| Name | Type | Description |
|---|---|---|
| product_id | str | The PID that was requested (NOT the canonical PID the resolver matched, which lives at downloaded.product_id on success). |
| downloaded | DownloadedProduct or None | Populated on success; None if this PID’s fetch raised. |
| exception | Exception or None | Captured exception on failure; None on success. |
OfflineError
catalog.OfflineError()Raised by :func:fetch_products when the preflight internet check fails.
The batch fetcher refuses to launch a parallel pool against a server it can’t reach — every worker would just error individually, polluting the report with redundant connection-failure noise. Pass skip_online_check=True to override (useful for offline mirrors or when have_internet() itself is the unreliable component).
Functions
| Name | Description |
|---|---|
| ambiguous_mappings | Return all instruments with ambiguous mission/instrument mapping. |
| build_catalog | Build (or rebuild) the PDS catalog database from pdr-tests. |
| example_products | Get all product entries for a given mission/instrument/product type. |
| fetch_product | Download a PDS product and return where it landed plus what was written. |
| fetch_products | Download a batch of PDS products in parallel; continues past failures. |
| get_catalog | Open and return a connection to the existing catalog database. |
| get_product_urls | Return the files and their URLs for a product. |
| list_instruments | List all instruments for a given mission. |
| list_missions | List all missions in the catalog. |
| list_products | List all product types for a given mission and instrument. |
| search | Search the catalog for products matching a query string. |
| summary | Get a summary of the catalog contents grouped by mission. |
ambiguous_mappings
catalog.ambiguous_mappings()Return all instruments with ambiguous mission/instrument mapping.
These need manual review by the user.
build_catalog
catalog.build_catalog(force=False)Build (or rebuild) the PDS catalog database from pdr-tests.
Clones the pdr-tests repository, parses all selection_rules.py and CSV test files, and populates a DuckDB database.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| force | bool | If True, force re-clone of pdr-tests and rebuild DB from scratch | False |
Returns
| Name | Type | Description |
|---|---|---|
| dict | Summary with counts of missions, instruments, product types, products |
example_products
catalog.example_products(key, instrument=None, product_key=None, *, phase=None)Get all product entries for a given mission/instrument/product type.
Accepts either dotted key or separate arguments: example_products(“cassini.iss.edr”) # all EDR phases example_products(“cassini.iss.edr”, phase=“saturn”) # Saturn only example_products(“cassini”, “iss”, “edr”)
The product type is matched against the normalized_type column first, then falls back to exact product_key match for backward compatibility.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| key | str | Either a dotted key ‘mission.instrument.product’ or just the mission name | required |
| instrument | str | Instrument name, required if key is not a dotted key | None |
| product_key | str | Product type key, required if key is not a dotted key | None |
| phase | str | Filter by mission phase (e.g. ‘saturn’, ‘jupiter’, ‘cruise’). Only used when matching by normalized_type. | None |
Returns
| Name | Type | Description |
|---|---|---|
| pd.DataFrame | Product entries with columns: product_id, label_file, url_stem, etc. |
fetch_product
catalog.fetch_product(
key,
product_id,
*,
instrument=None,
product_key=None,
files=None,
label_only=False,
force=False,
local_dir=None,
open=False,
)Download a PDS product and return where it landed plus what was written.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| key | str | Either a dotted key 'mission.instrument.product_type' or just the mission name (in which case instrument and product_key are required). |
required |
| product_id | str | Product identifier, e.g. 'P02_001916_2221_XI_42N027W'. Accepts the bare-PID form returned by :func:planetarypy.pds.get_example_pid; PDS path/extension and flight-software version suffixes are normalized away during matching. |
required |
| instrument | str | Instrument name. Required when key is just a mission. |
None |
| product_key | str | Product type key. Required when key is just a mission. |
None |
| files | list[str] | None | Specific filenames to download. None (default) downloads every file the resolver returns for this product. |
None |
| label_only | bool | If True, download only the PDS label file. Mutually exclusive with a populated files argument. |
False |
| force | bool | If True, re-download even if files already exist locally. |
False |
| local_dir | Path | Override the storage location. When None (default), the catalog’s per-instrument layout is used ({storage_root}/{mission}/{instrument}/{product_type}/{pid}/). |
None |
| open | bool | If True, open the downloaded product in memory and return that object (as :func:planetarypy.open) instead of the DownloadedProduct. Convenience for download-and-open in one call. |
False |
Returns
| Name | Type | Description |
|---|---|---|
| DownloadedProduct | Bundle containing local_dir (folder), files (absolute paths of every file written by this call), label_file (convenience pointer to the PDS label, if any), and product_id (the canonical identifier the resolver matched). When open=True, the opened in-memory object is returned instead. |
Examples
>>> from planetarypy.catalog import fetch_product
>>> r = fetch_product("mro.ctx.edr", "P02_001916_2221_XI_42N027W")
>>> r.local_dir
PosixPath('.../mro/ctx/edr/P02_001916_2221_XI_42N027W')
>>> r.files
[PosixPath('.../P02_001916_2221_XI_42N027W.IMG'),
PosixPath('.../P02_001916_2221_XI_42N027W.LBL')]
>>> r.label_file.name
'P02_001916_2221_XI_42N027W.LBL'fetch_products
catalog.fetch_products(
key,
product_ids,
*,
workers=4,
instrument=None,
product_key=None,
files=None,
label_only=False,
force=False,
local_dir=None,
skip_online_check=False,
)Download a batch of PDS products in parallel; continues past failures.
Thin wrapper over :func:fetch_product using :func:planetarypy.utils.parallel_map. Per-PID exceptions are captured into the returned :class:BatchFetchResult so a single bad PID never breaks the rest of the batch.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| key | str | Dotted key 'mission.instrument.product_type', or just the mission name (in which case instrument and product_key are required). Passed through verbatim to fetch_product per PID. |
required |
| product_ids | Iterable[str] | PIDs to download. Order is preserved in the returned list. | required |
| workers | int | Thread-pool size. Safe default for typical PDS servers; raise for bulk downloads against tolerant servers. | 4 |
| instrument | str | None | Forwarded to :func:fetch_product unchanged. local_dir is shared across all PIDs in the batch — pass None (default) to get the catalog’s per-PID layout. |
None |
| product_key | str | None | Forwarded to :func:fetch_product unchanged. local_dir is shared across all PIDs in the batch — pass None (default) to get the catalog’s per-PID layout. |
None |
| files | str | None | Forwarded to :func:fetch_product unchanged. local_dir is shared across all PIDs in the batch — pass None (default) to get the catalog’s per-PID layout. |
None |
| label_only | str | None | Forwarded to :func:fetch_product unchanged. local_dir is shared across all PIDs in the batch — pass None (default) to get the catalog’s per-PID layout. |
None |
| force | str | None | Forwarded to :func:fetch_product unchanged. local_dir is shared across all PIDs in the batch — pass None (default) to get the catalog’s per-PID layout. |
None |
| local_dir | str | None | Forwarded to :func:fetch_product unchanged. local_dir is shared across all PIDs in the batch — pass None (default) to get the catalog’s per-PID layout. |
None |
| skip_online_check | bool | When False (default), :func:planetarypy.utils.have_internet is called before launching the pool; if it returns False, :class:OfflineError is raised. Pass True to bypass the preflight (offline mirrors, captive networks, etc.). |
False |
Returns
| Name | Type | Description |
|---|---|---|
| list[BatchFetchResult] | One result per input PID, in input order. result.ok is the quickest success check; result.downloaded holds the same :class:DownloadedProduct you’d get from a single fetch_product call on success. |
Raises
| Name | Type | Description |
|---|---|---|
| OfflineError | If the preflight internet check fails and skip_online_check=False. |
Examples
>>> from planetarypy.catalog import fetch_products
>>> results = fetch_products(
... "mro.ctx.edr",
... ["P02_001916_2221_XI_42N027W", "P03_001234_2222_XI_43N028W"],
... workers=4,
... )
>>> [r.ok for r in results]
[True, True]
>>> [r.downloaded.product_id for r in results if r.ok]
['P02_001916_2221_XI_42N027W', 'P03_001234_2222_XI_43N028W']get_catalog
catalog.get_catalog()Open and return a connection to the existing catalog database.
Raises
| Name | Type | Description |
|---|---|---|
| FileNotFoundError | If the catalog has not been built yet. | |
| RuntimeError | If the catalog DB exists but its schema predates the current code (e.g. built with planetarypy ≤ 0.52). |
get_product_urls
catalog.get_product_urls(key, product_id, *, instrument=None, product_key=None)Return the files and their URLs for a product.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| key | str | Dotted key ‘mission.instrument.product_type’ or just mission | required |
| product_id | str | Product identifier | required |
Returns
| Name | Type | Description |
|---|---|---|
| dict[str, str] | Mapping of filename -> full URL |
list_instruments
catalog.list_instruments(mission, *, include_misc=False)List all instruments for a given mission.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| mission | str | Mission name (e.g. ‘cassini’) | required |
| include_misc | bool | If True, include the ’_misc’ catch-all instrument that holds unclassified product types. Default False. | False |
list_missions
catalog.list_missions()List all missions in the catalog.
list_products
catalog.list_products(key, instrument=None, *, include_phases=False)List all product types for a given mission and instrument.
By default, returns normalized product type names (e.g. ‘edr’ instead of ‘edr_sat’). Use include_phases=True to see the phase breakdown.
Accepts either dotted key or separate arguments: list_products(“cassini.iss”) list_products(“cassini”, “iss”) list_products(“cassini.iss”, include_phases=True)
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| key | str | Either a dotted key ‘mission.instrument’ or just the mission name | required |
| instrument | str | Instrument name, required if key is not a dotted key | None |
| include_phases | bool | If True, return a DataFrame with normalized_type, phase, format, product_key and source columns. The source column carries the pdr-tests definition folder (e.g. dawn__vir vs dawn_certified__vir) and explains why an instrument can have multiple rows that share the other four — they’re parallel archive provenances for the same logical product. If False (default), return a deduplicated list of normalized type names. |
False |
search
catalog.search(query)Search the catalog for products matching a query string.
Searches across mission, instrument, normalized_type, product_key, and product_id. Searching for ‘edr’ will find both ‘edr_sat’ and ‘edr_evj’ variants.
summary
catalog.summary()Get a summary of the catalog contents grouped by mission.