canvod-virtualiconvname¶

Purpose¶

The canvod-virtualiconvname package maps arbitrary GNSS observation filenames to a canonical naming convention. Physical files on disk keep their original names -- the package creates a virtual mapping layer that gives every file a unique, self-describing canonical name.

The CanVODFilename Convention¶

Every canonical filename follows this format:

{SIT}{T}{NN}{AGC}_R_{YYYY}{DOY}{HHMM}_{PERIOD}_{SAMPLING}_{CONTENT}.{TYPE}[.{COMPRESSION}]

Fields¶

Duration codes¶

Example¶

ROSR01TUW_R_20250010000_01D_05S_AA.rnx

VirtualFile¶

A VirtualFile pairs a physical file path with its canonical name:

from canvod.virtualiconvname import VirtualFile

vf.physical_path    # Path("/data/rref001a00.25_")
vf.canonical_str    # "ROSR01TUW_R_20250010000_15M_05S_AA.sbf"
vf.open("rb")       # opens the physical file

The physical file is never renamed. All downstream processing uses the canonical name for metadata, deduplication, and storage keys.

NamingRecipe¶

A NamingRecipe tells the system how to parse an arbitrary physical filename into a canonical name. Recipes are defined in YAML and referenced from sites.yaml.

How it works¶

flowchart TD
    PHYS["`**Physical file**
    rref001a00.25_`"]
    PHYS --> RECIPE["`**NamingRecipe**
    field extraction`"]
    RECIPE --> VF["VirtualFile"]
    VF --> CANON["`**Canonical name**
    ROSR01TUW_R_20250010000_15M_05S_AA.sbf`"]

The recipe defines:

1. Identity fields -- site, agency, receiver number/type (constant for a receiver)
2. Discovery -- glob pattern and directory layout to find files
3. Field extraction -- a sequence of {field: width} entries that parse the physical filename left-to-right

YAML example¶

name: rosalia_reference
description: Septentrio RINEX v2 files from Rosalia reference receiver
site: ROS
agency: TUW
receiver_number: 1
receiver_type: reference
sampling: "05S"
period: "15M"
file_type: rnx
layout: yyddd_subdirs
glob: "*.??o"
fields:
  - skip: 4          # "rref"
  - doy: 3           # "001"
  - hour_letter: 1   # "a"
  - minute: 2        # "15"
  - skip: 1          # "."
  - yy: 2            # "25"
  - skip: 1          # "o"

Recognized fields¶

Using recipes¶

Reference a recipe file from sites.yaml:

sites:
  rosalia:
    receivers:
      reference_01:
        recipe: rosalia_reference.yaml

When just config-init copies configuration templates, recipe files are included.

FilenameMapper¶

The FilenameMapper discovers physical files and maps them to VirtualFiles. It handles three directory layouts, configured via directory_layout in the receiver config or recipe.

Directory layouts¶

Most GNSS receivers output files into per-day subdirectories named by day-of-year. The directory_layout setting tells the mapper where to look for files.

How discovery differs¶

The layout controls where the mapper searches — it does not affect how filenames are parsed (that is determined by the source pattern or recipe).

receiver_base_dir/
├── 25001/
│   ├── rref001a00.25_     ← discovered
│   └── rref001a15.25_     ← discovered
├── 25002/
│   └── rref002a00.25_     ← discovered
└── rref003a00.25_         ← NOT discovered (at root level)

receiver_base_dir/
├── 2025001/
│   └── rref001a00.25_     ← discovered
└── 2025002/
    └── rref002a00.25_     ← discovered

receiver_base_dir/
├── rref001a00.25_         ← discovered
├── rref002a00.25_         ← discovered
└── notes.txt              ← ignored (not a GNSS file)

Configuration¶

In sites.yaml (legacy naming config):

receivers:
  reference_01:
    receiver_number: 1
    source_pattern: auto
    directory_layout: yyddd_subdirs   # or flat, yyyyddd_subdirs

In a NamingRecipe:

layout: yyddd_subdirs   # default if omitted

Usage¶

from canvod.virtualiconvname import FilenameMapper

mapper = FilenameMapper(
    site_naming=site_config,
    receiver_naming=receiver_config,
    receiver_type="reference",
    receiver_base_dir=Path("/data/rosalia/reference"),
)

# Discover and map all files
virtual_files = mapper.discover_all()

# Or for a specific date
virtual_files = mapper.discover_for_date(year=2025, doy=1)

Built-in patterns¶

The BUILTIN_PATTERNS registry handles common GNSS filename formats automatically:

When source_pattern: auto (the default), patterns are tried in order until one matches. Use a NamingRecipe for formats not covered by built-in patterns.

DataDirectoryValidator¶

The DataDirectoryValidator is a pre-pipeline hard gate. Before any processing begins, it checks that:

1. All files can be mapped -- every file in the receiver directory matches a naming pattern or recipe
2. No temporal overlaps -- no two files cover the same time window

If validation fails, the pipeline is blocked with a diagnostic message listing the unmatched files and/or overlapping pairs.

from canvod.virtualiconvname import DataDirectoryValidator

report = DataDirectoryValidator.validate_receiver(
    site_naming=site_config,
    receiver_naming=receiver_config,
    receiver_type="reference",
    receiver_base_dir=Path("/data/rosalia/reference"),
    reader_format="rinex3",  # optional filter
)

report.is_valid     # True if no unmatched or overlaps
report.matched      # list[VirtualFile]
report.unmatched    # list[Path]
report.overlaps     # list[tuple[VirtualFile, VirtualFile]]

FilenameCatalog¶

The FilenameCatalog persists file mappings in a local DuckDB database, enabling fast lookups without re-scanning directories.

from canvod.virtualiconvname import FilenameCatalog

with FilenameCatalog(db_path) as catalog:
    catalog.record_batch(virtual_files)

    # Lookup by canonical name
    path = catalog.lookup_by_conventional("ROSR01TUW_R_20250010000_01D_05S_AA.rnx")

    # Query a date range
    files = catalog.query_date_range(2025, 1, 2025, 31, receiver_type="R")

    # Export to Polars DataFrame
    df = catalog.to_polars()

The catalog stores file hashes (SHA-256 of first 64 KiB), sizes, and modification times alongside the canonical mapping.