Sky science
Light pollution data
How a satellite image of the Earth at night becomes a zone for every 0.7 km² cell, and how the app looks it up.
The pipeline
A satellite looks down and measures light that escapes upwards. A person on the ground looks up and sees light scattered through the air from cities tens of kilometres away. The pipeline turns the first into an estimate of the second, one cell at a time.
- Read
The annual night-lights image is read in strips of 200 rows. Each pixel above a small floor becomes an H3 cell, keeping the maximum radiance per cell in a disk-backed accumulator.
generate_zones_vnl.py - Downsample
The image is block-averaged by a factor of 12 into a coarse grid of about 5.5 km pixels.
apply_skyglow.py - Scatter
A scatter kernel is convolved with the coarse grid (FFT) to estimate skyglow from every lit pixel within 80 km.
apply_skyglow.py - Rescan
The full-resolution image is read again. Each pixel gets its direct radiance plus the scatter from the coarse grid. Cells at or above 0.25 nW/cm²/sr are kept.
apply_skyglow.py - Write
Cells are sorted by H3 index and written to a binary
zones.dbwith a zone, the radiance and a sky brightness per cell.apply_skyglow.py - Publish
The file is split into SQL parts for the D1 database and uploaded to R2 for download.
export_zones_to_sql.py,upload_to_r2.py
Cells below the floor are not stored. A location that is not in the file is a dark site and counts as zone 1.
Source data
The input is the VIIRS Nighttime Lights annual composite for 2024 from the Earth Observation Group at the Colorado School of Mines, a 15 arc-second global grid of average radiance in nW/cm²/sr. Many of the group's products are published under CC BY 4.0 and must be cited. The licence of the exact 2024 product used here is still to be confirmed (unverified).
Elvidge, C.D., Zhizhin, M., Ghosh, T., Hsu, F.-C., Taneja, J. (2021). Annual time series of global VIIRS nighttime lights derived from monthly averages: 2012 to 2019. Remote Sensing 13(5), 922.
Skyglow
Each lit source pixel adds a little light at the observer. The kernel is Garstang-type: an exponential loss with distance, then a power-law falloff.
| Symbol | Meaning | Value | Script constant |
|---|---|---|---|
| Scatter fraction: share of upward light that scatters horizontally (the script's default). | 0.12 | SCATTER_FRACTION | |
| Scale length, km: exponential attenuation length. | 20 | SCATTER_SCALE_KM | |
| Reference distance, km: where the power law takes over. | 10 | D_REF_KM | |
| Power: power-law falloff. | 2.5 | SCATTER_POWER | |
| Maximum radius, km: scatter beyond this is dropped. | 80 | MAX_RADIUS_KM | |
| Coarse pixel, km: size of one pixel of the downsampled grid at the equator. | 5.55 | PIXEL_KM | |
| Downsample factor: 15 arc-second pixels averaged into one coarse pixel. | 12 | DOWNSAMPLE | |
| H3 resolution: cell size of the stored zones. | 8 | H3_RESOLUTION |
Production data was regenerated with instead of the script's default. That run reset the accumulator and passed --fraction 0.06 on the command line, so the script's own default still reads 0.12 and a plain run does not reproduce the data. The earlier documentation also listed a table of distances that did not follow from the formula. This is the table the formula gives for the production fraction:
Contribution to the sky of one lit pixel of 40 nW/cm²/sr (vertical, nW/cm²/sr, log scale) against distance in kilometres (horizontal, log scale). Dashed: F = 0.12 (script default). Solid: F = 0.06 (production data).
| Distance km | k(d) | Contribution nW/cm²/sr |
|---|---|---|
| 5 | 3.97e-2 | 1.6 |
| 10 | 1.82e-2 | 0.73 |
| 20 | 3.32e-3 | 0.13 |
| 30 | 8.07e-4 | 0.032 |
| 50 | 8.66e-5 | 0.0035 |
| 80 | 6.04e-6 | 0.00024 |
References: Garstang (1986), PASP 98, 364; Cinzano, Falchi and Elvidge (2001), MNRAS 328, 689.
H3 cells
Cells use Uber's H3 grid at resolution 8: hexagons of about 0.74 km² with an edge of about 0.46 km. A latitude and longitude maps to one 64-bit cell index, so a lookup is a single key. The app and the pipeline both use resolution 8.
H3 indices use the full unsigned 64-bit range, but SQLite and D1 store signed integers, so the worker converts between them.
Zones from radiance
The stored zone comes from radiance thresholds, and the stored sky brightness from a fit. These are the legacy chain, described with the new definition on Zone scale.
# VNL radiance thresholds (nW/cm²/sr) → Bortle zoneZONE_THRESHOLDS = [ (125.0, 9), # Inner city (NYC, London centers) (50.0, 8), # Dense urban (20.0, 7), # Urban (Dehradun ~40 → Zone 7) (9.0, 6), # Bright suburban (3.0, 5), # Suburban (1.0, 4), # Rural/suburban transition (0.50, 3), # Rural sky (0.25, 2), # Typical dark site]def radiance_to_zone(radiance: float) -> int: if radiance <= 0: return 1 for threshold, zone in ZONE_THRESHOLDS: if radiance >= threshold: return zone return 1 if radiance < ZONE2_RADIANCE else 2def radiance_to_sqm(radiance: float) -> float: """Convert VNL radiance to Sky Quality Meter reading (mag/arcsec²).""" if radiance <= 0: return 22.0 sqm = 22.0 - 1.7 * math.log10(1.0 + 2.0 * radiance) return max(16.0, min(22.0, sqm))The comment on the thresholds still says "Bortle", which is a leftover from an earlier name. The zones are Astr zones and are not Bortle classes.
The zones.db file
| Part | Size | Content |
|---|---|---|
| Header | 16 bytes | The ASCII bytes ASTR, a version word (01 00 00 00), then the record count as an unsigned 64-bit little-endian integer |
| Record | 20 bytes | H3 index (unsigned 64-bit), zone (1 byte), radiance (32-bit float), sky brightness (32-bit float), 3 bytes of padding, all little-endian |
| Order | Sorted by H3 index, so a lookup is a binary search | |
| Stored | Only cells in zone 2 or above; absence means zone 1 |
The current file holds 37,528,537 records in 750,570,756 bytes (716 MB).
Serving and lookup
A Cloudflare Worker answers GET /zone/{h3 hex} from a D1 table zones(h3, zone, radiance, sqm) and returns JSON. For a cell that is not in the table it answers zone 1 with implicit: true, which matches the app's local behaviour. Responses are cached for an hour, because the data is static. GET /download streams the whole file from R2.
The app converts the location to a resolution-8 cell, then asks CachedZoneRepository: a Hive cache first, then the Worker, then an expired cache entry if the network fails, then the dark-site default. Cache entries expire after a year. See Offline and sync.
Validation
The final check before deployment compared 25 places across continents, urban cores and dark sites, against expectations derived from the Lorenz light-pollution atlas.
| Place | Expected | Stored | Radiance | Ladder zone | Note |
|---|---|---|---|---|---|
| New York, USA | 9 | 9 | 247.76 | 9 | |
| London, UK | 9 | 9 | 206.97 | 9 | |
| Tokyo, Japan | 8 | 8 | 67.43 | 8 | |
| Sydney, Australia | 9 | 9 | 148.36 | 9 | |
| Paris, France | 8 | 8 | 97.86 | 8 | |
| Berlin, Germany | 8 | 8 | 80.53 | 8 | |
| Mumbai, India | 8 | 8 | 93.52 | 8 | |
| São Paulo, Brazil | 9 | 9 | 179.61 | 9 | |
| Cairo, Egypt | 9 | 9 | 166.62 | 9 | |
| Cape Town, South Africa | 8 | 8 | 83.89 | 8 | |
| Death Valley, USA | 1 | 1 | none | 1 | |
| Atacama Desert, Chile | 1 | 1 | none | 1 | |
| Teide, Canary Islands | 3 | 3 | 0.96 | 3 | |
| Mauna Kea, Hawaii | 1 | 1 | none | 1 | |
| Namib Desert, Namibia | 1 | 1 | none | 1 | |
| Antarctica McMurdo | 1 | 1 | none | 1 | |
| Greenland Nuuk | 6 | 6 | 15.92 | 6 | |
| Sahara Desert | 1 | 1 | none | 1 | |
| Gobi Desert, Mongolia | 1 | 1 | none | 1 | |
| Outback, Australia | 1 | 1 | none | 1 | |
| Reykjavik, Iceland | 7 | 9 | 125.82 | 9 | A city of about 230 thousand at 125 nW matches New York in density, which is implausible. Likely aurora or polar-night noise at 65°N. |
| Singapore | 9 | 7 | 36.97 | 7 | One of the densest city centres in Asia reads only 37 nW. Likely an equatorial under-read from persistent cloud and a low viewing angle. |
| Wellington, NZ | 8 | 8 | 59.47 | 8 | |
| Anchorage, Alaska | 8 | 8 | 124.66 | 9 | |
| Ushuaia, Argentina | 7 | 8 | 105.75 | 8 | Reads 105.75 nW for a town of about 80 thousand. Likely high-latitude VIIRS noise. |
The last column shows the zone each stored radiance would get under the doubling ladder: only Anchorage changes. The three mismatches are traced to the satellite data, not to the scatter model, and no change to the scatter parameters can fix them.
Limits
- Blue light. VIIRS does not see below 500 nm, so white LED lighting is under-counted (Falchi et al. 2016).
- Clouds and snow. Clouds can amplify skyglow near cities up to ten times, and snow changes ground reflection. The model assumes a clear night.
- Polar and equatorial noise. Aurora and polar-night noise inflate high latitudes, and persistent cloud deflates equatorial cities.
- Kernel geometry. The kernel uses a fixed 5.55 km pixel, so its east-west footprint shrinks relative to the north-south one away from the equator.
- Near field. Scatter inside a coarse pixel is excluded, so light from the immediate neighbourhood of a cell may be under-counted.
- Radius. The 80 km cutoff is shorter than the 195 km Falchi integrates to.
- Terrain. No elevation, mountain screening or varying aerosol is modelled.
- Staleness. The data is one annual composite and is regenerated yearly.
Regenerating
Changing a scatter parameter downwards needs the accumulator reset, because it keeps the maximum value per cell. The retune to 0.06 took 17 batches of ten strips at about 3.5 minutes each, around 54 minutes in total.
python scripts/generate_zones_vnl.py --tif <VNL annual composite>python scripts/apply_skyglow.py --tif <VNL annual composite> --accum zones_accumulator.db \ --fraction 0.06 --reset-accum --batch 10python scripts/export_zones_to_sql.pypython scripts/upload_to_r2.pyThe commands are from the pipeline's own guide in archive/data-pipeline/README.md; the flag names are in the scripts below.
Implementation
These are excerpts of the real files, cut out by name, copied into the site and checked against the repository on every build.
def create_scatter_kernel(): """Atmospheric scatter PSF. Center zeroed (no self-scatter).""" radius_px = int(MAX_RADIUS_KM / PIXEL_KM) + 1 size = 2 * radius_px + 1 kernel = np.zeros((size, size), dtype=np.float64) c = radius_px for y in range(size): for x in range(size): d = math.sqrt(((y - c) * PIXEL_KM)**2 + ((x - c) * PIXEL_KM)**2) if d < 0.5 or d > MAX_RADIUS_KM: continue kernel[y, x] = (SCATTER_FRACTION * math.exp(-d / SCATTER_SCALE_KM) / (1.0 + (d / D_REF_KM) ** SCATTER_POWER)) return kernel.astype(np.float32)def write_zones_db(accum_path, output_path): conn = sqlite3.connect(str(accum_path)) total = conn.execute('SELECT COUNT(*) FROM cells').fetchone()[0] print(f"\nWriting {total:,} cells to {output_path}") written = 0 skipped = 0 with open(output_path, 'wb') as f: f.write(b'ASTR\x01\x00\x00\x00') count_pos = f.tell() f.write(struct.pack('<Q', 0)) cursor = conn.execute('SELECT h3, radiance FROM cells ORDER BY h3') while True: rows = cursor.fetchmany(100_000) if not rows: break for h3_int, rad in rows: zone = radiance_to_zone(rad) if zone <= 1: skipped += 1; continue sqm = radiance_to_sqm(rad) f.write(struct.pack('<Q', h3_int)) f.write(struct.pack('B', zone)) f.write(struct.pack('<f', rad)) f.write(struct.pack('<f', sqm)) f.write(b'\x00\x00\x00') written += 1 f.seek(count_pos) f.write(struct.pack('<Q', written)) sha = hashlib.sha256() with open(output_path, 'rb') as f: for chunk in iter(lambda: f.read(8192), b''): sha.update(chunk) size_mb = output_path.stat().st_size / (1024**2) print(f"\n{'='*50}") print(f"SUCCESS!") print(f" Records: {written:,} (skipped {skipped:,} Zone 1)") print(f" Size: {size_mb:.1f} MB") print(f" SHA-256: {sha.hexdigest()}") print(f"{'='*50}") # Quick validation print("\nValidation:") for name, lat, lon in [ ("Bhadraj Temple", 30.5167, 78.0333), ("Dehradun", 30.3165, 78.0322), ("Hanle", 32.7795, 78.9641), ("New York City", 40.7128, -74.0060), ("Null Island", 0.0, 0.0), ]: h3_cell = h3.latlng_to_cell(lat, lon, H3_RESOLUTION) h3_int = int(h3_cell, 16) row = conn.execute('SELECT radiance FROM cells WHERE h3=?', (h3_int,)).fetchone() if row: z = radiance_to_zone(row[0]) print(f" {name}: Zone {z} (radiance={row[0]:.4f})") else: print(f" {name}: Zone 1 (Implicit)") conn.close()/** * Look up zone data for a given H3 hex index. * * D1 table schema: * zones(h3 INTEGER PRIMARY KEY, zone INTEGER, radiance REAL, sqm REAL) * * "h3" is stored as a signed 64-bit integer (SQLite INTEGER). * We convert the hex string to a BigInt, then to a signed int for SQL. */async function handleZoneLookup(h3Hex, env) { try { // Convert hex to BigInt, then to signed 64-bit for D1 (SQLite stores signed) const h3BigInt = BigInt('0x' + h3Hex); // Convert unsigned to signed 64-bit representation const h3Signed = toSigned64(h3BigInt); const row = await env.DB.prepare( 'SELECT zone, radiance, sqm FROM zones WHERE h3 = ?' ).bind(h3Signed.toString()).first(); if (row) { return jsonResponse({ bortle: row.zone, ratio: Math.round(row.radiance * 100) / 100, sqm: Math.round(row.sqm * 100) / 100, h3: h3Hex, }, 200, { 'Cache-Control': `public, max-age=${CACHE_TTL}`, }); } // Not found → Zone 1 (pristine dark sky) return jsonResponse({ bortle: 1, ratio: 0.0, sqm: 22.0, h3: h3Hex, implicit: true, }, 200, { 'Cache-Control': `public, max-age=${CACHE_TTL}`, }); } catch (error) { console.error('Zone lookup error:', error); return jsonResponse({ error: 'Internal server error' }, 500); }}/** * Convert an unsigned 64-bit BigInt to signed 64-bit representation. * * SQLite/D1 stores INTEGER as signed. H3 indices use the full unsigned * 64-bit range, so values with the high bit set appear negative in SQL. */function toSigned64(unsigned) { const MAX_UINT64 = (1n << 64n) - 1n; const MAX_INT64 = (1n << 63n) - 1n; if (unsigned > MAX_UINT64) { throw new Error('Value exceeds uint64 range'); } if (unsigned > MAX_INT64) { // High bit set → negative in signed representation return unsigned - (1n << 64n); } return unsigned;}/// Fetches zone data for the given H3 index from remote API.////// Returns:/// - [ZoneData] on success/// - `null` on network error, timeout, or 404 (not found)////// Throws: Nothing - all errors are caught and return null for graceful fallback.Future<ZoneData?> getZoneData(BigInt h3Index) async { final String h3Hex = h3Index.toRadixString(16); final Uri uri = Uri.parse('$_baseUrl/zone/$h3Hex'); try { final http.Response response = await _client .get(uri) .timeout(_timeout); if (response.statusCode == 200) { final Map<String, dynamic> json = jsonDecode(response.body) as Map<String, dynamic>; return ZoneData( astrZone: (json['bortle'] as num).toInt(), ratio: (json['ratio'] as num).toDouble(), sqm: (json['sqm'] as num).toDouble(), ); } else if (response.statusCode == 404) { // H3 index not found in database - expected for ocean/unpopulated areas debugPrint('Zone not found for H3 $h3Hex'); return null; } else { debugPrint('Zone API error: ${response.statusCode}'); return null; } } catch (e) { debugPrint('Zone API request failed: $e'); return null; }}/// Get zone data for an H3 index, using cache-first strategy.////// Returns:/// - [ZoneData] from cache or remote API/// - [pristineDarkSky] if not found in database (dark sky location!)Future<ZoneData> getZoneData(BigInt h3Index) async { final String h3Hex = h3Index.toRadixString(16); final String cacheKey = '$_keyPrefix$h3Hex'; // 1. Check cache first final ZoneCacheEntry? cached = _cache.get(cacheKey); if (cached != null && !cached.isExpired) { debugPrint('Zone cache hit for $h3Hex'); return ZoneData( astrZone: cached.astrZone, ratio: cached.ratio, sqm: cached.sqm, ); } // 2. Cache miss — fetch from remote D1 API debugPrint('Zone cache miss for $h3Hex, fetching from remote...'); final ZoneData? remoteData = await _remote.getZoneData(h3Index); if (remoteData != null) { _cacheZoneData(cacheKey, h3Hex, remoteData); return remoteData; } // 3. Remote failed - check if we have stale cache if (cached != null) { debugPrint('Remote failed, using expired cache for $h3Hex'); return ZoneData( astrZone: cached.astrZone, ratio: cached.ratio, sqm: cached.sqm, ); } // 4. Not in database = pristine dark sky location debugPrint('$h3Hex not in lit-areas DB, returning pristine dark sky'); return pristineDarkSky;}