Characterization Metric Report Demo: Job 1163#
By Simon Krughoff and Michael Wood-Vasey
Abstract#
This brief report describes measurements of interest that were carried out for Job 1163 of the Science Pipeline.
In short, the astrometric performance is excellent in comparison to both the per cycle ramps as well as the design level KPMs. The photometric performance is not as good. The inclusion of jointcal in the processing in future cycles will improve photometric performance. Current reprocessing of HSC data using a precursor to jointcal called meas_mosaic shows significant improvement over single epoch processing.
Introduction#
Measured using validation_data_hsc, which consists of 8 HSC engineering images: 2 r-band, 4 i’-band, and 2 y-band. Measurements were made on individual, separately-processed, single frame images: jointcal and/or meas_mosaic were not run. For comparison, we provide the SRD required “design” value of each metric as defined in the Science Requirements Document [LPM-17], and, where available, the target for this release as defined in the Data Management Development Milestone Roadmap [LDM-240]. All values were computed using the examples/runHscTest.sh script in the validate_drp package.
Some KPMs (AF1, AD1) involve thresholds that are different for “design”, “minimum”, and “stretch” specifications. Thus comparing one of these metrics against a given target number is a two-level process. Both the threshold used in the calculation is dependent on the specifications, and the requirement on the computed number is dependent on the specifications.
The metrics in this report have all been computed relative to the “design” thresholds. The values of these KPMs would be different if computed against different thresholds.
Note also that the photometric performance of the pipelines in the y-band is an under estmate of expected delivered performance. For these tests, the y-band data was calibrated with z-band photometry. This is due to the lack of a reference catalog containing y-band information at this time. We recognize that the bandpass mismatch is certainly not the only source of scatter in the y-band photometry. These metric measurements are still worth noting in this report as a historical benchmark to track relative performance.
The per cycle target numbers come from the “KPMs” sheet of LDM-240.
from astropy.table import Table, Column
import numpy as np
band_map = {'HSC-R': 'r', 'HSC-I': 'i'}
jobs_ids = {'HSC-R': 1661, 'HSC-I': 1660}
data = {'Metric':['PA1', 'PA1', 'AM1', 'AM1'], 'band':['HSC-R', 'HSC-I', 'HSC-R', 'HSC-I'], 'metric value':[np.nan, np.nan, np.nan, np.nan],
'spec operator':['<=', '<=', '<=', '<='], 'spec value':[np.nan, np.nan, np.nan, np.nan]}
table = Table(data)
import requests
jobs = {}
for band, job_id in jobs_ids.items():
r = requests.get("https://squash-restful-api.lsst.codes/job/%i"%job_id)
jobs[band] = r.json()
def make_get_str(metric_name, band):
if metric_name.startswith('AM'):
get_str = 'https://squash-restful-api.lsst.codes/spec/validate_drp.%s.design'%(metric_name)
elif metric_name.startswith('PA'):
get_str = 'https://squash-restful-api.lsst.codes/spec/validate_drp.%s.hsc_design_%s'%(metric_name, band)
else:
raise ValueError('Only AM and PA metrics supported currently')
return get_str
units = []
for i, v in enumerate(zip(table['Metric'], table['band'])):
for measurement in jobs[v[1]]['measurements']:
if measurement['metric'] == 'validate_drp.'+v[0]:
table['metric value'][i] = measurement['value']
units.append(measurement['unit'])
r = requests.get(make_get_str(v[0], band_map[v[1]]))
result = r.json()
table['spec value'] = result['threshold']['value']
col = Column(units, name='unit')
table.add_column(col)
table.show_in_notebook()