Skip to content
Snippets Groups Projects
Commit 52d5f94c authored by Austin Sanders's avatar Austin Sanders
Browse files

Added warning to deactivate ALE environment when creating truth data

parent 1c151152
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
docs/getting-started/writing-tests/writing-tests-for-ale-drivers.ipynb
+ 8
− 0
View file @ 52d5f94c
%% Cell type:markdown id:83ffc4ff tags:
# Writing Tests for ALE Drivers
Before merging a new driver into ALE, it is necessary to create automated tests to ensure that the driver functions properly. Writing a test for ALE drivers entails two separate stages -- preparing the test data and writing the test itself.
<div class="admonition info">
<p class="admonition-title">Prerequisites</p>
<p>This tutorial assumes that the user has:
<list>
<li>an active conda environment with ALE and its dependencies</li>
<li>base and mission-specific SPICE data (camera kernels)</li>
<li>access to ISIS applications (spiceinit)</li>
</list>
</p>
</div>
## Prepare Test Data
Naturally, ALE drivers require instrument-specific data. If you have not yet identified a source of test data for your driver, the [PDS Image Atlas](https://pds.nasa.gov/datasearch/data-search/) provides a catalog of data from which you can select relevant images.
The data used in this tutorial can be found [here](https://wms.lroc.asu.edu/lroc/view_lroc/LRO-L-LROC-2-EDR-V1.0/M1435111335LE)
### Create and Spiceinit a Cube
After downloading image data, it is necessary to convert the image to ISIS cube format and spiceinit the data. The following command will convert and spiceinit an LRO NAC image, but the specific "2isis" command is mission-specific. Unlike other code segments in this notebook, these commands should be run in a terminal.
<div class="admonition warning">
<p class="admonition-title">Create the Truth Data with ISIS!</p>
<p> This cub file will be used as truth data, so it should be spiceinitted using ISIS camera models rather than the ALE driver. You should <em>deactivate</em> your ALE environment when spiceinitting this cube!
</p>
</div>
``` BASH
lronac2isis from=M1435111335LE.IMG to=M1435111335LE.cub
spiceinit from=M1435111335LE.cub
```
### Save the Cube Label
After creating the ISIS formatted cube, it is necessary to pull the label off the cube. This is easily performed using ISIS's `catlab` utility via the command line:
``` BASH
catlab from=M1435111335LE.cub to=M1435111335LE_isis3.lbl
```
### Slice the Kernels
Due to repository size limitations, it is necessary to 'slice' the kernels with *ckslicer*, which is a NAIF utility that can be downloaded [here](https://naif.jpl.nasa.gov/naif/utilities.html). Make sure that you download the utility that corresponds to your operating system and remember where you downloaded the program.
#### Import and Set Up Data Locations
This portion of the code is responsible for loading the libraries necessary for slicing and merging kernels. It's not necessary to alter any of the imports, but be sure to edit the cube locations so that they correspond to your directory structure and create the output_dir if necessary!
%% Cell type:code id:333c6885 tags:
``` python
import spiceypy as spice
import pvl
import os
import re
import subprocess
from ale import util
from itertools import chain
import io
import networkx as nx
# These should be provided when running this script.
cube = "M1435111335LE.cub"
output_dir = "kernels/" # Output dir for created kernel files
data_dir = "/Users/arsanders/isis_efs/isis_data/" # Dir of where to pull original kernels from
```
%% Cell type:markdown id:19895955 tags:
#### Create Utility Functions
Next, it is necessary to create utility functions to merge intervals together and add light-time correction. There is generally no need to adjust these functions, but it is necessary to load them into memory -- just run this cell!
%% Cell type:code id:6e3638b5 tags:
``` python
def merge_intervals(intervals):
"""
Merge a set of intervals. The intervals are assumed to be closed, that is they include the end-points.
Parameters
----------
intervals : list
The input list of intrvals where each interval is a tuple of (start, end)
Returns
-------
: list
A sorted list of the merged intervals
"""
sorted_intervals = sorted(intervals, key=lambda tup: tup[0])
merged = [sorted_intervals[0]]
for interval in sorted_intervals[1:]:
# No intersection
if interval[0] > merged[-1][1]:
merged.append(interval)
# Intersection, but new interval isn't wholly contained
elif interval[1] > merged[-1][1]:
merged[-1] = (merged[-1][0], interval[1])
return merged
def add_light_time_correction(cube_info, padding=120):
"""
Compute the time intervals for the image and any light time correction
Parameters
----------
cube_info : ordered dict
The cube info from ale.util.generate_kernels_from_cube
padding : float
Time padding in seconds to add to each interval
Returns
-------
: list
A sorted list of the intervals as (start_et, stop_et)
"""
image_start_et = spice.scs2e(cube_info['SpacecraftID'], cube_info['SpacecraftClockCount'])
image_end_et = image_start_et + cube_info['ExposureDuration'] * cube_info['Lines']
inst_state, inst_lt = spice.spkez(cube_info['SpacecraftID'], image_start_et, 'J2000', 'NONE', 0)
target_state, target_lt = spice.spkez(cube_info['TargetID'], image_start_et, 'J2000', 'NONE', 0)
sun_state, sun_lt = spice.spkez(10, image_start_et, 'J2000', 'NONE', cube_info['TargetID'])
intervals = [
(image_start_et - padding, image_end_et + padding),
(image_start_et - padding - inst_lt, image_end_et + padding - inst_lt),
(image_start_et - padding - target_lt, image_end_et + padding - target_lt),
(image_start_et - padding - sun_lt, image_end_et + padding - sun_lt)]
return merge_intervals(intervals)
```
%% Cell type:markdown id:37bcdec1 tags:
#### Read and Furnish Kernels
This section of the code is responsible for reading the list of kernels from a spiceinit'd cube and loading those kernels into memory. No edits are necessary to this portion of the notebook.
%% Cell type:code id:03fb3142 tags:
``` python
# Get dictionary of kernel lists from cube
cube_info = util.generate_kernels_from_cube(cube, format_as = 'dict')
# Replace path variables with absolute paths for kernels
for kernel_list in cube_info:
for index, kern in enumerate(cube_info[kernel_list]):
if kern is not None:
cube_info[kernel_list][index] = data_dir + kern.strip('$')
# Create ordered list of kernels for furnishing
kernels = [kernel for kernel in chain.from_iterable(cube_info.values()) if isinstance(kernel, str)]
spice.furnsh(kernels)
```
%% Cell type:markdown id:15a194ee tags:
#### Load and Update the Cube Information
This portion of the code is responsible for reading the .cub label and updating the dictionary with information relevant to the upcoming kernel slicing process.
<div class="admonition note">
<p class="admonition-title">Updating Dictionary Elements</p>
<p>
In some circumstances, it may be necessary to update the lowest-level dictionary elements to match those found in your cube's label, i.e. "SpacecraftClockStartCount" or "Line Exposure Duration" may not be present in your label. It may also be necessary to add (or remove) a list index if your value contains a unit specifier. For example, the "LineExposureDuration" keyword specifies units, so we have to add the [0] to pull the value out of the tuple. If your LineExposureDuration does not contain units, simply remove the [0].
</p>
</div>
%% Cell type:code id:b7ae7d49 tags:
``` python
# Loads cube as pvl to extract rest of data
cube_pvl = pvl.load(cube)
# Save other necessary info in cube_info dict
cube_info.update(Lines = cube_pvl['IsisCube']['Core']['Dimensions']['Lines'])
cube_info.update(SpacecraftClockCount = cube_pvl['IsisCube']['Instrument']['SpacecraftClockStartCount'])
cube_info.update(ExposureDuration = cube_pvl['IsisCube']['Instrument']['LineExposureDuration'][0])
cube_info.update(TargetID = spice.bods2c(cube_pvl['IsisCube']['Instrument']['TargetName']))
cube_info.update(SpacecraftID = spice.bods2c(cube_pvl['IsisCube']['Instrument']['SpacecraftName']))
```
%% Cell type:markdown id:58702769 tags:
#### Process and Slice Kernels
After collecting the necessary metadata, it is possible to 'slice' out the portions of kernels that are relevant to your specific image. The following code will slice the kernels and convert them to transfer format. This cell may require the user to specify the full path to the ckslicer executable. Alternatively, the user can place the utility in the same directory as the notebook.
%% Cell type:code id:0f65c42d tags:
``` python
# Account for light time correction
intervals = add_light_time_correction(cube_info)
# For each binary ck kernel specified in cube, run the ckslicer, comment and to-transfer commands
for ck in [k for k in kernels if k.lower().endswith('.bc')]:
ck_path, ck_file_extension = os.path.splitext(ck)
ck_basename = os.path.basename(ck_path)
for index, interval in enumerate(intervals):
for frame in util.get_ck_frames(ck):
output_basename = os.path.join(output_dir, ck_basename + '_' + str(index) + '_sliced_' + str(frame))
output_kern = output_basename + ck_file_extension
output_comments = output_basename + '.cmt'
start_sclk = spice.sce2s(cube_info['SpacecraftID'], interval[0])
end_sclk = spice.sce2s(cube_info['SpacecraftID'], interval[1])
# Create new sliced ck kernel
ckslicer_command = ["./ckslicer",
'-LSK {}'.format(cube_info['LeapSecond'][0]),
'-SCLK {}'.format(cube_info['SpacecraftClock'][0]),
'-INPUTCK {}'.format(ck),
'-OUTPUTCK {}'.format(output_kern),
'-ID {}'.format(str(frame)),
'-TIMETYPE {}'.format('SCLK'),
'-START {}'.format(start_sclk),
'-STOP {}'.format(end_sclk)]
subprocess.run(ckslicer_command, check=True)
# Remove old comments from new ck kernel
commnt_command = ['commnt', '-d {}'.format(output_kern)]
subprocess.run(commnt_command, check=True)
with open(output_comments, 'w+') as comment_file:
comment_file.write("This CK is for testing with the image: {}\n".format(cube))
comment_file.write("\nThis CK was generated using the following command: {}\n")
comment_file.write(" ".join(ckslicer_command))
# Add new comments to new ck kernel
new_commnts_command = ["commnt", "-a {}".format(output_kern), output_comments]
subprocess.run(new_commnts_command, check=True)
# Create the transfer file of the new ck kernel
subprocess.run(["toxfr", output_kern], check=True)
```
%% Cell type:markdown id:34d3ffe9 tags:
#### Merge the Kernels
After slicing the kernels, it is necessary to merge them. No edits are necessary for this portion of the notebook.
%% Cell type:code id:76a3955b tags:
``` python
# Create the config file for the spkmerge command
for index, interval in enumerate(intervals):
output_spk_basename = os.path.join(output_dir, os.path.basename(os.path.splitext(cube)[0]) + '_' + str(index))
output_spk = output_spk_basename + '.bsp'
start_utc = spice.et2utc(interval[0], 'c', 3)
end_utc = spice.et2utc(interval[1], 'c', 3)
spk_dep_tree = util.create_spk_dependency_tree([k for k in kernels if k.lower().endswith('.bsp')])
config_string = util.spkmerge_config_string(spk_dep_tree,
output_spk,
[cube_info['TargetID'], cube_info['SpacecraftID'], 10],
cube_info['LeapSecond'][0],
start_utc,
end_utc)
with open(output_spk_basename + '.conf', 'w+') as spk_config:
spk_config.write(config_string)
# Create the new SPK
spkmerge_command = ["spkmerge", spk_config.name]
subprocess.run(spkmerge_command, check=True)
# Create the transfer file of the new SPK kernel
subprocess.run(["toxfr", output_spk], check=True)
```
%% Cell type:markdown id:ec6e395d tags:
### Generate an ISD
ALE provides the isd_generate.py utility within the 'ale' directory. To use this utility, the user must ensure that the ALESPICEROOT environment variable is set to the data directory containing kernels. This can be achieved using
``` BASH
export ALESPICEROOT=<your_data_directory>
```
After setting ALESPICEROOT, the user can use the python utility to generate an ISD. An example is as follows:
``` BASH
python isd_generate.py M1435111335LE.cub -o lro_nac_isd.json
```
%% Cell type:markdown id:163b7bc7 tags:
### Place Test Data in ALE
There are numerous data locations and naming conventions that must be followed in order for the tests to function properly. To correctly integrate the test data into ALE's structure, the user should:
- Ensure that your files are named as follows:
- isd: \<instrument_name\>_isd.json
- cube label: \<filename\>_isis3.lbl
- Create a directory within ale/tests/pytests/data that shares a name with your test data
- Place the cube label into the directory that you created
- Place all transfer kernels into the directory that you created (*.xc, *.xsp, *.tls, *.tpc, *.ti, *.tsc)
- Place the isd within the existing ale/tests/pytests/data/isds directory
%% Cell type:markdown id:fa6aefcf tags:
## Writing the Test
ALE uses the PyTest library for automated testing. This section will guide the user through the process of creating a PyTest for an ALE driver using the test data that was generated using the first part of this tutorial.
### Create a Test Fixture
In PyTest, a test [fixture](https://docs.pytest.org/en/6.2.x/fixture.html) is an object that models test data, and it includes all the data, setup, and teardown instructions that are necessary to perform automated tests. The following cell demonstrates a test fixture that provides all the necessary test data for our automated tests.
%% Cell type:code id:20feebe3 tags:
``` python
@pytest.fixture(scope="module")
def test_kernels():
updated_kernels = {}
binary_kernels = {}
for image in image_dict.keys():
kernels = get_image_kernels(image)
updated_kernels[image], binary_kernels[image] = convert_kernels(kernels)
yield updated_kernels
for kern_list in binary_kernels.values():
for kern in kern_list:
os.remove(kern)
```
%% Cell type:markdown id:e69aff70 tags:
### Create a 'Loads' Test
The first test that a driver needs to pass is whether or not it is able to parse a label as expected. The following cell performs the parsing, loading, and comparison of the ISD to a 'truth' ISD that comes from the ISIS Camera Model.
%% Cell type:code id:40f5c86f tags:
``` python
# Test load of LROC labels
@pytest.mark.parametrize("label_type, kernel_type", [('isis3', 'naif'), ('isis3', 'isis')])
@pytest.mark.parametrize("image", ['M1435111335LE'])
def test_load(test_kernels, label_type, image, kernel_type):
if kernel_type == 'naif':
label_file = get_image_label(image, label_type)
isd_str = ale.loads(label_file, props={'kernels': test_kernels[image]})
compare_isd = image_dict[image]
else:
label_file = get_image(image)
isd_str = ale.loads(label_file)
compare_isd = get_isd('lro_isis')
isd_obj = json.loads(isd_str)
comparison = compare_dicts(isd_obj, compare_isd)
assert comparison == []
```
%% Cell type:markdown id:69ae1920 tags:
### Create Comparison Tests
After verifying that the driver passes the 'loads' test, it is necessary to test each function within the driver. For this portion of testing, it is necessary to test any function that your driver overrides. In general, this step involves testing that the driver's method returns the expected value from the cube label.
<div class="admonition note">
<p class="admonition-title">Notes on 'patch'</p>
<p>The unittest library provides a means of mocking objects using 'patch'. All spice calls are mocked using this process.For more information on the use of 'patch,' visit the unittest <a href=https://docs.python.org/3/library/unittest.mock.html>documentation</a>.</p>
</div>
%% Cell type:code id:eb129f6d tags:
``` python
# ========= Test isislabel and naifspice driver =========
class test_isis_naif(unittest.TestCase):
def setUp(self):
label = get_image_label('M1435111335LE', 'isis3')
self.driver = LroLrocNacIsisLabelNaifSpiceDriver(label)
def test_short_mission_name(self):
assert self.driver.short_mission_name == 'lro'
def test_intrument_id(self):
assert self.driver.instrument_id == 'LRO_LROCNACL'
def test_usgscsm_distortion_model(self):
with patch('ale.drivers.lro_drivers.spice.gdpool', return_value=np.array([1.0])) as gdpool, \
patch('ale.drivers.lro_drivers.spice.bods2c', return_value=-12345) as bods2c:
distortion_model = self.driver.usgscsm_distortion_model
assert distortion_model['lrolrocnac']['coefficients'] == [1.0]
gdpool.assert_called_with('INS-12345_OD_K', 0, 1)
bods2c.assert_called_with('LRO_LROCNACL')
def test_odtk(self):
with patch('ale.drivers.lro_drivers.spice.gdpool', return_value=np.array([1.0])) as gdpool, \
patch('ale.drivers.lro_drivers.spice.bods2c', return_value=-12345) as bods2c:
assert self.driver.odtk == [1.0]
gdpool.assert_called_with('INS-12345_OD_K', 0, 1)
bods2c.assert_called_with('LRO_LROCNACL')
def test_light_time_correction(self):
assert self.driver.light_time_correction == 'NONE'
def test_detector_center_sample(self):
with patch('ale.drivers.lro_drivers.spice.gdpool', return_value=np.array([1.0])) as gdpool, \
patch('ale.drivers.lro_drivers.spice.bods2c', return_value=-12345) as bods2c:
assert self.driver.detector_center_sample == 0.5
gdpool.assert_called_with('INS-12345_BORESIGHT_SAMPLE', 0, 1)
bods2c.assert_called_with('LRO_LROCNACL')
def test_exposure_duration(self):
np.testing.assert_almost_equal(self.driver.exposure_duration, .0010334296)
def test_ephemeris_start_time(self):
with patch('ale.drivers.lro_drivers.spice.scs2e', return_value=321) as scs2e:
np.testing.assert_almost_equal(self.driver.ephemeris_start_time, 322.05823191)
scs2e.assert_called_with(-85, '1/270649237:07208')
def test_multiplicative_line_error(self):
assert self.driver.multiplicative_line_error == 0.0045
def test_additive_line_error(self):
assert self.driver.additive_line_error == 0
def test_constant_time_offset(self):
assert self.driver.constant_time_offset == 0
def test_additional_preroll(self):
assert self.driver.additional_preroll == 1024
def test_sampling_factor(self):
assert self.driver.sampling_factor == 1
@patch('ale.transformation.FrameChain')
@patch('ale.transformation.FrameChain.from_spice', return_value=ale.transformation.FrameChain())
@patch('ale.transformation.FrameChain.compute_rotation', return_value=TimeDependentRotation([[0, 0, 1, 0]], [0], 0, 0))
def test_spacecraft_direction(self, compute_rotation, from_spice, frame_chain):
with patch('ale.drivers.lro_drivers.LroLrocNacIsisLabelNaifSpiceDriver.target_frame_id', \
new_callable=PropertyMock) as target_frame_id, \
patch('ale.drivers.lro_drivers.LroLrocNacIsisLabelNaifSpiceDriver.ephemeris_start_time', \
new_callable=PropertyMock) as ephemeris_start_time, \
patch('ale.drivers.lro_drivers.spice.cidfrm', return_value=[-12345]) as cidfrm, \
patch('ale.drivers.lro_drivers.spice.scs2e', return_value=0) as scs2e, \
patch('ale.drivers.lro_drivers.spice.bods2c', return_value=-12345) as bods2c, \
patch('ale.drivers.lro_drivers.spice.spkezr', return_value=[[1, 1, 1, 1, 1, 1], 0]) as spkezr, \
patch('ale.drivers.lro_drivers.spice.mxv', return_value=[1, 1, 1]) as mxv:
ephemeris_start_time.return_value = 0
assert self.driver.spacecraft_direction > 0
spkezr.assert_called_with(self.driver.spacecraft_name, 0, 'J2000', 'None', self.driver.target_name)
compute_rotation.assert_called_with(1, -12345)
np.testing.assert_array_equal(np.array([[-1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, 1.0]]), mxv.call_args[0][0])
np.testing.assert_array_equal(np.array([1, 1, 1]), mxv.call_args[0][1])
def test_focal2pixel_lines(self):
with patch('ale.drivers.lro_drivers.spice.gdpool', return_value=[0, 1, 0]) as gdpool, \
patch('ale.drivers.lro_drivers.LroLrocNacIsisLabelNaifSpiceDriver.ikid', \
new_callable=PropertyMock) as ikid, \
patch('ale.drivers.lro_drivers.LroLrocNacIsisLabelNaifSpiceDriver.spacecraft_direction', \
new_callable=PropertyMock) as spacecraft_direction:
spacecraft_direction.return_value = -1
np.testing.assert_array_equal(self.driver.focal2pixel_lines, [0, -1, 0])
spacecraft_direction.return_value = 1
np.testing.assert_array_equal(self.driver.focal2pixel_lines, [0, 1, 0])
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment