Skip to content
Snippets Groups Projects
Unverified Commit 5e3f64f2 authored by jlaura's avatar jlaura Committed by GitHub
Browse files

Update to #4937 - adds scripts for S3 download (#5013)


* rebase fix

* added pytest to Jenkinsfile

* updated Jenkinsfile

* made <dest> optional for lsf

* added configurable config

* typos

* simplified some logic

* cleaned up verbose flags a little

* cleaned up configs too

* removed extra '/'

* Fixed jenkins pytests path

* fixed a few bugs

* disabled other tests for now

* jenkinsfile path is off

* simplified things

* updated tests, installing now

* tweaking Jenkinsfile

* typo

* missed a spot

* mock, mocks, mo-mo-mo-mo mocks

* reverted jenkinsfile

* updates: readme for s3 data downloads

* Updates: remove unused sync and update filename

* Updates for comments

* fixes: typo in readme found in review

Co-authored-by: default avatarGavin <nelsong916@gmail.com>
Co-authored-by: default avatarKelvin Rodriguez <krodriguez@usgs.gov>
Co-authored-by: default avatarJesse Mapel <jam826@nau.edu>
parent 793dcff5
No related branches found
No related tags found
No related merge requests found
...@@ -93,6 +93,14 @@ for (lbl in labels) { ...@@ -93,6 +93,14 @@ for (lbl in labels) {
errors.add(stageStatus) errors.add(stageStatus)
osFailed = true osFailed = true
} }
// pytests
stageStatus = "Running pytests on ${label}"
try {
loginShell "cd $WORKSPACE/isis/pytests && pytest ."
} catch(e) {
errors.add(stageStatus)
osFailed = true
}
if (osFailed) { if (osFailed) {
error "Failed on ${label}" error "Failed on ${label}"
......
...@@ -376,19 +376,8 @@ Under the root directory of the ISIS Data Area pointed to by the ISISDATA/ISIS3D ...@@ -376,19 +376,8 @@ Under the root directory of the ISIS Data Area pointed to by the ISISDATA/ISIS3D
### Versions of the ISIS Data Area ### Versions of the ISIS Data Area
In ISIS version 4.1.0, several files previously stored in the data area closely associated with ISIS applications were moved into version control with the ISIS source code. Additionally, the environment variables used for the ISIS Data Area and the rsync location for the ISIS Data Area were also updated. In ISIS version 4.1.0 and later, several files previously stored in the data area closely associated with ISIS applications were moved into version control with the ISIS source code. To support the use of data in ISIS versions predating 4.1.0 the `downloadIsisData` application will need to download the data named `legacybase`. This is explained further in the [Full ISIS Data Download](README.md#Full-ISIS-Data-Download) section.
The correct environment variable names and rsync modules to use for the ISIS Data Area for each version of ISIS are summarized in the table below:
ISIS version | ISIS Data Environment variable name | ISIS Data rsync module
---|---|---
3.x | `$ISIS3DATA` | `isis3data`
4.0.x | `$ISIS3DATA` | `isis3data`
4.1.0 | `$ISISDATA` | `isisdata`
The ISIS Data rsync module specifies where to rsync the data from and is the name used after the `::` in the rsync download commands below. For example, the rsync module is in bold in the following example rsync command:
``rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::``**``isis3data``**``/data/`` .
### Size of the ISIS Data Area ### Size of the ISIS Data Area
...@@ -396,17 +385,17 @@ If you plan to work with data from all missions, then the download will require ...@@ -396,17 +385,17 @@ If you plan to work with data from all missions, then the download will require
### Full ISIS Data Download ### Full ISIS Data Download
The ISIS Data Area is hosted on rsync servers and not through conda channels like the ISIS binaries. This requires using the rsync command from within a terminal window within your Unix distribution, or from within WSL if running Windows 10. Downloading all mission data requires over 520 GB of disk space. If you want to acquire only certain mission data [click here](#Mission-Specific-Data-Downloads). To download all ISIS data files, continue reading. > Warning if you are looking to download ISIS data via rsync the servers will be shutdown in November of 2022.
the outdated rsync download information can be found [here](https://github.com/USGS-Astrogeology/ISIS3/wiki/Outdated-ISIS-Data-Information)
To download all ISIS data, enter the following commands in the location where you want to install the ISIS Data Area, for versions of ISIS 4.1.0 and later:
cd $ISISDATA The ISIS Data Area is hosted on a combination of AWS S3 buckets and public http servers e.g. NAIF, Jaxa, ESA and not through conda channels like the ISIS binaries. This requires using the `downloadIsisData.py` script from within a terminal window within your Unix distribution, or from within WSL if running Windows 10. Downloading all mission data requires over 520 GB of disk space. If you want to acquire only certain mission data [click here](#Mission-Specific-Data-Downloads). To download all ISIS data files, continue reading.
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/ .
For earlier versions, use: To download all ISIS data, use the following command:
cd $ISIS3DATA downloadIsisData all $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isis3data/data/ .
> Note: this applicaion takes in 3 parameters in the following order \<rclone command> \<mission> \<download destination>
> Note: The above command downloads all ISIS data including the required base data area and all of the optional mission data areas. > Note: The above command downloads all ISIS data including the required base data area and all of the optional mission data areas.
...@@ -414,11 +403,11 @@ For earlier versions, use: ...@@ -414,11 +403,11 @@ For earlier versions, use:
This data area contains data that is common between multiple missions such as DEMS and leap second kernels. As of ISIS 4.1, the base data area is no longer required to run many applications as data such as icons and templates has been moved into the binary distribution. If you plan to work with any applications that use camera models (e.g., cam2map, campt, qview), it is still recommended you download the base data area. To download the base data area run the following commands: This data area contains data that is common between multiple missions such as DEMS and leap second kernels. As of ISIS 4.1, the base data area is no longer required to run many applications as data such as icons and templates has been moved into the binary distribution. If you plan to work with any applications that use camera models (e.g., cam2map, campt, qview), it is still recommended you download the base data area. To download the base data area run the following commands:
cd $ISISDATA downloadIsisData base $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/base .
For versions of ISIS prior to ISIS 4.1.0, please use `isis3data` instead of `isisdata` in the above command. > Note: For accessing ISIS Data for versions of ISIS prior to ISIS 4.1.0, you must download the `legacybase` area and not the base area when using this application as shown below:
downloadIsisData legacybase $ISISDATA
### Partial Download of Mission Specific Data ### Partial Download of Mission Specific Data
There are many missions supported by ISIS. If you are only working with a few missions then you can save disk space by downloading only those specific data areas. If you want to limit the download even further, read the next section about the SPICE Web Service. Otherwise [jump](#Mission-Specific-Data-Downloads) to the mission specific sections. There are many missions supported by ISIS. If you are only working with a few missions then you can save disk space by downloading only those specific data areas. If you want to limit the download even further, read the next section about the SPICE Web Service. Otherwise [jump](#Mission-Specific-Data-Downloads) to the mission specific sections.
...@@ -440,179 +429,47 @@ rsync -azv <b>--exclude='kernels'</b> --delete --partial isisdist.astrogeology.u ...@@ -440,179 +429,47 @@ rsync -azv <b>--exclude='kernels'</b> --delete --partial isisdist.astrogeology.u
### Mission Specific Data Downloads ### Mission Specific Data Downloads
For versions of ISIS prior to ISIS 4.1.0, please cd into `$ISIS3DATA` instead of `$ISISDATA` and use `isis3data` instead of `isisdata` in all the below rsync commands. For versions of ISIS prior to ISIS 4.1.0, please use the `--legacy` flag
**Apollo Mission:** | Mission | Command |
| ------ | ------ |
cd $ISISDATA | Apollo 15 | `downloadIsisData apollo15 $ISISDATA` |
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/apollo15 . | Apollo 16 | `downloadIsisData apollo16 $ISISDATA` |
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/apollo16 . | Apollo 17 | `downloadIsisData apollo17 $ISISDATA` |
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/apollo17 . | Cassini | `downloadIsisData cassini $ISISDATA` |
| Chandrayaan 1 | `downloadIsisData chandrayaan1 $ISISDATA` |
| Clementine 1 | `downloadIsisData clementine1 $ISISDATA` |
**Cassini Mission:** | Dawn | `downloadIsisData dawn $ISISDATA` |
| ExoMars | `downloadIsisData tgo $ISISDATA` |
cd $ISISDATA | Galileo | `downloadIsisData galileo $ISISDATA` |
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/cassini . | Hayabusa 2 | `downloadIsisData hayabusa2 $ISISDATA` |
| Juno | `downloadIsisData juno $ISISDATA` |
| Kaguya | `downloadIsisData kaguya $ISISDATA` |
**Chandrayaan Mission:** | Lunar Orbiter | `downloadIsisData lo $ISISDATA` |
| Lunar Reconnaissance Orbiter | `downloadIsisData lro $ISISDATA` |
cd $ISISDATA | Mars Exploration Rover | `downloadIsisData mer $ISISDATA` |
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/chandrayaan1 . | Mariner10 | `downloadIsisData mariner10 $ISISDATA` |
| Messenger | `downloadIsisData messenger $ISISDATA` |
| Mars Express | `downloadIsisData mex $ISISDATA` |
**Clementine Mission:** | Mars Global Surveyor | `downloadIsisData mgs $ISISDATA` |
| Mars Reconnaissance Orbiter | `downloadIsisData mro $ISISDATA` |
cd $ISISDATA | Mars Odyssey | `downloadIsisData odyssey $ISISDATA` |
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/clementine1 . | Near | `downloadIsisData near $ISISDATA` |
| New Horizons | `downloadIsisData newhorizons $ISISDATA` |
| OSIRIS-REx | `downloadIsisData osirisrex $ISISDATA` |
**Dawn Mission:** | Rolo | `downloadIsisData rolo $ISISDATA` |
| Rosetta | `downloadIsisData rosetta $ISISDATA` |
cd $ISISDATA | Smart1 | `downloadIsisData smart1 $ISISDATA` |
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/dawn . | Viking 1 | `downloadIsisData viking1 $ISISDATA` |
| Viking 2 | `downloadIsisData viking2 $ISISDATA` |
| Voyager 1 | `downloadIsisData voyager1 $ISISDATA` |
**ExoMars Trace Gas Orbiter Mission:** | Voyager 2 | `downloadIsisData voyager2 $ISISDATA` |
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/tgo .
**Galileo Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/galileo .
**Hayabusa Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/hayabusa .
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/hayabusa2 .
**Juno Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/juno .
**Kaguya Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/kaguya .
**Lunar Orbiter Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/lo .
**Lunar Reconnaissance Orbiter Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/lro .
**Mars Exploration Rover Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mer .
**Mariner10 Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mariner10 .
**Messenger Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/messenger .
**Mars Express Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mex .
**Mars Global Surveyor Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mgs .
**Mars Reconnaissance Orbiter Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mro .
**Mars Odyssey Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/odyssey .
**Near Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/near .
**New Horizons Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/newhorizons .
**OSIRIS-REx Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/osirisrex .
**Rolo Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/rolo .
**Rosetta Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/rosetta .
**Smart1 Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/smart1 .
**Viking Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/viking1 .
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/viking2 .
**Voyager Mission:**
cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/voyager1 .
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/voyager2 .
### ISIS Test Data ### ISIS Test Data
ISIS is comprised of two types of tests, custom Makefile based tests, and GTest based tests. Those that are GTest based, make economical use of data that exists on the ISIS3 repo along with the source, so no special data is required to run those other than the ISIS data area. The Makefile tests depend on a separate source of data that consists of a few gigabytes of input and expected output data used for testing ISIS applications. The Makefile based tests use the ISISTESTDATA environment variable to know where the required data are located. The total size of this test data decreases as we work towards converting Makefile tests to GTests. ISIS is comprised of two types of tests, custom Makefile based tests, and GTest based tests. Those that are GTest based, make economical use of data that exists on the ISIS3 repo along with the source, so no special data is required to run those other than the ISIS data area. The Makefile tests depend on a separate source of data that consists of a few gigabytes of input and expected output data used for testing ISIS applications. The Makefile based tests use the `ISISTESTDATA` environment variable to know where the required data are located. The total size of this test data decreases as we work towards converting Makefile tests to GTests.
###How to download the ISIS test data with rclone ###How to download the ISIS test data with rclone
Test data is hosted using Amazon S3 storage buckets. We recommend using rclone to pull the data into a local directory. You can download rclone using their instructions (see: https://rclone.org/downloads/) or by using an anaconda environment (see: https://docs.anaconda.com/anaconda/install/). If you already have an anaconda environment up, install rclone with: conda install –c conda-forge rclone Test data is hosted using Amazon S3 storage buckets. We recommend using rclone to pull the data into a local directory. You can download rclone using their instructions (see: https://rclone.org/downloads/) or by using an anaconda environment (see: https://docs.anaconda.com/anaconda/install/). If you already have an anaconda environment up, install rclone with: `conda install –c conda-forge rclone`
Next, you will want to configure rclone using a default S3 configuration. See: https://rclone.org/s3/ for detailed information on how to configure S3, but for the purposes of downloading the ISIS3 test data, you simply run rclone config which will start an interactive menu. Press enter through it all except for these details: Next, you will want to configure rclone using a default S3 configuration. See: https://rclone.org/s3/ for detailed information on how to configure S3, but for the purposes of downloading the ISIS3 test data, you simply run rclone config which will start an interactive menu. Press enter through it all except for these details:
...@@ -645,4 +502,22 @@ If you are looking for ISIS2, please [refer to the ISIS 2 Installation Guide](ht ...@@ -645,4 +502,22 @@ If you are looking for ISIS2, please [refer to the ISIS 2 Installation Guide](ht
### How do I install ISIS3.5.2 or earlier? ### How do I install ISIS3.5.2 or earlier?
If you are looking for a version of ISIS prior to 3.6.0, please [refer to the Legacy ISIS3 Installation Guide](https://isis.astrogeology.usgs.gov/documents/LegacyInstallGuide/index.html) for instructions on downloading and installing ISIS, versions prior to 3.6.0. If you are looking for a version of ISIS prior to 3.6.0, please [refer to the Legacy ISIS3 Installation Guide](https://isis.astrogeology.usgs.gov/documents/LegacyInstallGuide/index.html) for instructions on downloading and installing ISIS, versions prior to 3.6.0
### How do I access the ISISDATA download script with ISIS 7.0.0 or earlier
You can download the script and config file from the repo:
```
# install rclone
conda install -c conda-forge rclone
# download the script and rclone config file
curl https://github.com/USGS-Astrogeology/ISIS3/raw/dev/isis/scripts/downloadIsisData.py -o downloadIsisData.py
curl https://github.com/USGS-Astrogeology/ISIS3/raw/dev/isis/config/rclone.conf -o rclone.conf
# run the script as normal, using --config to point to where you downloaded the config file
python downloadIsisData.py sync <mission> $ISISDATA --config rclone.conf
```
...@@ -49,6 +49,8 @@ dependencies: ...@@ -49,6 +49,8 @@ dependencies:
- pcl >= 1.10.0 - pcl >= 1.10.0
- protobuf<3.20 - protobuf<3.20
- python>=3.7.11 - python>=3.7.11
- pytest
- rclone
- qhull - qhull
- qt>=5.9.6 - qt>=5.9.6
- qwt - qwt
......
...@@ -609,6 +609,8 @@ install(FILES ${CMAKE_SOURCE_DIR}/../CHANGELOG.md DESTINATION ${CMAKE_INSTA ...@@ -609,6 +609,8 @@ install(FILES ${CMAKE_SOURCE_DIR}/../CHANGELOG.md DESTINATION ${CMAKE_INSTA
install(FILES ${CMAKE_BINARY_DIR}/version DESTINATION ${CMAKE_INSTALL_PREFIX}) install(FILES ${CMAKE_BINARY_DIR}/version DESTINATION ${CMAKE_INSTALL_PREFIX})
install(DIRECTORY ${CMAKE_SOURCE_DIR}/scripts DESTINATION ${CMAKE_INSTALL_PREFIX}) install(DIRECTORY ${CMAKE_SOURCE_DIR}/scripts DESTINATION ${CMAKE_INSTALL_PREFIX})
install(PROGRAMS ${CMAKE_BINARY_DIR}/lib/Camera.plugin DESTINATION ${CMAKE_INSTALL_PREFIX}/lib/) install(PROGRAMS ${CMAKE_BINARY_DIR}/lib/Camera.plugin DESTINATION ${CMAKE_INSTALL_PREFIX}/lib/)
install(PROGRAMS ${CMAKE_SOURCE_DIR}/scripts/downloadIsisData.py DESTINATION ${CMAKE_INSTALL_PREFIX}/bin/)
install(PROGRAMS ${CMAKE_SOURCE_DIR}/config/rclone.conf DESTINATION ${CMAKE_INSTALL_PREFIX}/etc/isis/)
# Trigger all post-install behavior. # Trigger all post-install behavior.
# - The only way to run commands post-install in CMake is to add a subdirectory at # - The only way to run commands post-install in CMake is to add a subdirectory at
......
[asc_s3]
type = s3
provider = AWS
region = us-west-2
location_constraint = us-west-2
[esa]
type = http
url = http://spiftp.esac.esa.int/
[naif]
type = http
url = http://naif.jpl.nasa.gov/
[jaxa]
type = http
url = http://www.darts.isas.jaxa.jp/
[apollo15_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/apollo15
[apollo16_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/apollo16
[apollo17_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/apollo17
[lro_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/lro
[tgo_naifKernels]
type = alias
remote = esa:/data/SPICE/ExoMars2016/kernels/
[tgo_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/tgo
[dawn_naifKernels]
type = alias
remote = naif:/pub/naif/DAWN/kernels
[dawn_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/dawn
[cassini_naifKernels]
type = alias
remote = naif:/pub/naif/CASSINI/kernels
[cassini_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/cassini
[hayabusa2_naifKernels]
type = alias
remote = jaxa:/pub/hayabusa2/spice_bundle/spice_kernels
[hayabusa2_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/hayabusa2
[juno_naifKernels]
type = alias
remote = naif:/pub/naif/pds/data/jno-j_e_ss-spice-6-v1.0/jnosp_1000/data/
[juno_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/juno
[odyssey_naifKernels]
type = alias
remote = naif:/pub/naif/M01/kernels
[odyssey_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/odyssey
[mro_naifKernels]
type = alias
remote = naif:/pub/naif/MRO/kernels
[mro_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/mro
[mex_naifKernels]
type = alias
remote = naif:/pub/naif/MEX/kernels
[mex_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/mex
[legacybase_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/legacy_base
[hayabusa_naifKernels]
type = alias
remote = jaxa:/pub/spice/HAYABUSA/kernels
[hayabusa_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/hayabusa
[chandrayaan1_naifKernels]
type = alias
remote = esa:/data/SPICE/CHANDRAYAAN-1/kernels
[chandrayaan1_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/chandrayaan1
[clementine1_naifKernels]
type = alias
remote = naif:/pub/naif/pds/data/clem1-l-spice-6-v1.0/clsp_1000/data/
[clementine1_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/clementine1
[kaguya_naifKernels]
type = alias
remote = jaxa:/pub/pds3/sln-l-spice-6-v1.0/slnsp_1000/data/
[kaguya_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/kaguya
[mariner10_naifKernels]
type = alias
remote = naif:/pub/naif/M10/kernels
[mariner10_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/mariner10
[messenger_naifKernels]
type = alias
remote = naif:/pub/naif/pds/data/mess-e_v_h-spice-6-v1.0/messsp_1000/data/
[messenger_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/messenger
[mgs_naifKernels]
type = alias
remote = naif:/pub/naif/pds/data/mgs-m-spice-6-v1.0/mgsp_1000/data/
[mgs_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/mgs
[near_naifKernels]
type = alias
remote = naif:/pub/naif/pds/data/near-a-spice-6-v1.0/nearsp_1000/data/
[near_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/near
[newhorizons_naifKernels]
type = alias
remote = naif:/pub/naif/pds/data/nh-j_p_ss-spice-6-v1.0/nhsp_1000/data/
[newhorizons_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/newhorizons
[osirisrex_naifKernels]
type = alias
remote = naif:/pub/naif/pds/pds4/orex/orex_spice/spice_kernels/
[osirisrex_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/osirisrex/
[rolo_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/rolo
[rosetta_naifKernels]
type = alias
remote = naif:/pub/naif/ROSETTA/kernels
[rosetta_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/rosetta
[smart1_naifKernels]
type = alias
remote = esa:/data/SPICE/SMART-1/kernels
[smart1_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/smart1
[viking1_naifKernels]
type = alias
remote = naif:/pub/naif/VIKING/kernels
[viking1_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/viking1
[viking2_naifKernels]
type = alias
remote = naif:/pub/naif/VIKING/kernels
[viking2_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/viking2
[voyager1_naifKernels]
type = alias
remote = naif:/pub/naif/VOYAGER/kernels
[voyager1_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/voyager1
[voyager2_naifKernels]
type = alias
remote = naif:/pub/naif/VOYAGER/kernels
[voyager2_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/voyager2
[base_usgs]
type = alias
remote = asc_s3:asc-isisdata/isis_data/base
import sys
sys.path.append('../scripts/')
import os
import pytest
from unittest import mock
from tempfile import TemporaryDirectory
from pathlib import Path
from downloadIsisData import rclone, create_rclone_arguments
class MockedPopen:
def __init__(self, args, **kwargs):
self.args = args
self.returncode = 0
def __enter__(self):
return self
def __exit__(self, exc_type, value, traceback):
pass
def communicate(self, input=None, timeout=None):
if self.args[0] == 'rclone':
stdout = "Success".encode("utf-8")
stderr = ''.encode("utf-8")
self.returncode = 0
else:
raise Exception()
return stdout, stderr
class MockedBustedPopen:
def __init__(self, args, **kwargs):
raise Exception("idk what happened")
@mock.patch('downloadIsisData.which', return_value=False)
def test_rclone_not_installed(mock_which):
with pytest.raises(ProcessLookupError, match="rclone is not installed"):
rclone("lsf", "test", extra_args=["-l", "-R", "--format", "p", "--files-only"], redirect_stdout=True, redirect_stderr=True)
@mock.patch("subprocess.Popen", MockedPopen)
def test_rclone():
res = rclone("lsf", "test", extra_args=["-l", "-R", "--format", "p", "--files-only"], redirect_stdout=True, redirect_stderr=True)
assert res["out"].decode() == "Success"
@mock.patch("subprocess.Popen", MockedBustedPopen)
def test_rclone_unknown_exception():
with pytest.raises(Exception, match="idk"):
res = rclone("lsf", "test", extra_args=["-l", "-R", "--format", "p", "--files-only"], redirect_stdout=True, redirect_stderr=True)
def test_create_rclone_args():
with TemporaryDirectory() as tdir:
dest = Path(tdir) / "test"
args = create_rclone_arguments(str(dest), "lro_naifKernels:", dry_run=False, ntransfers=100)
assert args == ['lro_naifKernels:', str(dest/"lro"/"kernels"), '--progress', '--checkers=100', '--transfers=100', '--track-renames', '--log-level=WARNING']
assert dest.exists()
#!/usr/bin/env python
from multiprocessing import ProcessError
from multiprocessing.dummy import Process
from pprint import pprint
import logging as log
import os
import json
import argparse
import subprocess
import tempfile
from shutil import which
from os import path
def find_conf():
from pathlib import Path
local_path = Path("rclone.conf")
install_path = Path(os.environ.get("CONDA_PREFIX")) / "etc" / "isis" / local_path
repo_path = Path(os.path.dirname(__file__)) / '..' / 'config' / 'rclone.conf'
if local_path.exists():
return str(local_path)
elif repo_path.exists():
return str(repo_path)
elif install_path.exists():
return str(install_path)
else:
return ""
# set log level to debug
def call_subprocess(command, redirect_stdout=True, redirect_stderr=False):
stdout = subprocess.PIPE if redirect_stdout else None
stderr = subprocess.PIPE if redirect_stderr else None
if not which(command[0]):
raise ProcessLookupError(f"{command[0]} is not installed.")
log.debug("Invoking : %s", " ".join(command))
with subprocess.Popen(
command,
stdout=stdout,
stderr=stderr) as proc:
(out, err) = proc.communicate()
if out:
log.debug("Process output: ")
log.debug(out.decode())
if err:
log.warning(err.decode("utf-8").replace("\\n", "\n"))
return {
"code": proc.returncode,
"out": out,
"error": err
}
def rclone(command, config=None, extra_args=[], redirect_stdout=True, redirect_stderr=False):
try:
if path.isfile(config):
log.debug(f"rclone() : using config file {config}")
# this is probably a config file on disk so pass it through
config_path = config
command_with_args = ["rclone", command, f'--config={config_path}', *extra_args]
log.debug("Invoking : %s", " ".join(command_with_args))
return call_subprocess(command_with_args, redirect_stdout, redirect_stderr)
else:
log.debug(f"rclone() : using config string {config}")
# most likely a config string
with tempfile.NamedTemporaryFile() as f:
config_path = path.realpath(f.name)
log.debug(f"USING CONFIG:\n{config}")
f.write(config.encode())
command_with_args = ["rclone", command, f"--config={config_path}", *extra_args]
return call_subprocess(command_with_args, redirect_stdout, redirect_stderr)
except ProcessLookupError as not_found_e:
log.error("Executable not found. %s", not_found_e)
raise ProcessLookupError("rclone is not installed, please install with: 'conda install -c conda-forge rclone'")
except Exception as generic_e:
message = f"Error running command. Reason: {generic_e}"
log.exception(message)
raise Exception(message)
def create_rclone_arguments(destination, mission_name, dry_run=False, ntransfers=10):
"""
Parameters
----------
destination str
path to location where files will be copied/downloaded too
set_of_pub : set(str)
set of kernels that will be downloaded
"""
log.debug(f"Creating RClone command for {mission_name}")
mission_dir_name, source_type = mission_name.replace(":", "").split("_")
log.debug(f"Mission_dir_name: {mission_dir_name}, source_type: {source_type}")
destination = os.path.join(destination, str(mission_dir_name).replace(":",""))
if source_type == "naifKernels":
destination = os.path.join(destination, "kernels")
if not os.path.exists(destination) and dry_run==False:
log.debug("{destination} does not exist, making directory")
print(destination)
os.makedirs(destination)
extra_args=[f"{mission_name}",f"{destination}", "--progress", f"--checkers={ntransfers}", f"--transfers={ntransfers}", "--track-renames", f"--log-level={log.getLevelName(log.getLogger().getEffectiveLevel())}"]
if dry_run:
extra_args.append("--dry-run")
log.debug(f"Args created: {extra_args}")
return extra_args
def main(mission, dest, cfg_path, dry_run, ntransfers):
"""
Parameters
----------
source str
path to location where files are being copied/downloaded from
dest : str
path to location where files will be copied/downloaded too
"""
log.info("---Script starting---")
log.debug(f"Using config: {cfg_path}")
result = rclone("listremotes", config=cfg_path)
config_sources = result.get('out').decode("utf-8").split("\n")
if config_sources == ['']:
log.error("Remote sources came back empty. Get more info by re-running with verbose flag.")
quit(-1)
log.debug(f"Remote Sources: {config_sources}")
supported_missions = {}
for source in config_sources:
parsed_name = source.split("_")
# If it is a mission, it should be in the format <mission_nam>_<source_type>
if len(parsed_name) == 2 and parsed_name[1] in ["usgs:", "naifKernels:"]:
remotes_mission_name = parsed_name[0]
supported_missions[remotes_mission_name] = supported_missions.get(remotes_mission_name, []) + [source]
log.debug(f"Supported missions:\n {supported_missions.keys()}")
log.debug(f"Complete Dictionary:\n {json.dumps(supported_missions, indent=2)}")
if not mission in supported_missions.keys():
raise LookupError(f"{mission} is not in the list of supported missions: {supported_missions.keys()}")
if mission == "legacybase":
args = create_rclone_arguments(dest, "legacybase_usgs:", dry_run, ntransfers)
rclone(command="sync", extra_args=args, config=cfg_path, redirect_stdout=False, redirect_stderr=False)
elif(mission.upper() == "ALL"):
supported_missions.pop("legacybase")
for mission, remotes in supported_missions.items():
for remote in remotes:
args = create_rclone_arguments(dest, remote, dry_run, ntransfers)
rclone(command="sync", extra_args=args, config=cfg_path, redirect_stdout=False, redirect_stderr=False)
else:
for remote in supported_missions[mission]:
args = create_rclone_arguments(dest, remote, dry_run, ntransfers)
rclone(command="sync", extra_args=args, config=cfg_path, redirect_stdout=False, redirect_stderr=False)
if __name__ == '__main__':
helpString = (
'''This will allow for a user to download isis data directly to their machine from the USGS S3 buckets as well as public end points\n\n'''
'''To use the download ISIS Data script you must supply 3 parameters with an optional 4th.\n\n'''
'''<rclone command> <Mission name> <destination to copy to> <--dry-run (optional)> \n'''
'''Example of how to run this program:\n'''
'''python downloadIsisData.py tgo ~/isisData/tgo\n\n'''
'''NOTE: if you would like to download the data for every mission supply the value ALL for the <Mission name> parameter''')
parser = argparse.ArgumentParser(description = helpString, formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument('mission', help='mission for files to be downloaded')
parser.add_argument('dest', help='the destination to download files from source')
parser.add_argument('--dry-run', help="run a dry run for rclone value should be a boolean", default=False, action='store_true')
parser.add_argument('-v', '--verbose', action='count', default=0)
parser.add_argument('-n', '--num-transfers', action='store', default=10)
parser.add_argument('--config', action='store', default=find_conf())
args = parser.parse_args()
log_kwargs = {
'format': '%(asctime)s %(levelname)-8s %(message)s',
'level': log.WARN,
'datefmt' : '%Y-%m-%d %H:%M:%S'
}
if args.verbose == 0:
log_kwargs['level'] = log.WARN
elif args.verbose == 1:
log_kwargs['level'] = log.INFO
elif args.verbose >= 2:
log_kwargs['level'] = log.DEBUG
log.basicConfig(**log_kwargs)
main(args.mission, args.dest, os.path.expanduser(args.config), args.dry_run, args.num_transfers)
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment