From 5e3f64f21ad98aaa05f517ba085b7d621c65a0ff Mon Sep 17 00:00:00 2001
From: jlaura <jlaura@usgs.gov>
Date: Mon, 25 Jul 2022 15:18:43 -0700
Subject: [PATCH] Update to #4937 - adds scripts for S3 download (#5013)

* rebase fix

* added pytest to Jenkinsfile

* updated Jenkinsfile

* made <dest> optional for lsf

* added configurable config

* typos

* simplified some logic

* cleaned up verbose flags a little

* cleaned up configs too

* removed extra '/'

* Fixed jenkins pytests path

* fixed a few bugs

* disabled other tests for now

* jenkinsfile path is off

* simplified things

* updated tests, installing now

* tweaking Jenkinsfile

* typo

* missed a spot

* mock, mocks, mo-mo-mo-mo mocks

* reverted jenkinsfile

* updates: readme for s3 data downloads

* Updates: remove unused sync and update filename

* Updates for comments

* fixes: typo in readme found in review

Co-authored-by: Gavin <nelsong916@gmail.com>
Co-authored-by: Kelvin Rodriguez <krodriguez@usgs.gov>
Co-authored-by: Jesse Mapel <jam826@nau.edu>
---
 Jenkinsfile                       |   8 +
 README.md                         | 259 ++++++++----------------------
 environment.yml                   |   2 +
 isis/CMakeLists.txt               |   2 +
 isis/config/rclone.conf           | 238 +++++++++++++++++++++++++++
 isis/pytests/test_IsisDownload.py |  63 ++++++++
 isis/scripts/downloadIsisData     | 208 ++++++++++++++++++++++++
 7 files changed, 588 insertions(+), 192 deletions(-)
 create mode 100644 isis/config/rclone.conf
 create mode 100644 isis/pytests/test_IsisDownload.py
 create mode 100755 isis/scripts/downloadIsisData

diff --git a/Jenkinsfile b/Jenkinsfile
index 8e3cf6349b..e99ec84369 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -93,6 +93,14 @@ for (lbl in labels) {
                                 errors.add(stageStatus)
                                 osFailed = true
                             }
+                            // pytests
+                            stageStatus = "Running pytests on ${label}"
+                            try {
+                                loginShell "cd $WORKSPACE/isis/pytests && pytest ."
+                            } catch(e) {
+                                errors.add(stageStatus)
+                                osFailed = true
+                            }
 
                             if (osFailed) {
                                 error "Failed on ${label}"
diff --git a/README.md b/README.md
index 6516caa3e3..ae6e1733d7 100644
--- a/README.md
+++ b/README.md
@@ -376,19 +376,8 @@ Under the root directory of the ISIS Data Area pointed to by the ISISDATA/ISIS3D
 
 ### Versions of the ISIS Data Area
 
-In ISIS version 4.1.0, several files previously stored in the data area closely associated with ISIS applications were moved into version control with the ISIS source code. Additionally, the environment variables used for the ISIS Data Area and the rsync location for the ISIS Data Area were also updated.
+In ISIS version 4.1.0 and later, several files previously stored in the data area closely associated with ISIS applications were moved into version control with the ISIS source code. To support the use of data in ISIS versions predating 4.1.0 the `downloadIsisData` application will need to download the data named `legacybase`. This is explained further in the [Full ISIS Data Download](README.md#Full-ISIS-Data-Download) section. 
 
-The correct environment variable names and rsync modules to use for the ISIS Data Area for each version of ISIS are summarized in the table below:
-
-ISIS version | ISIS Data Environment variable name | ISIS Data rsync module
----|---|---
-3.x | `$ISIS3DATA` | `isis3data`
-4.0.x | `$ISIS3DATA` | `isis3data`
-4.1.0 | `$ISISDATA` | `isisdata`
-
-The ISIS Data rsync module specifies where to rsync the data from and is the name used after the `::` in the rsync download commands below. For example, the rsync module is in bold in the following example rsync command:
-
-``rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::``**``isis3data``**``/data/`` .
 
 ### Size of the ISIS Data Area
 
@@ -396,17 +385,17 @@ If you plan to work with data from all missions, then the download will require
 
 ### Full ISIS Data Download
 
-The ISIS Data Area is hosted on rsync servers and not through conda channels like the ISIS binaries. This requires using the rsync command from within a terminal window within your Unix distribution, or from within WSL if running Windows 10.  Downloading all mission data requires over 520 GB of disk space. If you want to acquire only certain mission data [click here](#Mission-Specific-Data-Downloads). To download all ISIS data files, continue reading.
+> Warning if you are looking to download ISIS data via rsync the servers will be shutdown in November of 2022.
+the outdated rsync download information can be found [here](https://github.com/USGS-Astrogeology/ISIS3/wiki/Outdated-ISIS-Data-Information)
 
-To download all ISIS data, enter the following commands in the location where you want to install the ISIS Data Area, for versions of ISIS 4.1.0 and later:
 
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/ .
+The ISIS Data Area is hosted on a combination of AWS S3 buckets and public http servers e.g. NAIF, Jaxa, ESA and not through conda channels like the ISIS binaries. This requires using the `downloadIsisData.py` script from within a terminal window within your Unix distribution, or from within WSL if running Windows 10. Downloading all mission data requires over 520 GB of disk space. If you want to acquire only certain mission data [click here](#Mission-Specific-Data-Downloads). To download all ISIS data files, continue reading.
 
-For earlier versions, use:
+To download all ISIS data, use the following command:
 
-    cd $ISIS3DATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isis3data/data/ .
+    downloadIsisData  all $ISISDATA
+
+> Note: this applicaion takes in 3 parameters in the following order \<rclone command> \<mission> \<download destination>
 
 > Note: The above command downloads all ISIS data including the required base data area and all of the optional mission data areas.
 
@@ -414,11 +403,11 @@ For earlier versions, use:
 
 This data area contains data that is common between multiple missions such as DEMS and leap second kernels. As of ISIS 4.1, the base data area is no longer required to run many applications as data such as icons and templates has been moved into the binary distribution. If you plan to work with any applications that use camera models (e.g., cam2map, campt, qview), it is still recommended you download the base data area. To download the base data area run the following commands:
 
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/base .
+    downloadIsisData base $ISISDATA
 
-For versions of ISIS prior to ISIS 4.1.0, please use `isis3data` instead of `isisdata` in the above command.
+> Note: For accessing ISIS Data for versions of ISIS prior to ISIS 4.1.0, you must download the `legacybase` area and not the base area when using this application as shown below:
 
+    downloadIsisData legacybase $ISISDATA
 ### Partial Download of Mission Specific Data
 
 There are many missions supported by ISIS. If you are only working with a few missions then you can save disk space by downloading only those specific data areas. If you want to limit the download even further, read the next section about the SPICE Web Service. Otherwise [jump](#Mission-Specific-Data-Downloads) to the mission specific sections.
@@ -440,179 +429,47 @@ rsync -azv <b>--exclude='kernels'</b> --delete --partial isisdist.astrogeology.u
 
 ### Mission Specific Data Downloads
 
-For versions of ISIS prior to ISIS 4.1.0, please cd into `$ISIS3DATA` instead of `$ISISDATA` and use `isis3data` instead of `isisdata` in all the below rsync commands.
-
-**Apollo Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/apollo15 .
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/apollo16 .
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/apollo17 .
-
-
-**Cassini Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/cassini .
-
-
-**Chandrayaan Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/chandrayaan1 .
-
-
-**Clementine Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/clementine1 .
-
-
-**Dawn Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/dawn .
-
-
-**ExoMars Trace Gas Orbiter Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/tgo .
-
-
-**Galileo Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/galileo .
-
-
-**Hayabusa Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/hayabusa .
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/hayabusa2 .
-
-
-**Juno Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/juno .
-
-
-**Kaguya Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/kaguya .
-
-
-**Lunar Orbiter Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/lo .
-
-
-**Lunar Reconnaissance Orbiter Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/lro .
-
-
-**Mars Exploration Rover Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mer .
-
-
-**Mariner10 Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mariner10 .
-
-
-**Messenger Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/messenger .
-
-
-**Mars Express Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mex .
-
-
-**Mars Global Surveyor Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mgs .
-
-
-**Mars Reconnaissance Orbiter Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/mro .
-
-
-**Mars Odyssey Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/odyssey .
-
-
-**Near Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/near .
-
-
-**New Horizons Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/newhorizons .
-
-
-**OSIRIS-REx Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/osirisrex .
-
-
-**Rolo Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/rolo .
-
-
-**Rosetta Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/rosetta .
-
-
-**Smart1 Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/smart1 .
-
-
-**Viking Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/viking1 .
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/viking2 .
-
-
-**Voyager Mission:**
-
-    cd $ISISDATA
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/voyager1 .
-    rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/voyager2 .
+For versions of ISIS prior to ISIS 4.1.0, please use the `--legacy` flag
+
+| Mission | Command |
+| ------ | ------ |
+| Apollo 15 | `downloadIsisData apollo15 $ISISDATA` |
+| Apollo 16 | `downloadIsisData apollo16 $ISISDATA` |
+| Apollo 17 | `downloadIsisData apollo17 $ISISDATA` |
+| Cassini | `downloadIsisData cassini $ISISDATA` | 
+| Chandrayaan 1 | `downloadIsisData chandrayaan1 $ISISDATA` |
+| Clementine 1 | `downloadIsisData clementine1 $ISISDATA` |
+| Dawn | `downloadIsisData dawn $ISISDATA` |
+| ExoMars | `downloadIsisData tgo $ISISDATA` |
+| Galileo | `downloadIsisData galileo $ISISDATA` | 
+| Hayabusa 2 | `downloadIsisData hayabusa2 $ISISDATA` |
+| Juno | `downloadIsisData juno $ISISDATA` |
+| Kaguya | `downloadIsisData kaguya $ISISDATA` |
+| Lunar Orbiter | `downloadIsisData lo $ISISDATA` |
+| Lunar Reconnaissance Orbiter | `downloadIsisData lro $ISISDATA` |
+| Mars Exploration Rover  | `downloadIsisData mer $ISISDATA` |
+| Mariner10  | `downloadIsisData mariner10 $ISISDATA` |
+| Messenger | `downloadIsisData messenger $ISISDATA` |
+| Mars Express  | `downloadIsisData mex $ISISDATA` |
+| Mars Global Surveyor  | `downloadIsisData mgs $ISISDATA` |
+| Mars Reconnaissance Orbiter  | `downloadIsisData mro $ISISDATA` |
+| Mars Odyssey  | `downloadIsisData odyssey $ISISDATA` |
+| Near  | `downloadIsisData near $ISISDATA` |
+| New Horizons  | `downloadIsisData newhorizons $ISISDATA` |
+| OSIRIS-REx  | `downloadIsisData osirisrex $ISISDATA` |
+| Rolo  | `downloadIsisData rolo $ISISDATA` |
+| Rosetta  | `downloadIsisData rosetta $ISISDATA` |
+| Smart1  | `downloadIsisData smart1 $ISISDATA` |
+| Viking 1 | `downloadIsisData viking1 $ISISDATA` |
+| Viking 2 | `downloadIsisData viking2 $ISISDATA` |
+| Voyager 1 | `downloadIsisData voyager1 $ISISDATA` |
+| Voyager 2 | `downloadIsisData voyager2 $ISISDATA` |
 
 ### ISIS Test Data 
-ISIS is comprised of two types of tests, custom Makefile based tests, and GTest based tests. Those that are GTest based, make economical use of data that exists on the ISIS3 repo along with the source, so no special data is required to run those other than the ISIS data area. The Makefile tests depend on a separate source of data that consists of a few gigabytes of input and expected output data used for testing ISIS applications. The Makefile based tests use the ISISTESTDATA environment variable to know where the required data are located. The total size of this test data decreases as we work towards converting Makefile tests to GTests.  
+ISIS is comprised of two types of tests, custom Makefile based tests, and GTest based tests. Those that are GTest based, make economical use of data that exists on the ISIS3 repo along with the source, so no special data is required to run those other than the ISIS data area. The Makefile tests depend on a separate source of data that consists of a few gigabytes of input and expected output data used for testing ISIS applications. The Makefile based tests use the `ISISTESTDATA` environment variable to know where the required data are located. The total size of this test data decreases as we work towards converting Makefile tests to GTests.  
  
 ###How to download the ISIS test data with rclone  
-Test data is hosted using Amazon S3 storage buckets. We recommend using rclone to pull the data into a local directory. You can download rclone using their instructions (see: https://rclone.org/downloads/) or by using an anaconda environment (see: https://docs.anaconda.com/anaconda/install/). If you already have an anaconda environment up, install rclone with: conda install –c conda-forge rclone  
+Test data is hosted using Amazon S3 storage buckets. We recommend using rclone to pull the data into a local directory. You can download rclone using their instructions (see: https://rclone.org/downloads/) or by using an anaconda environment (see: https://docs.anaconda.com/anaconda/install/). If you already have an anaconda environment up, install rclone with: `conda install –c conda-forge rclone` 
 
 Next, you will want to configure rclone using a default S3 configuration. See: https://rclone.org/s3/ for detailed information on how to configure S3, but for the purposes of downloading the ISIS3 test data, you simply run rclone config which will start an interactive menu. Press enter through it all except for these details: 
 
@@ -645,4 +502,22 @@ If you are looking for ISIS2, please [refer to the ISIS 2 Installation Guide](ht
 
 ### How do I install ISIS3.5.2 or earlier?
 
-If you are looking for a version of ISIS prior to 3.6.0, please [refer to the Legacy ISIS3 Installation Guide](https://isis.astrogeology.usgs.gov/documents/LegacyInstallGuide/index.html) for instructions on downloading and installing ISIS, versions prior to 3.6.0.
+If you are looking for a version of ISIS prior to 3.6.0, please [refer to the Legacy ISIS3 Installation Guide](https://isis.astrogeology.usgs.gov/documents/LegacyInstallGuide/index.html) for instructions on downloading and installing ISIS, versions prior to 3.6.0
+
+### How do I access the ISISDATA download script with ISIS 7.0.0 or earlier 
+
+You can download the script and config file from the repo:
+
+```
+# install rclone 
+conda install -c conda-forge rclone
+
+# download the script and rclone config file
+curl https://github.com/USGS-Astrogeology/ISIS3/raw/dev/isis/scripts/downloadIsisData.py -o downloadIsisData.py
+
+curl https://github.com/USGS-Astrogeology/ISIS3/raw/dev/isis/config/rclone.conf -o rclone.conf
+
+# run the script as normal, using --config to point to where you downloaded the config file 
+python downloadIsisData.py sync <mission> $ISISDATA --config rclone.conf
+
+```
diff --git a/environment.yml b/environment.yml
index 08a4aab370..422fb015ff 100644
--- a/environment.yml
+++ b/environment.yml
@@ -49,6 +49,8 @@ dependencies:
   - pcl >= 1.10.0
   - protobuf<3.20
   - python>=3.7.11
+  - pytest
+  - rclone
   - qhull
   - qt>=5.9.6
   - qwt
diff --git a/isis/CMakeLists.txt b/isis/CMakeLists.txt
index f839ee1c60..71bf8c3a96 100644
--- a/isis/CMakeLists.txt
+++ b/isis/CMakeLists.txt
@@ -609,6 +609,8 @@ install(FILES     ${CMAKE_SOURCE_DIR}/../CHANGELOG.md DESTINATION  ${CMAKE_INSTA
 install(FILES     ${CMAKE_BINARY_DIR}/version         DESTINATION  ${CMAKE_INSTALL_PREFIX})
 install(DIRECTORY ${CMAKE_SOURCE_DIR}/scripts         DESTINATION  ${CMAKE_INSTALL_PREFIX})
 install(PROGRAMS ${CMAKE_BINARY_DIR}/lib/Camera.plugin DESTINATION ${CMAKE_INSTALL_PREFIX}/lib/)
+install(PROGRAMS ${CMAKE_SOURCE_DIR}/scripts/downloadIsisData.py DESTINATION ${CMAKE_INSTALL_PREFIX}/bin/)
+install(PROGRAMS ${CMAKE_SOURCE_DIR}/config/rclone.conf DESTINATION ${CMAKE_INSTALL_PREFIX}/etc/isis/)
 
 # Trigger all post-install behavior.
 # - The only way to run commands post-install in CMake is to add a subdirectory at
diff --git a/isis/config/rclone.conf b/isis/config/rclone.conf
new file mode 100644
index 0000000000..0b86075c79
--- /dev/null
+++ b/isis/config/rclone.conf
@@ -0,0 +1,238 @@
+[asc_s3]
+type = s3
+provider = AWS
+region = us-west-2
+location_constraint = us-west-2
+
+[esa]
+type = http
+url = http://spiftp.esac.esa.int/
+
+[naif]
+type = http
+url = http://naif.jpl.nasa.gov/
+
+[jaxa]
+type = http
+url = http://www.darts.isas.jaxa.jp/
+
+[apollo15_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/apollo15
+
+[apollo16_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/apollo16
+
+[apollo17_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/apollo17
+
+[lro_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/lro
+
+[tgo_naifKernels]
+type = alias
+remote = esa:/data/SPICE/ExoMars2016/kernels/
+
+[tgo_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/tgo
+
+[dawn_naifKernels]
+type = alias
+remote = naif:/pub/naif/DAWN/kernels
+
+[dawn_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/dawn
+
+[cassini_naifKernels]
+type = alias
+remote = naif:/pub/naif/CASSINI/kernels
+
+[cassini_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/cassini
+
+[hayabusa2_naifKernels]
+type = alias
+remote = jaxa:/pub/hayabusa2/spice_bundle/spice_kernels
+
+[hayabusa2_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/hayabusa2
+
+[juno_naifKernels]
+type = alias
+remote = naif:/pub/naif/pds/data/jno-j_e_ss-spice-6-v1.0/jnosp_1000/data/
+
+[juno_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/juno
+
+[odyssey_naifKernels]
+type = alias
+remote = naif:/pub/naif/M01/kernels
+
+[odyssey_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/odyssey
+
+[mro_naifKernels]
+type = alias
+remote = naif:/pub/naif/MRO/kernels
+
+[mro_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/mro
+
+[mex_naifKernels]
+type = alias
+remote = naif:/pub/naif/MEX/kernels
+
+[mex_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/mex
+
+[legacybase_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/legacy_base
+
+[hayabusa_naifKernels]
+type = alias
+remote = jaxa:/pub/spice/HAYABUSA/kernels
+
+[hayabusa_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/hayabusa
+
+[chandrayaan1_naifKernels]
+type = alias
+remote = esa:/data/SPICE/CHANDRAYAAN-1/kernels
+
+[chandrayaan1_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/chandrayaan1
+
+[clementine1_naifKernels]
+type = alias
+remote = naif:/pub/naif/pds/data/clem1-l-spice-6-v1.0/clsp_1000/data/
+
+[clementine1_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/clementine1
+
+[kaguya_naifKernels]
+type = alias
+remote = jaxa:/pub/pds3/sln-l-spice-6-v1.0/slnsp_1000/data/
+
+[kaguya_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/kaguya
+
+[mariner10_naifKernels]
+type = alias
+remote = naif:/pub/naif/M10/kernels
+
+[mariner10_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/mariner10
+
+[messenger_naifKernels]
+type = alias
+remote = naif:/pub/naif/pds/data/mess-e_v_h-spice-6-v1.0/messsp_1000/data/
+
+[messenger_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/messenger
+
+[mgs_naifKernels]
+type = alias
+remote = naif:/pub/naif/pds/data/mgs-m-spice-6-v1.0/mgsp_1000/data/
+
+[mgs_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/mgs
+
+[near_naifKernels]
+type = alias
+remote = naif:/pub/naif/pds/data/near-a-spice-6-v1.0/nearsp_1000/data/
+
+[near_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/near
+
+[newhorizons_naifKernels]
+type = alias
+remote = naif:/pub/naif/pds/data/nh-j_p_ss-spice-6-v1.0/nhsp_1000/data/
+
+[newhorizons_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/newhorizons
+
+[osirisrex_naifKernels]
+type = alias
+remote = naif:/pub/naif/pds/pds4/orex/orex_spice/spice_kernels/
+
+[osirisrex_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/osirisrex/
+
+[rolo_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/rolo
+
+[rosetta_naifKernels]
+type = alias
+remote = naif:/pub/naif/ROSETTA/kernels
+
+[rosetta_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/rosetta
+
+[smart1_naifKernels]
+type = alias
+remote = esa:/data/SPICE/SMART-1/kernels
+
+[smart1_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/smart1
+
+[viking1_naifKernels]
+type = alias
+remote = naif:/pub/naif/VIKING/kernels
+
+[viking1_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/viking1
+
+[viking2_naifKernels]
+type = alias
+remote = naif:/pub/naif/VIKING/kernels
+
+[viking2_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/viking2
+
+[voyager1_naifKernels]
+type = alias
+remote = naif:/pub/naif/VOYAGER/kernels
+
+[voyager1_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/voyager1
+
+[voyager2_naifKernels]
+type = alias
+remote = naif:/pub/naif/VOYAGER/kernels
+
+[voyager2_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/voyager2
+
+[base_usgs]
+type = alias
+remote = asc_s3:asc-isisdata/isis_data/base
+
diff --git a/isis/pytests/test_IsisDownload.py b/isis/pytests/test_IsisDownload.py
new file mode 100644
index 0000000000..548b39be41
--- /dev/null
+++ b/isis/pytests/test_IsisDownload.py
@@ -0,0 +1,63 @@
+
+import sys
+sys.path.append('../scripts/')
+
+import os
+import pytest 
+from unittest import mock
+from tempfile import TemporaryDirectory
+from pathlib import Path
+
+from downloadIsisData import rclone, create_rclone_arguments
+
+class MockedPopen:
+    def __init__(self, args, **kwargs):
+        self.args = args
+        self.returncode = 0
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, value, traceback):
+        pass
+
+    def communicate(self, input=None, timeout=None): 
+        if self.args[0] == 'rclone':
+            stdout = "Success".encode("utf-8")
+            stderr = ''.encode("utf-8")
+            self.returncode = 0
+        else:
+            raise Exception()
+
+        return stdout, stderr
+
+
+class MockedBustedPopen:
+    def __init__(self, args, **kwargs):
+        raise Exception("idk what happened")
+
+@mock.patch('downloadIsisData.which', return_value=False)
+def test_rclone_not_installed(mock_which):
+    with pytest.raises(ProcessLookupError, match="rclone is not installed"):
+        rclone("lsf", "test", extra_args=["-l", "-R", "--format", "p", "--files-only"], redirect_stdout=True, redirect_stderr=True)
+
+
+@mock.patch("subprocess.Popen", MockedPopen)
+def test_rclone():
+    res = rclone("lsf", "test", extra_args=["-l", "-R", "--format", "p", "--files-only"], redirect_stdout=True, redirect_stderr=True)
+    assert res["out"].decode() == "Success"
+
+@mock.patch("subprocess.Popen", MockedBustedPopen)
+def test_rclone_unknown_exception():
+    with pytest.raises(Exception, match="idk"):
+        res = rclone("lsf", "test", extra_args=["-l", "-R", "--format", "p", "--files-only"], redirect_stdout=True, redirect_stderr=True)
+
+
+def test_create_rclone_args():
+    with TemporaryDirectory() as tdir: 
+        dest = Path(tdir) / "test"
+        args = create_rclone_arguments(str(dest), "lro_naifKernels:", dry_run=False, ntransfers=100)
+        assert args == ['lro_naifKernels:', str(dest/"lro"/"kernels"), '--progress', '--checkers=100', '--transfers=100', '--track-renames', '--log-level=WARNING']
+        assert dest.exists()
+
+
diff --git a/isis/scripts/downloadIsisData b/isis/scripts/downloadIsisData
new file mode 100755
index 0000000000..b3a4ca1ee2
--- /dev/null
+++ b/isis/scripts/downloadIsisData
@@ -0,0 +1,208 @@
+#!/usr/bin/env python
+
+from multiprocessing import ProcessError
+from multiprocessing.dummy import Process
+from pprint import pprint
+import logging as log
+import os
+import json
+import argparse
+import subprocess
+import tempfile
+from shutil import which
+from os import path
+
+
+def find_conf():
+    from pathlib import Path
+    local_path = Path("rclone.conf")
+    install_path = Path(os.environ.get("CONDA_PREFIX")) / "etc" / "isis" / local_path
+    repo_path = Path(os.path.dirname(__file__)) / '..' / 'config' / 'rclone.conf' 
+    
+    if local_path.exists():
+        return str(local_path) 
+    elif repo_path.exists():
+        return str(repo_path)
+    elif install_path.exists():
+        return str(install_path)
+    else:
+        return ""
+
+
+# set log level to debug
+def call_subprocess(command, redirect_stdout=True, redirect_stderr=False):
+    stdout = subprocess.PIPE if redirect_stdout else None
+    stderr = subprocess.PIPE if redirect_stderr else None
+
+    if not which(command[0]):
+        raise ProcessLookupError(f"{command[0]} is not installed.")
+
+    log.debug("Invoking : %s", " ".join(command))
+
+    with subprocess.Popen(
+            command,
+            stdout=stdout,
+            stderr=stderr) as proc:
+        (out, err) = proc.communicate()
+
+        if out:
+            log.debug("Process output: ")
+            log.debug(out.decode())
+        if err:
+            log.warning(err.decode("utf-8").replace("\\n", "\n"))
+
+        return {
+            "code": proc.returncode,
+            "out": out,
+            "error": err
+        }
+
+
+def rclone(command, config=None, extra_args=[], redirect_stdout=True, redirect_stderr=False):
+    try:
+        if path.isfile(config):
+            log.debug(f"rclone() : using config file {config}")
+
+            # this is probably a config file on disk so pass it through
+            config_path = config
+            command_with_args = ["rclone", command, f'--config={config_path}', *extra_args]
+            log.debug("Invoking : %s", " ".join(command_with_args))
+            return call_subprocess(command_with_args, redirect_stdout, redirect_stderr)
+        else:
+            log.debug(f"rclone() : using config string {config}")
+
+            # most likely a config string
+            with tempfile.NamedTemporaryFile() as f:
+                config_path = path.realpath(f.name)
+
+                log.debug(f"USING CONFIG:\n{config}")
+
+                f.write(config.encode())
+                command_with_args = ["rclone", command, f"--config={config_path}", *extra_args]
+                return call_subprocess(command_with_args, redirect_stdout, redirect_stderr)
+    except ProcessLookupError as not_found_e:
+        log.error("Executable not found. %s", not_found_e)
+        raise ProcessLookupError("rclone is not installed, please install with: 'conda install -c conda-forge rclone'")
+    except Exception as generic_e:
+        message = f"Error running command. Reason: {generic_e}"
+        log.exception(message)
+        raise Exception(message)
+
+
+def create_rclone_arguments(destination, mission_name, dry_run=False, ntransfers=10):
+    """
+    Parameters
+    ----------
+
+    destination str
+            path to location where files will be copied/downloaded too
+
+    set_of_pub : set(str)
+            set of kernels that will be downloaded
+    """
+    log.debug(f"Creating RClone command for {mission_name}")
+    mission_dir_name, source_type = mission_name.replace(":", "").split("_")
+
+    log.debug(f"Mission_dir_name: {mission_dir_name}, source_type: {source_type}")
+
+    destination = os.path.join(destination, str(mission_dir_name).replace(":",""))
+    if source_type == "naifKernels":
+        destination = os.path.join(destination, "kernels")
+
+    if not os.path.exists(destination) and dry_run==False:
+        log.debug("{destination} does not exist, making directory")
+        print(destination)
+        os.makedirs(destination)
+
+    extra_args=[f"{mission_name}",f"{destination}", "--progress", f"--checkers={ntransfers}", f"--transfers={ntransfers}", "--track-renames", f"--log-level={log.getLevelName(log.getLogger().getEffectiveLevel())}"]
+    
+    if dry_run:
+        extra_args.append("--dry-run")
+
+    log.debug(f"Args created: {extra_args}")
+    return extra_args
+
+
+def main(mission, dest, cfg_path, dry_run, ntransfers):
+    """
+    Parameters
+    ----------
+
+    source str
+            path to location where files are being copied/downloaded from
+
+    dest : str
+               path to location where files will be copied/downloaded too
+    """
+    log.info("---Script starting---")
+
+    log.debug(f"Using config: {cfg_path}")
+    result = rclone("listremotes", config=cfg_path)
+    config_sources = result.get('out').decode("utf-8").split("\n")
+    if config_sources == ['']: 
+        log.error("Remote sources came back empty. Get more info by re-running with verbose flag.")
+        quit(-1)
+    log.debug(f"Remote Sources: {config_sources}")
+
+    supported_missions = {}
+    for source in config_sources: 
+        parsed_name = source.split("_")
+        # If it is a mission, it should be in the format <mission_nam>_<source_type>
+        if len(parsed_name) == 2 and parsed_name[1] in ["usgs:", "naifKernels:"]: 
+            remotes_mission_name = parsed_name[0]
+            supported_missions[remotes_mission_name] = supported_missions.get(remotes_mission_name, []) + [source]
+
+    log.debug(f"Supported missions:\n {supported_missions.keys()}")
+    log.debug(f"Complete Dictionary:\n {json.dumps(supported_missions, indent=2)}")
+
+    if not mission in supported_missions.keys():
+        raise LookupError(f"{mission} is not in the list of supported missions: {supported_missions.keys()}")
+ 
+    if mission == "legacybase":
+        args = create_rclone_arguments(dest, "legacybase_usgs:", dry_run, ntransfers)
+        rclone(command="sync", extra_args=args, config=cfg_path, redirect_stdout=False, redirect_stderr=False)
+    elif(mission.upper() == "ALL"):
+        supported_missions.pop("legacybase")
+        for mission, remotes in supported_missions.items():
+            for remote in remotes:
+                args = create_rclone_arguments(dest, remote, dry_run, ntransfers)
+                rclone(command="sync", extra_args=args, config=cfg_path, redirect_stdout=False, redirect_stderr=False)
+    else:
+        for remote in supported_missions[mission]:
+            args = create_rclone_arguments(dest, remote, dry_run, ntransfers)
+            rclone(command="sync", extra_args=args, config=cfg_path, redirect_stdout=False, redirect_stderr=False) 
+
+
+if __name__ == '__main__': 
+    helpString = (
+    '''This will allow for a user to download isis data directly to their machine from the USGS S3 buckets as well as public end points\n\n'''
+    '''To use the download ISIS Data script you must supply 3 parameters with an optional 4th.\n\n'''
+    '''<rclone command> <Mission name> <destination to copy to> <--dry-run (optional)> \n'''
+    '''Example of how to run this program:\n'''
+    '''python downloadIsisData.py tgo ~/isisData/tgo\n\n'''
+    '''NOTE: if you would like to download the data for every mission supply the value ALL for the <Mission name> parameter''')
+    parser = argparse.ArgumentParser(description = helpString, formatter_class=argparse.RawTextHelpFormatter)
+    parser.add_argument('mission', help='mission for files to be downloaded')
+    parser.add_argument('dest', help='the destination to download files from source')
+    parser.add_argument('--dry-run', help="run a dry run for rclone value should be a boolean", default=False, action='store_true')
+    parser.add_argument('-v', '--verbose', action='count', default=0)
+    parser.add_argument('-n', '--num-transfers', action='store', default=10)
+    parser.add_argument('--config', action='store', default=find_conf())
+    args = parser.parse_args()
+
+    log_kwargs = {
+      'format': '%(asctime)s %(levelname)-8s %(message)s',
+      'level': log.WARN,
+      'datefmt' : '%Y-%m-%d %H:%M:%S'
+    }
+
+    if args.verbose == 0:
+        log_kwargs['level'] = log.WARN
+    elif args.verbose == 1:
+        log_kwargs['level'] = log.INFO
+    elif args.verbose >= 2:
+        log_kwargs['level'] = log.DEBUG
+
+    log.basicConfig(**log_kwargs)
+
+    main(args.mission, args.dest, os.path.expanduser(args.config), args.dry_run, args.num_transfers)
-- 
GitLab