Skip to content
Snippets Groups Projects
Commit 921632f1 authored by Thorsten Alteholz's avatar Thorsten Alteholz
Browse files

initial commit of Python3 port

parent e213bd56
Branches
No related tags found
No related merge requests found
Version 0.4.0
general:
port to python3
Version 0.3.1
general:
Disable by default datacenter
plot:
Change default plot size
Use tight_layout
Improve detection of AM/PM dates
Allow to plot only the 2nd plot (NSB vs datetime)
plot.py now works also as a standalone tool (with user provided data file path).
Version 0.3.0
general:
Added datacenter support
Version 0.2.2
general:
Adopt v1.0 of the standard format
(including the filename for the daily data and plots)
read:
put the rx,cx and ix data in the header
plot:
Change de Serial number label
Version 0.2.1
read:
Print the errors in make_plot call on screen.
plot:
Only print the PM/AM/Moon labels on one panel.
Print the SQM serial number.
Version 0.2.0
general:
Deep changes to make the program more modular.
The program now can be packaged as a single .exe file with PyInstaller.
The program can also be packaged for Linux systems.
read:
Try to use the fixed device address before looking for it automatically
this should allow the use of multiple devices in a single computer.
plot:
Code cleanup.
Use local date/time in plots.
Write statistics file.
Use pyephem to calculate the moon phase (more accurate).
Show the Moon max altitude (transit altitude or culmination).
Plot the astronomical twilights.
Object Oriented programming.
email:
Now the program can be distributed without email module.
Version 0.1.X
read:
Variables moved to config file.
Clean-up of the code.
Improve device reset.
New read software. OO programing.
plot:
Variables moved to config file.
Renamed from plot_sqmle.py to pysqm_plot.py
Make the code and linebreaks less ugly
Fixed axis.
Moon phase plot.
email:
Renamed from email_sqmle.py to pysqm_email.py
Version 0.0.X
First version.
include LICENSE.txt
PySQM
=====
PySQM is a multi-platform, open-source software designed to read and
plot data from Unihedron SQM-LE and SQM-LU photometers, giving as
an output files with the 'International Dark Sky Association (IDA)
NSBM Community Standards for Reporting Skyglow Observations' format
(http://www.darksky.org/night-sky-conservation/248).
PySQM is distributed under GNU GPL, either version 3 of the License,
or (at your option) any later version. See the file LICENSE.txt for details.
This software has been developed by Mireia Nievas <mnievas@ucm.es> with
the invaluable help of:
- Sergio Pascual (UCM)
- Jaime Zamorano (UCM)
- Laura Barbas (OAN)
- Pablo de Vicente (OAN)
The initial port to Python3 has been done by Anthony Tekatch (Unihedron).
SETUP
=====
After downloading the software, you need to modify the file config.py.
In this file you will find several variables that need to be configured
to match your hardware settings. For example:
- Location of the observatory (geographical coordinates).
- Device identifier.
- Device address (either IP address for SQM-LE or COM/ttyUSB port).
- Location of the data files.
- Axis limits for the plot.
Remember that python (2.7) syntax is mandatory in this file
HOW TO USE THE SOFTWARE
=======================
After configuring the software, make sure you are in the parent directory were
the README, LICENSE and MANIFEST files are located
> ls
LICENSE.txt MANIFEST.in README.txt pysqm config.py setup.py
And then run the software.
> python -m pysqm
The program should find your SQM device and the data adquisition.will start
(if it's night-time).
In some systems, where python3 is the default version of python, you need
to specify python2 as the interpreter to use. This is done usually running
it as:
> python2 -m pysqm
or
> python2.7 -m pysqm
Note: running the setup.py script is neither tested nor required.
The program is currently being redesigned as a normal python package, but at
present no setup is required.
HOW IT WORKS
============
In a first step, the program tries to connect to the SQM photometer and takes
some 'tests' measures (metadata/information, calibration and data) to check
that the device is working as expected.
After that, the program begins data acdquisition. In each iteration, it checks
whether it is night-time. In that case new data is taken.
Each N measurements, the main program calls a plotting function to generate
a graphical representation of the current nightly data.
PySQM known issues
==================
Non-ASCII characters are not supported in the config.py file. Please, avoid using 'ñ', accented vowels, etc.
In headless systems, such as the Raspberry PI, if you run the program without X, you may suffer from the following fatal error when the program tries to generate the plot:
This application failed to start because it could not find or load the Qt platform plugin “xcb”.
Available platform plugins are: eglfs, kms, linuxfb, minimal, minimalegl, offscreen, xcb.
Reinstalling the application may fix this problem. Aborted (core dumped)
In order to avoid this problem, you need to create (or modify if the file exists) in your HOME directory the following file:
.config/matplotlib/matplotlibrc
You just need to set the matplotlib backend to Agg:
backend : Agg
Save the changes and exit. Now, PySQM should make the plots without issues. You may need to restart PySQM to apply the changes.
Path to EXE files (windows only):
https://www.dropbox.com/s/xlbr6ktk8spjsse/PySQM.exe?dl=0
CHANGELOG
=========
v0.3:
Added datacenter option (optional, disabled by default)
v0.2:
...
v0.1:
...
config.py 0 → 100644
#!/usr/bin/env python
'''
PySQM configuration File.
____________________________
Copyright (c) Mireia Nievas <mnievas[at]ucm[dot]es>
This file is part of PySQM.
PySQM is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
PySQM is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with PySQM. If not, see <http://www.gnu.org/licenses/>.
____________________________
Notes:
You may need to change the following variables to match your
observatory coordinates, instrumental properties, etc.
Python syntax is mandatory.
____________________________
'''
'''
-------------
SITE location
-------------
'''
_observatory_name = 'GURUGU'
_observatory_latitude = 40.447862
_observatory_longitude = -3.364992
_observatory_altitude = 680
_observatory_horizon = 10 # If Sun is below this altitude, the program will take data
_device_shorttype = 'SQM' # Device STR in the file
_device_type = 'SQM_LU' # Device type in the Header
_device_id = _device_type + '-' + _observatory_name # Long Device lame
_device_locationname = 'Villalbilla/Spain - Observatorio GURUGU' # Device location in the world
_data_supplier = 'Mireia Nievas / Universidad Complutense de Madrid' # Data supplier (contact)
_device_addr = '/dev/ttyUSB1' # Default IP address of the ethernet device (if not automatically found)
_measures_to_promediate = 1 # Take the mean of N measures
_delay_between_measures = 2 # Delay between two measures. In seconds.
_cache_measures = 1 # Get X measures before writing on screen/file
_plot_each = 1 # Call the plot function each X measures.
_use_mysql = False # Set to True if you want to store data on a MySQL db.
_mysql_host = None # Host (ip:port / localhost) of the MySQL engine.
_mysql_user = None # User with write permission on the db.
_mysql_pass = None # Password for that user.
_mysql_database = None # Name of the database.
_mysql_dbtable = None # Name of the table
_mysql_port = None # Port of the MySQL server.
_local_timezone = +1 # UTC+1
_computer_timezone = +0 # UTC
_offset_calibration = -0.11 # magnitude = read_magnitude + offset
_reboot_on_connlost = False # Reboot if we loose connection
# Monthly (permanent) data
monthly_data_directory = "/tmp/sqm_gurugu/"
# Daily (permanent) data
daily_data_directory = monthly_data_directory+"/datos_diarios/"
limits_nsb = [20.0,16.5] # Limits in Y-axis
# Daily (permanent) graph
daily_graph_directory = monthly_data_directory+"/graficos_diarios/"
# Current data, deleted each day.
current_data_directory = monthly_data_directory
# Current graph, deleted each day.
current_graph_directory = monthly_data_directory
# Summary with statistics for the night
summary_data_directory = monthly_data_directory
'''
----------------------------
PySQM data center (OPTIONAL)
----------------------------
'''
# Send the data to the data center
_send_to_datacenter = False
'''
Ploting options
'''
full_plot = True
limits_nsb = [20.0,16.5] # Limits in Y-axis
limits_time = [17,9] # Hours
limits_sunalt = [-80,5] # Degrees
'''
Email options
'''
_send_data_by_email = False
#!/usr/bin/env python
'''
PySQM __init__ code
____________________________
Copyright (c) Mireia Nievas <mnievas[at]ucm[dot]es>
This file is part of PySQM.
PySQM is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
PySQM is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with PySQM. If not, see <http://www.gnu.org/licenses/>.
'''
__author__ = "Mireia Nievas"
__copyright__ = "Copyright (c) 2014 Mireia Nievas"
__credits__ = [\
"Mireia Nievas @ UCM",\
"Jaime Zamorano @ UCM",\
"Laura Barbas @ OAN",\
"Pablo de Vicente @ OAN"\
"Anthony Tekatch @ Unihedron "\
]
__license__ = "GNU GPL v3"
__shortname__ = "PySQM"
__longname__ = "Python Sky Quality Meter pipeline"
__version__ = "0.4.0"
__maintainer__ = "Thorsten Alteholz"
__email__ = "python[at]alteholz[dot]de"
__status__ = "Development" # "Prototype", "Development", or "Production"
from types import ModuleType
import sys
#!/usr/bin/env python
'''
PySQM __main__ code
____________________________
Copyright (c) Mireia Nievas <mnievas[at]ucm[dot]es>
This file is part of PySQM.
PySQM is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
PySQM is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with PySQM. If not, see <http://www.gnu.org/licenses/>.
____________________________
'''
#from types import ModuleType
#import sys
import pysqm.main as main
while(1==1):
# Loop forever to make sure the program does not die.
try:
main.loop()
except Exception as e:
print('')
print('FATAL ERROR while running the main loop !!')
print('Error was:')
print(e)
print('Trying to restart')
print('')
#!/usr/bin/env python
'''
PySQM common code
____________________________
Copyright (c) Mireia Nievas <mnievas[at]ucm[dot]es>
This file is part of PySQM.
PySQM is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
PySQM is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with PySQM. If not, see <http://www.gnu.org/licenses/>.
____________________________
'''
import math
import ephem
import datetime
# Read the config variables from config.py
import pysqm.settings as settings
config = settings.GlobalConfig.config
def define_ephem_observatory():
''' Define the Observatory in Pyephem '''
OBS = ephem.Observer()
OBS.lat = config._observatory_latitude*ephem.pi/180
OBS.lon = config._observatory_longitude*ephem.pi/180
OBS.elev = config._observatory_altitude
return(OBS)
def remove_linebreaks(data):
# Remove line breaks from data
data = data.replace('\r\n','')
data = data.replace('\r','')
data = data.replace('\n','')
return(data)
def format_value(data,remove_str=' '):
# Remove string and spaces from data
data = remove_linebreaks(data)
data = data.replace(remove_str,'')
data = data.replace(' ','')
return(data)
def format_value_list(data,remove_str=' '):
# Remove string and spaces from data array/list
data = [format_value(line,remove_str).split(';') for line in data]
return(data)
def set_decimals(number,dec=3):
str_number = str(number)
int_,dec_ = str_number.split('.')
while len(dec_)<=dec:
dec_=dec_+'0'
return(int_+'.'+dec_[:dec])
class observatory(object):
def read_datetime(self):
# Get UTC datetime from the computer.
utc_dt = datetime.datetime.utcnow()
#utc_dt = datetime.datetime.now() - datetime.timedelta(hours=config._computer_timezone)
#time.localtime(); daylight_saving=_.tm_isdst>0
return(utc_dt)
def local_datetime(self,utc_dt):
# Get Local datetime from the computer, without daylight saving.
return(utc_dt + datetime.timedelta(hours=config._local_timezone))
def calculate_sun_altitude(self,OBS,timeutc):
# Calculate Sun altitude
OBS.date = ephem.date(timeutc)
Sun = ephem.Sun(OBS)
return(Sun.alt)
def next_sunset(self,OBS):
# Next sunset calculation
previous_horizon = OBS.horizon
OBS.horizon = str(config._observatory_horizon)
next_setting = OBS.next_setting(ephem.Sun()).datetime()
next_setting = next_setting.strftime("%Y-%m-%d %H:%M:%S")
OBS.horizon = previous_horizon
return(next_setting)
def is_nighttime(self,OBS):
# Is nightime (sun below a given altitude)
timeutc = self.read_datetime()
if self.calculate_sun_altitude(OBS,timeutc)*180./math.pi>config._observatory_horizon:
return False
else:
return True
RAWHeaderContent = '''# Definition of the community standard for skyglow observations 1.0
# URL: http://www.darksky.org/NSBM/sdf1.0.pdf
# Number of header lines: 35
# This data is released under the following license: ODbL 1.0 http://opendatacommons.org/licenses/odbl/summary/
# Device type: $DEVICE_TYPE
# Instrument ID: $DEVICE_ID
# Data supplier: $DATA_SUPPLIER
# Location name: $LOCATION_NAME
# Position: $OBSLAT, $OBSLON, $OBSALT
# Local timezone: $TIMEZONE
# Time Synchronization: NTP
# Moving / Stationary position: STATIONARY
# Moving / Fixed look direction: FIXED
# Number of channels: 1
# Filters per channel: HOYA CM-500
# Measurement direction per channel: 0., 0.
# Field of view: 20
# Number of fields per line: 6
# SQM serial number: $SERIAL_NUMBER
# SQM firmware version: $FEATURE_NUMBER
# SQM cover offset value: $OFFSET
# SQM readout test ix: $IXREADOUT
# SQM readout test rx: $RXREADOUT
# SQM readout test cx: $CXREADOUT
# Comment:
# Comment:
# Comment:
# Comment:
# Comment: Capture program: PySQM
# blank line 30
# blank line 31
# blank line 32
# UTC Date & Time, Local Date & Time, Temperature, Counts, Frequency, MSAS
# YYYY-MM-DDTHH:mm:ss.fff;YYYY-MM-DDTHH:mm:ss.fff;Celsius;number;Hz;mag/arcsec^2
# END OF HEADER
'''
#!/usr/bin/env python
'''
PySQM main program
____________________________
Copyright (c) Mireia Nievas <mnievas[at]ucm[dot]es>
This file is part of PySQM.
PySQM is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
PySQM is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with PySQM. If not, see <http://www.gnu.org/licenses/>.
____________________________
'''
import os,sys
import time
import datetime
import argparse
'''
Read input arguments (if any)
'''
import pysqm.settings as settings
InputArguments = settings.ArgParser()
configfilename = InputArguments.config
# Load config contents into GlobalConfig
settings.GlobalConfig.read_config_file(configfilename)
# Get the actual config
config = settings.GlobalConfig.config
### Load now the rest of the modules
from pysqm.read import *
import pysqm.plot
'''
This import section is only for software build purposes.
Dont worry if some of these are missing in your setup.
'''
def relaxed_import(themodule):
try: exec('import '+str(themodule))
except: pass
relaxed_import('socket')
relaxed_import('serial')
relaxed_import('_mysql')
relaxed_import('pysqm.email')
'''
Conditional imports
'''
# If the old format (SQM_LE/SQM_LU) is used, replace _ with -
config._device_type = config._device_type.replace('_','-')
if config._device_type == 'SQM-LE':
import socket
elif config._device_type == 'SQM-LU':
import serial
if config._use_mysql == True:
import _mysql
# Create directories if needed
for directory in [config.monthly_data_directory,config.daily_data_directory,config.current_data_directory]:
if not os.path.exists(directory):
os.makedirs(directory)
'''
Select the device to be used based on user input
and start the measures
'''
if config._device_type=='SQM-LU':
mydevice = SQMLU()
elif config._device_type=='SQM-LE':
mydevice = SQMLE()
else:
print(('ERROR. Unknown device type '+str(config._device_type)))
exit(0)
def loop():
'''
Ephem is used to calculate moon position (if above horizon)
and to determine start-end times of the measures
'''
observ = define_ephem_observatory()
niter = 0
DaytimePrint=True
print('Starting readings ...')
while 1<2:
''' The programs works as a daemon '''
utcdt = mydevice.read_datetime()
#print (str(mydevice.local_datetime(utcdt))),
if mydevice.is_nighttime(observ):
# If we are in a new night, create the new file.
config._send_to_datacenter = False ### Not enabled by default
try:
assert(config._send_to_datacenter == True)
assert(niter == 0)
mydevice.save_data_datacenter("NEWFILE")
except: pass
StartDateTime = datetime.datetime.now()
niter += 1
mydevice.define_filenames()
''' Get values from the photometer '''
try:
timeutc_mean,timelocal_mean,temp_sensor,\
freq_sensor,ticks_uC,sky_brightness = \
mydevice.read_photometer(\
Nmeasures=config._measures_to_promediate,PauseMeasures=10)
except:
print('Connection lost')
if config._reboot_on_connlost == True:
sleep(600)
os.system('reboot.bat')
time.sleep(1)
mydevice.reset_device()
formatted_data = mydevice.format_content(\
timeutc_mean,timelocal_mean,temp_sensor,\
freq_sensor,ticks_uC,sky_brightness)
try:
assert(config._use_mysql == True)
mydevice.save_data_mysql(formatted_data)
except: pass
try:
assert(config._send_to_datacenter == True)
mydevice.save_data_datacenter(formatted_data)
except: pass
mydevice.data_cache(formatted_data,number_measures=config._cache_measures,niter=niter)
if niter%config._plot_each == 0:
''' Each X minutes, plot a new graph '''
try: pysqm.plot.make_plot(send_emails=False,write_stats=False)
except:
print('Warning: Error plotting data.')
print((sys.exc_info()))
if DaytimePrint==False:
DaytimePrint=True
MainDeltaSeconds = (datetime.datetime.now()-StartDateTime).total_seconds()
time.sleep(max(1,config._delay_between_measures-MainDeltaSeconds))
else:
''' Daytime, print info '''
if DaytimePrint==True:
utcdt = utcdt.strftime("%Y-%m-%d %H:%M:%S")
print((utcdt), end=' ')
print(('. Daytime. Waiting until '+str(mydevice.next_sunset(observ))))
DaytimePrint=False
if niter>0:
mydevice.flush_cache()
if config._send_data_by_email==True:
try: pysqm.plot.make_plot(send_emails=True,write_stats=True)
except:
print('Warning: Error plotting data / sending email.')
print((sys.exc_info()))
else:
try: pysqm.plot.make_plot(send_emails=False,write_stats=True)
except:
print('Warning: Error plotting data.')
print((sys.exc_info()))
niter = 0
# Send data that is still in the datacenter buffer
try:
assert(config._send_to_datacenter == True)
mydevice.save_data_datacenter("")
except: pass
time.sleep(300)
This diff is collapsed.
This diff is collapsed.
#!/usr/bin/env python
'''
PySQM plotting program
____________________________
Copyright (c) Mireia Nievas <mnievas[at]ucm[dot]es>
This file is part of PySQM.
PySQM is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
PySQM is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with PySQM. If not, see <http://www.gnu.org/licenses/>.
____________________________
'''
import os,sys
class ArgParser:
def __init__(self,inputfile=False):
self.parse_arguments(inputfile)
def parse_arguments(self,inputfile):
import argparse
# Return config filename
self.parser = argparse.ArgumentParser()
self.parser.add_argument('-c', '--config', default="config.py")
if (inputfile):
self.parser.add_argument('-i', '--input', default=None)
args = self.parser.parse_args()
vars(self).update(args.__dict__)
def print_help(self):
self.parser.print_help()
class ConfigFile:
def __init__(self, path="config.py"):
# Guess the selected dir and config filename
# Should accept:
# - absolute path (inc. filename)
# - relative path (inc. filename)
# - absolute path (exc. filename)
# - relative path (exc. filename)
# - shortcouts like ~ . etc
self.path = path
self.config = None
def read_config_file(self,path):
# Get the absolute path
abspath = os.path.abspath(path)
# Is a dir? Then add config.py (default filename)
if os.path.isdir(abspath):
abspath += "/config.py"
# split directory and filename
directory = os.path.dirname(abspath)
filename = os.path.basename(abspath)
old_syspath = sys.path
sys.path.append(directory)
import config
self.config = config
# Create an object (by default empty) accessible from everywhere
# After read_config_file is called, GlobalConfig.config will be accessible
GlobalConfig = ConfigFile()
setup.py 0 → 100644
#!/usr/bin/env python
from distutils.core import setup
setup(name='pysqm',
version='3.1',
maintainer='Thorsten Alteholz',
maintainer_email='python@alteholz.de',
url='https://github.com/alteholz/PySQM',
license='GPLv3',
description='SQM reading and plotting software',
packages=['pysqm'],
install_requires=['pyephem','numpy','matplotlib'],
classifiers=[
"Programming Language :: C",
"Programming Language :: Cython",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: Implementation :: CPython",
'Development Status :: 3 - Alpha',
"Environment :: Other Environment",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: GNU General Public License (GPL)",
"Operating System :: OS Independent",
"Topic :: Scientific/Engineering :: Astronomy",
],
long_description=open('README.txt').read()
)
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment