diff --git a/README.md b/README.md index e0c40fcbdd0462453382378ef370ad41ec32e738..f5b9878ec83f42417d80bc99a1cd42fa6218dcf9 100644 --- a/README.md +++ b/README.md @@ -1,92 +1,9 @@ -# YAPSUT +YAPSUT +Yet Another Python Sciece Utilty Toolkit +M.Maris - 0.0 - 2022 dec 01 - -## Getting started - -To make it easy for you to get started with GitLab, here's a list of recommended next steps. - -Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)! - -## Add your files - -- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files -- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command: - -``` -cd existing_repo -git remote add origin https://www.ict.inaf.it/gitlab/michele.maris/yapsut.git -git branch -M main -git push -uf origin main -``` - -## Integrate with your tools - -- [ ] [Set up project integrations](https://www.ict.inaf.it/gitlab/michele.maris/yapsut/-/settings/integrations) - -## Collaborate with your team - -- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/) -- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html) -- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically) -- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/) -- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html) - -## Test and Deploy - -Use the built-in continuous integration in GitLab. - -- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html) -- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/) -- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html) -- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/) -- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html) - -*** - -# Editing this README - -When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template. - -## Suggestions for a good README -Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information. - -## Name -Choose a self-explaining name for your project. - -## Description -Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors. - -## Badges -On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge. - -## Visuals -Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method. - -## Installation -Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection. - -## Usage -Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README. - -## Support -Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc. - -## Roadmap -If you have ideas for releases in the future, it is a good idea to list them in the README. - -## Contributing -State if you are open to contributions and what your requirements are for accepting them. - -For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self. - -You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser. - -## Authors and acknowledgment -Show your appreciation to those who have contributed to the project. - -## License -For open source projects, say how it is licensed. - -## Project status -If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers. +This is a simple example package. You can use +[Github-flavored Markdown](https://guides.github.com/features/mastering-markdown/) +to write your content. diff --git a/src/YAPSUT/AppendableDict.py b/src/YAPSUT/AppendableDict.py new file mode 100644 index 0000000000000000000000000000000000000000..e8ebb94d379781996b798d0e64d71dbd81e4a3b3 --- /dev/null +++ b/src/YAPSUT/AppendableDict.py @@ -0,0 +1,189 @@ +from collections import OrderedDict +import numpy as np +import pandas +import io +import time + +class AppendableDict : + """ a dictionary, whose arguments are lists which can be appended + + Example: + >AD=AppendableDict() + > + >AD['line1']=1 + >AD['line2']=10 + > + >AD['line1']=2 + >AD['line2']=20 + > + >AD['line1']=3 + >AD['line2']=30 + > + >print(AD['line1']) + [1,2,3] + + """ + def __init__(self,comment=None) : + """ + Keys: + comment : comment string + """ + self._k=[] + self._d={} + self._comment="" if comment is None else comment + def keys(self) : + """ list of arguments """ + return self._k + def __getitem__(self,this) : + return self._d[this] + def __setitem__(self,this,that) : + if this in self.keys() : + self._d[this].append(that) + else : + self.keys().append(this) + self._d[this]=[that] + def isempty(self) : + """ true if empty dictionary """ + return len(self.keys()) == 0 + def same_columns_length(self) : + """ true if all the arguments have the same lenght """ + out=np.array([len(self[k]) for k in self.keys()]) + if out.ptp()==0 : return True + return False + def appendDict(self,_dict) : + """ append another dictionary """ + for k in _dict.keys() : + self[k]=_dict[k] + def set(self,key,value) : + if not key in self.keys() : + self._k.append(key) + self._d[key]=value + def flatten_nested_columns(self) : + """ treats contents as nested arrays and serializes them + beware: after this operation any appending or inserting + of new variables/values will produce unpredictable results + """ + for k in self._k : + self._d[k]=np.concatenate(self._d[k]) + def to_arrayDict(self,flatten=False) : + """ dictionary to numpy.array + if flatten==True applies .flatten_nested_columns() method + """ + out=OrderedDict() + if flatten : + for k in self.keys() : + out[k]=np.concatenate(self[k]) + else : + for k in self.keys() : + out[k]=np.array(self[k]) + return out + def to_pandas(self,flatten=False) : + """ dictionary to pandas.DataFrame + if flatten==True applies .flatten_nested_columns() method + """ + out=self.to_arrayDict(flatten=flatten) + return pandas.DataFrame(out) + def to_csv_slow(self,csvfile,justBody=False,creator="",index_label=None,sep=',',comment=None,addEnd=False,mode='w') : + """ dictionary to csv, slow version """ +# from collections import OrderedDict +# import pandas +# import time +# import io + # + Dict=self._d + # + headf=','.join(Dict.keys()) + # + n=np.array([len(self[i]) for i in Dict.keys()]) + if (n.ptp()!=0) : + raise Error("columns of different lenght ",n) + # + body=io.StringIO() + for i in range(int(n[0])) : + l=[] + for k in Dict.keys() : + l.append(str(Dict[k][i])) + l=(','.join(l))+'\n' + body.write(l) + print(l) + body=body.getvalue() + # + if justBody : + if not csvfile is None : + open(csvfile,mode).write(body+'\n') + return body + # + out=io.StringIO() + out.write("#\n") + out.write("#!filename="+(csvfile if not csvfile is None else "")+'\n') + out.write("#!created="+time.asctime()+'\n') + out.write("#!creatore="+(creator if type(creator)==type("") else "")+'\n') + out.write("#\n") + if not self._comment is None : + cc='\n'.join(['#'+k for k in self._comment.split('\n')]) + out.write(cc+'\n') + out.write('#\n') + if not comment is None : + cc='\n'.join(['#'+k for k in comment.split('\n')]) + out.write(cc+'\n') + out.write('#\n') + out.write("#!BEGIN\n") + out.write(headf+'\n') + out.write(body+'\n') + if addEnd : + out.write("#!END\n") + out=out.getvalue() + if not csvfile is None : + open(csvfile,mode).write(out) + return out + def to_csv(self,csvfile,justBody=False,creator="",index_label='index',sep=',',comment=None,addEnd=True,mode='w',sort_by=None) : + """ dictionary to csv""" +# from collections import OrderedDict +# import pandas +# import time +# import io + # + opd=self.to_pandas() + if not sort_by is None: + opd['_'+sort_by]=np.array(opd[sort_by].values) + opd.sort_values('_'+sort_by,inplace=True) + opd=opd.reindex() + opd.drop('_'+sort_by,inplace=True,axis=1) + else : + for k in self.keys() : + opd[k]=np.array(self[k]) + #opd=self.to_pandas() + # + out=io.StringIO() + if not justBody : + out.write("#\n") + out.write("#!filename="+csvfile+'\n') + out.write("#!created="+time.asctime()+'\n') + out.write("#!creator="+(creator if type(creator)==type("") else "")+'\n') + out.write("#\n") + if not self._comment is None : + cc='\n'.join(['#'+k for k in self._comment.split('\n')]) + out.write(cc+'\n') + out.write('#\n') + if not comment is None : + cc='\n'.join(['#'+k for k in comment.split('\n')]) + out.write(cc+'\n') + out.write('#\n') + out.write("#!BEGIN\n") + opd.to_csv(out,sep=',',index=(not index_label is None),index_label=index_label) + # + if not justBody and addEnd : + out.write("#!END\n") + # + #removes empty lines, if any + out=[k.strip() for k in out.getvalue().split('\n') if k.strip()!=''] + # + #removes header if just body is needed + if justBody : + out=out[1:] + # + out='\n'.join(out)+'\n' + if not csvfile is None : + open(csvfile,mode).write(out) + return out + diff --git a/src/YAPSUT/README.md b/src/YAPSUT/README.md new file mode 100644 index 0000000000000000000000000000000000000000..f5b9878ec83f42417d80bc99a1cd42fa6218dcf9 --- /dev/null +++ b/src/YAPSUT/README.md @@ -0,0 +1,9 @@ +YAPSUT +Yet Another Python Sciece Utilty Toolkit + +M.Maris - 0.0 - 2022 dec 01 - + + +This is a simple example package. You can use +[Github-flavored Markdown](https://guides.github.com/features/mastering-markdown/) +to write your content. diff --git a/src/YAPSUT/__init__.py b/src/YAPSUT/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..7134cd989b3a6f81b963d40f058c90f0e0d798e1 --- /dev/null +++ b/src/YAPSUT/__init__.py @@ -0,0 +1,8 @@ +# Created : Thu Dec 1 20:04:27 2022 +from .graphics import odot,ocirc,curveXY2patch,SaveFig,STANDARDFIG,StandarFig +from .pyblocks import pyBlocks +from .AppendableDict import AppendableDict +from .ya_config import YA_CONFIG +from .ya_hdr_parser import YA_HDR_PARSER +from .ya_extended_csv import ya_extended_csv +from .files import moveFile2backup diff --git a/src/YAPSUT/files.py b/src/YAPSUT/files.py new file mode 100644 index 0000000000000000000000000000000000000000..c0996382e2ed4d3d703df1a0901b2d730fcc6395 --- /dev/null +++ b/src/YAPSUT/files.py @@ -0,0 +1,10 @@ +def moveFile2backup(fname,skipTestExists=False) : + """if exists creates a backup file for the file fname + skipTestExists=True skips the test + """ + import time + import os + if os.path.exists(fname)==False and skipTestExists==False : return + tt=fname+'.'+time.strftime('%y%m%dl%H%M%S',time.localtime())+'~' + print('Existing '+fname+' -> ',tt) + os.rename(fname,tt) diff --git a/src/YAPSUT/graphics.py b/src/YAPSUT/graphics.py new file mode 100644 index 0000000000000000000000000000000000000000..8b77594ddcd1cef92a9c8cc2a2f5ff06c703c4e8 --- /dev/null +++ b/src/YAPSUT/graphics.py @@ -0,0 +1,428 @@ +def odot(x,y,color=['yellow','red'],markersize=[30,10],scale=1.,zorder=None,mew=[4,0]) : + """ draw an odot""" + from matplotlib import pyplot as plt + a=plt.plot(x,y,'o',color=color[0],markersize=markersize[0]*scale,mew=mew[0],markeredgecolor=color[1]) + b=plt.plot(x,y,'o',color=color[1],markersize=markersize[1]*scale,mew=mew[1]) + if type(zorder) == type(1) : + a[0].set_zorder(zorder) + b[0].set_zorder(zorder+1) + return (a,b) + +def ocirc(x,y,color=['white','black'],markersize=[30],scale=1.,zorder=None,mew=[4,0]) : + """ draw an odot""" + from matplotlib import pyplot as plt + a=plt.plot(x,y,'o',color=color[0],markersize=markersize[0]*scale,mew=mew[0],markeredgecolor=color[1]) + #b=plt.plot(x,y,'o',color=color[1],markersize=markersize[1]*scale,mew=mew[1]) + if type(zorder) == type(1) : + a[0].set_zorder(zorder) + #b[0].set_zorder(zorder+1) + return a + +def curveXY2patch(x,y,returnVectors=False,closed=True,draw=True, ax=None,color=None) : + """ Converts a curve in a patch """ + import numpy as np + from matplotlib import pyplot as plt + from matplotlib.patches import Polygon + from matplotlib.collections import PatchCollection + from matplotlib.pyplot import gca + xy=np.array([x,y]).T + if returnVectors : + return xy + pol=Polygon(xy,closed=closed,facecolor=color,edgecolor=color) + if draw: + if ax is not None: + return ax.add_collection(PatchCollection([pol])) + else: + return plt.gca().add_collection(PatchCollection([pol])) + + +class _SaveFig : + class __history_fig : + def __init__(self) : + self.clear() + def keys(self) : + return self._d.keys() + def clear(self) : + self._d={} + def __getitem__(self,this) : + if not this in self.keys() : + return [] + return self._d[this] + def __setitem__(self,this,that) : + if not this in self.keys() : + self._d[this]=[] + self._d[this].append(that) + def append(self,this) : + xt=this.split('.')[-1] + self[xt]=this + # + def __init__(self) : + self.set_abilited(False) + self.set_fig2paper(False) + self.set_path('/tmp') + self.set_paperpath('/tmp') + self.set_png(False) + self.set_pdf(True) + self.set_extensions(['pdf']) + self.history=self.__history_fig() + def set_abilited(self,v) : + self.abilited=(v==True) or (v==1) + def set_path(self,v) : + self.path=v + def set_paperpath(self,v) : + self.paperpath=v + def set_fig2paper(self,v) : + self.fig2paper=(v==True) or (v==1) + def set_png(self,v) : + self.png=v + def set_pdf(self,v) : + self.pdf=v + def set_extensions(self,v) : + self.extensions=[k for k in v] + def __str__(self) : + a='Abilited' if self.abilited else 'Not Abilited' + return 'SaveFig '+a+', path='+self.path + def __call__(self,fname,pdf=True,png=False,extensions=None) : + import numpy as np + import os + from matplotlib import pyplot as plt + # + #creates list of extensions + ext=[] + if self.pdf : ext.append('pdf') + if self.png : ext.append('png') + if pdf : ext.append('pdf') + if png : ext.append('png') + if type(extensions) in [type([]),type(())] : + for k in extl :ext.append(k) + else : + for k in self.extensions :ext.append(k) + # + if self.abilited : + fig=plt.gcf() + for xt in np.unique(ext) : + out1=self.path+'/'+xt+'/'+fname + out=out1+'.'+xt + print ("Saving in ",out) + plt.savefig(out,dpi=fig.get_dpi()) + # + self.history.append(out) + # + if self.fig2paper and type(self.paperpath)==type("") : + command='cp -v '+out+' '+self.paperpath + print (command) + print (os.system(command)) + def create_animated_gif(self,xt='png',delay=100,loop=0,out=None,execute=True) : + import os + if len(self.history[xt]) == 0 : + print('No figures with %s extension'%xt) + return + command='convert ' + command+=' -loop '+str(loop) + command+=' -delay '+str(delay) + for k in self.history[xt] : + k1=k.replace('(','\(') + k1=k1.replace(')','\)') + command+=' '+k1 + if out is None : + oname=self.history[xt][0].split('=')[0] + oname='/gif/'.join(oname.split('/'+xt+'/')) + else : + oname=out + if not '.gif' in oname : + oname+='.gif' + oname=oname.replace('(','\(') + oname=oname.replace(')','\)') + command+=' '+oname + if execute : + print(command) + os.system(command) + return oname + return command +SaveFig=_SaveFig() + +class StandardFig() : + """ class to handle a Standardized Figure layout """ + @property + def activated(self) : + return self._activated + @activated.setter + def activated(self,this) : + self._activated=(this==True) + @property + def figpath(self) : + return self._figpath + @figpath.setter + def figpath(self,this) : + # removes the trailing / + ll=this.strip().split('/') + while ((ll[-1] if len(ll)>0 else '') =='' and len(ll)>0) : ll=ll[:-1] + if len(ll)==0 : + self._figpath='.' + else : + self._figpath='/'.join(ll) + if self.split_path : + self._figpath_pdf=self._figpath+'/pdf' + self._figpath_png=self._figpath+'/png' + else : + self._figpath_pdf=self._figpath+'' + self._figpath_png=self._figpath+'' + def make_figpath(self,figpath) : + import os + self.figpath=figpath + if self._figpath!='.' : + os.system('mkdir -p '+self._figpath) + os.system('mkdir -p '+self._figpath_pdf) + os.system('mkdir -p '+self._figpath_png) + @property + def figratio(self) : + return self._figratio + @figratio.setter + def figratio(self,this) : + self._figratio=this + x=18 if self.figside == None else self.figside + y=(2/3 if self._figratio is None else self._figratio) + self._figsize=(x,x*y) + @property + def figside(self) : + return self._figside + @figside.setter + def figside(self,this) : + self._figside=this + x=18 if self.figside == None else self.figside + y=(2/3 if self._figratio is None else self._figratio) + self._figsize=(x,x*y) + @property + def figsize(self) : + return self._figsize + # + @property + def split_path(self) : + """ if split_path==True a separed subdir for png and pdf is created and used""" + return self._split_path + @split_path.setter + def split_path_setter(self,this) : + self._split_path=(this==True) + # + def __init__(self,figpath=None,figside=16,figratio=2/3,Creator='', Copyright='',Author='',split_path=False) : + self.free_metadata() + # + self._activated=True + # + self._figpath=None + self._figpath_pdf=None + self._figpath_png=None + # + self._split_path=split_path + # + self._figratio=figratio + self.figside=figside + # + if not figpath is None : + self.figpath=figpath + # + self.Author=Author + self.Creator=Creator + self.Copyright=Copyright + # + self.fig=None + # + self.fontsize_ticks=16 + self.fontsize_labels=18 + self.fontsize_xylabels=18 + self.fontsize_title=18 + self.fontsize_legend=18 + def copy(self) : + import copy + return self.deepcopy(self) + def __call__(self,squared=False) : + return self.new(squared=squared) + def new(self,squared=False) : + from matplotlib import pyplot as plt + if squared : + self.fig=plt.figure(figsize=(self.figside,self.figside)); + else : + self.fig=plt.figure(figsize=self.figsize); + self.fig.canvas.toolbar_visible = True + self.fig.canvas.header_visible = False + self.fig.canvas.resizable = True + return self.fig + def legend(self,ncol=1,loc=1,title='') : + from matplotlib import pyplot as plt + return plt.legend(ncol=ncol,loc=loc,fontsize=self.fontsize_legend,title=title) + def title(self,txt) : + from matplotlib import pyplot as plt + return plt.title(txt,fontsize=self.fontsize_title) + def xlabel(self,txt) : + from matplotlib import pyplot as plt + return plt.xlabel(txt,fontsize=self.fontsize_xylabels) + def ylabel(self,txt) : + from matplotlib import pyplot as plt + return plt.ylabel(txt,fontsize=self.fontsize_xylabels) + def XTICKS(self,ax=None) : + from matplotlib import pyplot as plt + aa=plt.gca().get_xticklabels() if ax is None else ax.get_xticklabels() + for k in aa : k.set_fontsize(self.fontsize_ticks) + def YTICKS(self,ax=None) : + from matplotlib import pyplot as plt + aa=plt.gca().get_yticklabels() if ax is None else ax.get_yticklabels() + for k in aa : k.set_fontsize(self.fontsize_ticks) + # + # metadata PROPERTIES + @property + def Copyright(self) : + return self._Copyright + @Copyright.setter + def Copyright(self,this) : + if this is None : return + self._Copyright=str(this).strip() + # + @property + def CreationDate(self) : + import time + self._CreationDate=time.asctime() + return self._CreationDate + # + @property + def ModDate(self) : + import time + self._ModDate=time.asctime() + return self._ModDate + # + @property + def Subject(self) : + return self._Subject + @Subject.setter + def Subject(self,this) : + if this is None : return + self._Subject=str(this).strip() + # + @property + def Keywords(self) : + return self._Keywords + @Keywords.setter + def Keywords(self,this) : + if this is None : return + self._Keywords=str(this).strip() + # + @property + def Author(self) : + return self._Author + @Author.setter + def Author(self,this) : + if this is None : return + self._Author=str(this).strip() + # + @property + def Creator(self) : + return self._Creator + @Creator.setter + def Creator(self,this) : + if this is None : return + self._Creator=str(this).strip() + # + @property + def WARNING(self) : + return self._WARNING + @WARNING.setter + def WARNING(self,this) : + if this is None : return + self._WARNING=str(this).strip() + # + @property + def Comment(self) : + return self._Comment + @Comment.setter + def Comment(self,this) : + if this is None : return + self._Comment=str(this).strip() + # + @property + def Disclaimer(self) : + return self._Disclaimer + @Comment.setter + def Disclaimer(self,this) : + if this is None : return + self._Disclaimer=str(this).strip() + + def free_metadata(self): + self.Title='' + self.Subject='' + self.Keywords='' + self._Comment='' + self._CreationDate=None + self._ModDate=None + self._Description='' + self._WARNING='' + self._Disclaimer='' + # + @property + def pdf_metadata(self) : + return { + 'Creator':self.Creator + ,'Author':self.Author + ,'Title':self.Title + ,'Subject':self.Subject + ,'Keywords':self.Keywords + ,'Producer':self.Copyright + #,'CreationDate':self.CreationDate + #,'ModDate':self.ModDate + } + @property + def png_metadata(self) : + return { + 'Description':self.Subject + ,'Author':self.Author + ,'Copyright':self.Copyright + ,'Title':self.Title + ,'Source':'-- plot --' + ,'Software':self.Creator + ,'Disclaimer':self.Disclaimer + ,'Copyright':self.Copyright + #,'Creation Time':self.CreationDate + ,'Warning':self.WARNING + ,'Comment':self.Comment + } + def __repr__(self) : + return str(self.__class__)+' figsize: '+str(self._figsize)+' figpath: '+self._figpath+' Creator: '+self.Creator+' activated: '+str(self.activated) + def save(self,name, + Author=None + ,Comment=None + ,Subject=None + ,Disclaimer=None + ,WARNING=None + ,Creator=None + ,Keywords=None + ,Producer=None + ) : + from matplotlib import pyplot as plt + if self.figpath is None : + raise Error('figpath is None') + # + self.Comment=Comment + self.Subject=Subject + self.Disclaimer=Disclaimer + self.WARNING=WARNING + self.Creator=Creator + self.Keywords=Keywords + self.Producer=Producer + # + if not self.activated : + print('SAVING OF %s not activated'%name) + return + # + for k in ['png','pdf'] : + # + self.Title=name + # + metadata=None + if k == 'pdf' : + metadata=self.pdf_metadata + oname=self._figpath_pdf+'/'+name+'.pdf' + if k == 'png' : + metadata=self.png_metadata + oname=self._figpath_png+'/'+name+'.png' + print('savefig ',oname) + # + plt.savefig(oname,dpi=plt.gcf().get_dpi(),metadata=metadata) +STANDARDFIG=StandardFig() diff --git a/src/YAPSUT/pyblocks.py b/src/YAPSUT/pyblocks.py new file mode 100644 index 0000000000000000000000000000000000000000..5559217bd6553abd020d7a9e718090d347c808be --- /dev/null +++ b/src/YAPSUT/pyblocks.py @@ -0,0 +1,74 @@ +__DESCRIPTION__=""" +""" +class pyBlocks : + """support class to list products in the help file""" + def __init__(self) : + from collections import OrderedDict + self.M=OrderedDict() + self.N=0 + self._cod=[] + self._dsc=[] + self._pdf=[] + self._tab=[] + self._deprecated=[] + self._ack=False + # + def __len__(self) : + return self.N + # + def keys(self) : + return self.M.keys() + # + def description(self,code) : + return self(code)[0] + # + def pdf(self,code) : + return self(code)[1] + # + def tab(self,code) : + return self(code)[2] + # + def __call__(self,*arg,**karg) : + """ + __call__(code) : returns description + __call__(iPlot,code,dsc,[pdf],[tab]) : returns iPlot==code and stores description + """ + #from collections import OrderedDict + if len(arg) == 1 : + _k=self.M[arg[0]] + return self._dsc[_k],self._pdf[_k],self._tab[_k] + # + self.N+=1 + iPlot=arg[0] + _c=arg[1] + self._cod.append(_c) + self._dsc.append(arg[2]) + self._pdf.append(arg[3] if len(arg)>=4 else '') + self._tab.append(arg[4] if len(arg)>=5 else '') + self.M[_c]=self.N-1 + + # + if 'deprecated' in karg : + self._deprecated.append(karg['deprecated']==True) + else : + self._deprecated.append(False) + + # if another code have been already acknowledged does nothing + if self._ack : return False + + self._ack=_c==iPlot + + if self._ack : + print("\n*************************************\n",_c," : ",arg[2]) + return self._ack + # + def acknowledged(self) : return self._ack + # + def __str__(self) : + import numpy as np + if len(self) == 0 : return '' + l=[] + idx=np.argsort(self._cod) + for k in idx : + l.append('%5d : %s%s'%(self._cod[k],self._dsc[k],' <>' if self._deprecated[k] else '' )) + return '\n'.join(l) diff --git a/src/YAPSUT/ya_config.py b/src/YAPSUT/ya_config.py new file mode 100644 index 0000000000000000000000000000000000000000..da5674b3c0d350718a93e37d64b6f71435dee60f --- /dev/null +++ b/src/YAPSUT/ya_config.py @@ -0,0 +1,48 @@ +__DESCRIPTION__=""" +Yet Another Configuration Loader + +M.Maris 24 Sep 2021 + +As usual I lost the already prepared one +""" + +class YA_CONFIG : + def __init__(self,fName) : + from collections import OrderedDict + self._o=OrderedDict() + self._n=0 + self._fname=fName + self._COMMENT_CHAR='#' + if not fName is None : + self.load(fName) + def keys(self) : + return self._o.keys() + def __getitem__(self,this) : + return self._o[this] + def load(self,fname) : + import re + import numpy as np + self._fname=fname + txt=[k.strip() for k in open(fname,'r') if k.strip() != ''] + self.parse(txt) + def parse(self,txt) : + for l in txt : + l1=l.strip() + if l1=='' : + pass + elif l1[0]==self._COMMENT_CHAR : + pass + elif '=' in l1 : + ll=[k.strip() for k in l1.split('=')] + ll.append('') + if ll[0] in self.keys() : + print("Warning duplicated key ",ll[0]) + self._o[ll[0]]=ll[1] + else : + pass + def __str__(self) : + return str(self._o) + def __copy__(self) : + import copy + return copy.deepcopy(self) + diff --git a/src/YAPSUT/ya_extended_csv.py b/src/YAPSUT/ya_extended_csv.py new file mode 100644 index 0000000000000000000000000000000000000000..2eeaf424daaf88cc57045066970e5faad283640c --- /dev/null +++ b/src/YAPSUT/ya_extended_csv.py @@ -0,0 +1,355 @@ +""" Yet Another extended csv manages + +extended csv is csv with: + +. comment lines +. parameter (constant parameters) +. name files +. declared separator +. BEGIN/END data marker + +comment line: +^\s*#{.}\n + +parameter +^\s*#!=\n + +BEGIN marker +^\s*#!BEGIN\s*\n + +END marker +^\s*#!END\s*\n + +standard fields used for formatting + +declares the used separator +#!__sep__= + +declares the used comment character +#!__comment__= + +declares the python interpreter +#!__interpreter__= + +the separator can be a single character or a sequence +""" + +from collections import OrderedDict + +import pandas +class ya_extended_csv : + @property + def header(self) : + return self._header + # + @property + def skipUndef(self) : + return self._skipUndef + @skipUndef.setter + def skipUndef(self,this) : + self._skipUndef= (this==True) or (this==1) or (str(this).lower().strip()=='true') + # + @property + def sep(self) : + return self._sep + # + @sep.setter + def sep(self,this) : + self._sep=this + # + @property + def reserved_tags(self) : + return ['creator','created','creator_version','header_version','sep','tag_comment','tag_variable','filename','author' + ,'!__index_label','!__sep__','!__tag_comment__','!__tag_variable__','!__assign_operator__','!__index_label__'] + # + @property + def doNotLoadTags(self) : + return ['BEGIN','END','!__index_label','!__sep__','!__tag_comment__','!__tag_variable__','!__assign_operator__','!__index_label__'] + # + def __init__(self,sep=',',tag_comment='#',tag_variable='#!',assign_operator='=',skipUndef=True,undefValue=None,index_column=0) : + """ +Instantiates an extended_csv manager with default syntax +syntax elements can be changed modifiing parameters + :keyword: sep separator, character, default ',' + :keyword: tag_comment, comment line marker, default '#' + :keyword: tag_variable, variable line marker, default '#!' + :keyword: assign_operator, marker for assignement, default '=' + :keyword: skipUndef, if True undefined variables are skipped (variables without the assign_operator), default True + :keyword: undefValue, specifies value for undefined variables if skipUndef is False, default None + :keyword: index_column, index or name of the column used for indexing the csv table, default 0 +""" + self.clear() + self._sep=sep + self._tag_comment=tag_comment + self._tag_variable=tag_variable + self._assign_operator=assign_operator + self._skipUndef=skipUndef + self._undefValue=undefValue + self._index_column=index_column + self.__formatter_message=' *** FORMATTED BY YA_EXTENDED_CSV - V 1.0 - M.Maris - 2020 Jan 20 - ***' + self._verbose=False + # + def clear(self) : + self._content=None + self._hdrl=None + self._header=None + return + # + def hdr_parser(self) : + "parses the header" + self._header=OrderedDict() + for k in open(self._fname,'r') : + l=k.strip() + l1=l+' ' + if (l+' ')[0:len(self._tag_variable)]==self._tag_variable : + a=(l+' ')[len(self._tag_variable):] + a=a.strip() + if self._assign_operator in a : + a=[k.strip() for k in a.split(self._assign_operator)] + self._header[a[0]]=a[1] + else : + if not self._skipUndef : + self._header[a]=self._undefValue + # + def get_data(self,sep) : + engine='c' if len(sep)==1 else 'python' + if self._verbose : print(engine,sep) + self._csv=pandas.read_csv(self._fname,sep=sep,comment=self._tag_comment,engine=engine,index_col=self._index_column) + # + def set_pandas_attrs(self) : + if len(self._header.keys())>0 : + for k in self._header.keys() : + if k not in self.doNotLoadTags : + self._csv.attrs[k]=self._header[k] + # + def to_pandas(self) : + return self._csv + # + def to_dict(self) : + out=OrderedDict() + for k in self._csv.keys() : + out[k]=self._csv[k].values + # + def load(self,fname,strip_spaces=True,verbose=False,sep=None) : + """ load an extended csv into a pandas dataframe + :parameter: fname is the filename + :parameter: strip_spaces if True leading and trailing spaces in columns are removed (slow readout) + :parameter: verbose if True verbose messages + """ + self._verbose=verbose + self._fname=fname + self.hdr_parser() + # + _sep_used=sep if not sep is None else self._sep + if strip_spaces : + _sep_used='\s*'+_sep_used+'\s*' + # + self.get_data(_sep_used) + self.set_pandas_attrs() + return self.to_pandas() + # + def save(self,fname + ,pdtbl + ,sep=None + ,creator='' + ,creator_version='' + ,header_version='' + ,author='' + ,content=None + ,headerDict=None + ,body=None + ,add_pandas_attrs=True + ,index_label='__index__') : + """ saves an extended csv into a pandas dataframe + + :parameter: fname is the filename + :parameter: pdtbl is the pandas table + :parameter: creator the code which created the file + :parameter: creator_version the version of the code which created the file + :parameter: header_version the version of the header + :parameter: author the author of the file + :parameter: content a string used as file content description (*) + :parameter: headerDict a dictionary with variables to be stored in header (after conversion in string) + :parameter: body a string to be placed at the end of the header (*) + :parameter: add_pandas_attrs if True content of pdtbl.attrs is converted to string and added to the header + :parameter: index_label label for the index of the pandas table + + (*) it can be a single string, multilines content can be added either using a list of strings or dividing the string in lines by using the '\n' character + """ + import time + from io import StringIO + self._csv=pdtbl + hdr=[self._tag_comment] + # + # append a content - a content is the initial part of hdr + # it can be a string, a string separed by \n or an array of strings + if not content is None : + if type(content) == type("") : + if '\n' in content : + ll=content.split('\n') + else : + ll=[content] + else : + ll=[k.split('\n')[0] for k in content] + for k in ll : + hdr.append(self._tag_comment+k) + # + hdr.append(self._tag_comment) + hdr.append(self._tag_variable+'filename'+self._assign_operator+fname) + # + hdr.append(self._tag_comment) + if author is None : + hdr.append(self._tag_variable+'author'+self._assign_operator+'') + else : + hdr.append(self._tag_variable+'author'+self._assign_operator+author) + # + hdr.append(self._tag_comment) + if creator is None : + hdr.append(self._tag_variable+'creator'+self._assign_operator+'') + else : + hdr.append(self._tag_variable+'creator'+self._assign_operator+creator) + # + if creator_version is None : + hdr.append(self._tag_variable+'creator_version'+self._assign_operator+'') + else : + hdr.append(self._tag_variable+'creator_version'+self._assign_operator+creator_version) + hdr.append(self._tag_variable+'created'+self._assign_operator+time.asctime()) + # + hdr.append(self._tag_comment) + if header_version is None : + hdr.append(self._tag_variable+'header_version'+self._assign_operator+'') + else : + hdr.append(self._tag_variable+'header_version'+self._assign_operator+header_version) + # + hdr.append(self._tag_comment) + if not headerDict is None : + for k in headerDict.keys() : + if not k in self.reserved_tags : + hdr.append(self._tag_variable+k+self._assign_operator+str(headerDict[k])) + # + if add_pandas_attrs : + if len(self._csv.attrs.keys()) > 0 : + hdr.append(self._tag_comment) + hdr.append(self._tag_comment+' pandas attrs ') + for k in self._csv.attrs.keys() : + k1=[j for j in k.strip().split() if j!=''] + if not k in self.reserved_tags and not 'BEGIN' in k1 and not 'END' in k1: + hdr.append(self._tag_variable+k+self._assign_operator+str(self._csv.attrs[k])) + hdr.append(self._tag_comment) + # + # append a body + # it can be a string, a string separed by \n or an array of strings + if not body is None : + if type(body) == type("") : + if '\n' in body : + ll=body.split('\n') + else : + ll=[body] + else : + ll=[k.split('\n')[0] for k in body] + for k in ll : + hdr.append(self._tag_comment+k) + # + _sep_used=sep.strip() if sep!=None else self._sep.strip() + # + hdr.append(self._tag_comment) + hdr.append(self._tag_comment+' ====================== ') + hdr.append(self._tag_comment+' formatting description ') + hdr.append(self._tag_variable+'!__index_label__'+self._assign_operator+str(index_label)) + hdr.append(self._tag_variable+'!__sep__'+self._assign_operator+str(_sep_used)) + hdr.append(self._tag_variable+'!__tag_comment__'+self._assign_operator+str(self._tag_comment)) + hdr.append(self._tag_variable+'!__tag_variable__'+self._assign_operator+str(self._tag_variable)) + hdr.append(self._tag_variable+'!__assign_operator__'+self._assign_operator+str(self._assign_operator)) + hdr.append(self._tag_comment+' ====================== ') + # + hdr.append(self._tag_comment) + hdr.append(self._tag_comment+self.__formatter_message) + # + hdr.append(self._tag_comment) + hdr.append(self._tag_comment+'!BEGIN') + # + out=StringIO() + out.write('\n'.join(hdr)+'\n') + # + pdtbl.to_csv(out,sep=_sep_used,index_label=index_label) + out.write('#!END\n') + # + out.seek(0) + open(fname,'w').write(''.join(out.readlines())) + +if __name__=="__main__" : + from collections import OrderedDict + import numpy as np + import sys + + print("CREATES A PANDAS TABLE TO BE SAVED") + out=OrderedDict() + out['a']=np.arange(10) + out['b']=-np.arange(10) + out['c']=np.array([' s'+str(k) for k in np.arange(10)]) + out['d']=np.array([' '+str(k) for k in np.arange(10)]) + + out=pandas.DataFrame(out) + + print('===============') + print('Simple csv file') + ofile='/tmp/ya_extended_csv_simple.csv' + print(ofile) + + ya_extended_csv().save(ofile,out) + + for k in open(ofile,'r') : + print(k.strip()) + print('===============') + + print() + print("Sets some attributes to the pandas table") + out.attrs['mio']='mia' + print() + + print('=======================') + print('A more complex csv file') + ofile='/tmp/ya_extended_csv.csv' + print(ofile) + + ya_extended_csv().save(ofile,out, + creator=sys.argv[0], + creator_version='0.0 ', + author='M.Maris', + header_version='0.0', + headerDict={'pippo':10,'pallino':'ecco'}, + content='Exaple of Table\nTo test Creation\n', + body='columns are a and b' + ) + + for k in open(ofile,'r') : + print(k.strip()) + print('=======================') + + print() + + print('==============================') + print('Read the more complex csv file in a new pandas table') + CSV=ya_extended_csv().load(ofile) + + print() + print('The pandas table') + print(CSV) + + print() + print('The pandas table attributes from the csv file') + print(CSV.attrs) + print('==============================') + + print('=======================') + print('A csv file using & as separator ') + ofile='/tmp/ya_extended_csv_different.csv' + ya_extended_csv().save(ofile,out,sep='&') + print(ofile) + + for k in open(ofile,'r') : + print(k.strip()) + + CSV=ya_extended_csv().load(ofile,sep='&') + print('=======================') + diff --git a/src/YAPSUT/ya_hdr_parser.py b/src/YAPSUT/ya_hdr_parser.py new file mode 100644 index 0000000000000000000000000000000000000000..e54f21cbf1a1831fa1f6873ada5f39844462c13a --- /dev/null +++ b/src/YAPSUT/ya_hdr_parser.py @@ -0,0 +1,35 @@ +__DESCRIPTION__=""" +YA parses for header of a csv file + +M.Maris - 2021 dec 10 +""" + +from collections import OrderedDict + +class YA_HDR_PARSER(OrderedDict) : + def fileName(self) : + return self.____filename + def __init__(self,fileName,sep=',',tag_comment='#',tag_variable='#!',assign_operator='=',skipUndef=True,undefValue=None) : + #OrderedDict.__init__(None) + self.____filename=fileName + self.____comment=tag_comment + self.____sep=sep + self.____variable_marker=tag_variable + self.____assign_operator=assign_operator + self.____load(skipUndef,undefValue) + def ____load(self,skipUndef,undefValue) : + for k in open(self.____filename,'r') : + l=k.strip() + l1=l+' ' + if (l+' ')[0:len(self.____variable_marker)]==self.____variable_marker : + a=(l+' ')[len(self.____variable_marker):] + a=a.strip() + if self.____assign_operator in a : + a=[k.strip() for k in a.split(self.____assign_operator)] + self[a[0]]=a[1] + else : + if not skipUndef : + self[a]=undefValue + + +