Skip to content
Snippets Groups Projects
Select Git revision
  • 1986ed0e61420852edb4d8cb455ca13f3b142089
  • master default protected
  • ia2
  • adql2.1-ia2
  • private_rows
5 results

ADQLColumn.java

Blame
    • gmantele's avatar
      496e769c
      [ADQL,TAP] Add STC-S and UDFs support in the ADQL parser. Now, it is possible... · 496e769c
      gmantele authored
      [ADQL,TAP] Add STC-S and UDFs support in the ADQL parser. Now, it is possible to provide a list of allowed UDFs, regions and coordinate systems. The ServiceConnection of TAP is now able to provide these lists and to propagate them to the ADQLExecutor. UDFs and allowed regions are now listed automatically in the /capabilities resource of TAP. The type 'geometry' is now fully supported in ADQL. That's why the new function 'isGeometry()' has been added to all ADQLOperand extensions. Now the DBChecker is also able to check roughly types of columns and UDFs (unknown when parsing syntactically a query). The syntax of STC-S regions (expressed in the REGION function) are now checked by DBChecker. However, for the moment, geometries are not serialized in STC-S in the output....but it should be possible in some way in the next commit(s).
      496e769c
      History
      [ADQL,TAP] Add STC-S and UDFs support in the ADQL parser. Now, it is possible...
      gmantele authored
      [ADQL,TAP] Add STC-S and UDFs support in the ADQL parser. Now, it is possible to provide a list of allowed UDFs, regions and coordinate systems. The ServiceConnection of TAP is now able to provide these lists and to propagate them to the ADQLExecutor. UDFs and allowed regions are now listed automatically in the /capabilities resource of TAP. The type 'geometry' is now fully supported in ADQL. That's why the new function 'isGeometry()' has been added to all ADQLOperand extensions. Now the DBChecker is also able to check roughly types of columns and UDFs (unknown when parsing syntactically a query). The syntax of STC-S regions (expressed in the REGION function) are now checked by DBChecker. However, for the moment, geometries are not serialized in STC-S in the output....but it should be possible in some way in the next commit(s).
    MatMust.m 14.59 KiB
    classdef MatMust < handle
        %MATMUST A library to get deal with ESA webmust REST API
        % Follow indication in methods help.
        %
        % +++++++++++++++++++++++++++++++++++++
        % Creator: Carmelo Magnafico IAPS/INAF
        % date: 20/06/2019
        %
        % +++++++++++++++++++++++++++++++++++++
        
        properties
           POSToptions
           GEToptions
           loginstatus
           url
           urlsub
           userProjects
        end
        
        properties (SetAccess = private)
            tockenfilename
            tockenstruct
        end
       
       properties (Transient)
       end
        
        methods
            function obj = MatMust(varargin)
                % Initialize the values
                obj.POSToptions = weboptions('RequestMethod', 'POST', 'ArrayFormat', 'json', 'Timeout',30, 'MediaType','application/json');
                obj.GEToptions = weboptions('RequestMethod', 'GET', 'ArrayFormat', 'repeating', 'Timeout',30, 'MediaType','application/x-www-form-urlencoded');
                obj.url = 'https://bepicolombo.esac.esa.int/webclient-must/';
                obj.urlsub = 'mustlink/';
                obj.tockenfilename = 'auth.json';
                obj.loginstatus = false;
                
                %check the url
                if(nargin > 0)
                    if isfield(varargin{1},'url')
                        obj.url = varargin{1}.url;
                    end
                end
                
                %check url
                [~,status] = urlread(obj.url);
                if ~status
                   error('invalid URL')
                end
                
                if(MatMust_logincheck(obj))
                    disp('++++       MATMUST initialization       ++++');
                    disp('++++ Access token valid. User logged in ++++');
                    disp('++++++++++++++++++++++++++++++++++++++++++++');
                    return
                end
                
            end
            function e = MatMust_logincheck(obj)
                f = fopen(obj.tockenfilename);
                e = false;
                % check if user is already logged in
                if f > 0
                    raw = fread(f,inf);
                    tockenfilejson = jsondecode(char(raw));
                    if isfield(tockenfilejson,'token') &  isfield(tockenfilejson,'expiresAt')                 
                        if length(split(tockenfilejson.token,'.')) == 3 & datetime(datenum(tockenfilejson.expiresAt, 'yyyy-mm-dd HH:MM:SS'), 'ConvertFrom','datenum', 'TimeZone','utc') > datetime(datetime(now, 'ConvertFrom','datenum', 'TimeZone','local'), 'TimeZone','utc')
                            obj.loginstatus = true;
                            obj.tockenstruct = tockenfilejson;
                            MatMust_getUserLinkedProjects(obj);
                            e=true; 
                        else
                            disp('++++               MATMUST initialization                    ++++');
                            disp('++++ invalid or exiperd access token. User not logged in yet ++++');
                            disp('++++     User login needed - use MatMust_login to access     ++++');
                            disp('+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++');
                        end
                    end
                else
                    disp('++++               MATMUST initialization                    ++++');
                    disp('++++      cannot find access token. User not logged          ++++');
                    disp('++++     User login needed - use MatMust_login to access     ++++');
                    disp('+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++');
                end
            end
            function t = MatMust_getTockenStatus(obj)
                t = obj.loginresponse;
            end
            function t = MatMust_getUserProjects(obj)
                if obj.loginstatus
                    query = 'usermanagement/projects';
                    obj.GEToptions.HeaderFields = {'Authorization', obj.tockenstruct.token};
                    try
                        response = webread([obj.url,obj.urlsub, query], obj.GEToptions);
                        t = response ;
                    catch ME
                        fprintf('+++ WARNING\n%s - %s\n', ME.identifier, ME.message)
                    end
    
                end
            end
            function t = MatMust_getUserLinkedProjects(obj, varargin)
                % Get the project to witch the logged in User belongs and have access to. 
                % if 'dump' option is set a screenslash is done
                userInfo = MatMust_getUserInfo(obj);
                obj.userProjects = userInfo.projects;
                t = obj.userProjects;
                if nargin > 1
                    if strcmp(varargin{1},'dump')
                        fprintf('+++++++++++++ PROJECTS LIST +++++++++++++\n');
                        for i = 1:numel(t)
                            fprintf('\tname: %s\n\tdescription: %s\n\tid: %i\n', t(i).name, t(i).description, t(i).id);
                            fprintf('+++++++++++++++++\n');
                        end
                    end
                end
            end
            function t = MatMust_getUserInfo(obj)
                if obj.loginstatus
                    query = 'usermanagement/userinfo';
                    obj.GEToptions.HeaderFields = {'Authorization', obj.tockenstruct.token};
                    try
                        response = webread([obj.url,obj.urlsub, query], obj.GEToptions);
                        t = response ;
                    catch ME
                        fprintf('+++ WARNING\n%s - %s\n', ME.identifier, ME.message)
                    end
                end
            end 
            function data = MatMust_getDataFromName(obj, ds, parname, dateStart, dateStop, varargin   )
                % data = MatMust_getDataFromName(obj, ds, parname, dateStart, dateStop, options  )
                % MatMust_getDataFromName get the data name parameter from the project name ds
                % in the time interval dateStart, dateStop.
                %
                % inputs
                % ------
                % ds = 'string' -> name of the project i.e.'BEPICRUISE'
                % parname = 'string' -> a parameter name i.e. 'NCAD7EB7'
                % dateStart = 'yyyy-mm-dd HH:MM:SS' format date UTC time
                % dateStop = 'yyyy-mm-dd HH:MM:SS' format date UTC time
                %   'dump','plot' optional parameter to dump and plot the data
              
                % intialize data output
                data = [];
                if ~(MatMust_logincheck(obj))
                    return
                end
                
                
                %validate ds
                if isfield(obj.userProjects,'name')
                    if ~any(strcmp({obj.userProjects.name},ds))
                        disp('++++ Project not in Available User List, check ds name or check you access rigth ++++');
                        return
                    end
                end
                   
    
                query = ['dataproviders/',ds,'/parameters/data'];
                obj.GEToptions.HeaderFields = {'Authorization', obj.tockenstruct.token};
    
                if isa(parname,'cell')
                    parname = char(join(parname,','));
                end
    
                args = {'key', 'name',...
                        'values', (parname),...
                        'from', (dateStart),...
                        'to', (dateStop),...
                        'calibrate', 'false',...
                        'mode', 'SIMPLE'};
    
                try
                    data = webread([obj.url,obj.urlsub, query], args{:} ,obj.GEToptions);
                catch ME
                    fprintf('+++ WARNING\n%s - %s\n', ME.identifier, ME.message)
                    return
                end
    
                %refine date string to adapt MATLAB date format
                for i=1:numel(data)
                    data_par = data(i).data;
                    t = num2cell(datenum(datetime('1970-01-01', 'TimeZone','utc'))+(str2num(str2mat(data_par.date))/1000/86400));
                    [data(i).data.dateMAT] = deal(t{:});
                    clear t
                end
    
                if nargin > 5
                    if any(ismember(varargin,'dump'))
                        fprintf('+++++++++++++ DATA RESULT LIST +++++++++++++\n');
                        for i=1:numel(data)
                            data_par = data(i).data;
                            fprintf('%s-%s: %s -> %s\n', data(i).subsystem, data(i).type, data(i).name, data(i).description );
                            %TODO filter for several kind of dataType 'DOUBLE',
                            %'SIGNED_SMALL_INT', 'STRING'
                            switch data(i).dataType
                                case 'DOUBLE'
                                    format = '%f';
                                case 'SIGNED_SMALL_INT'
                                    format = '%i';
                                case 'STRING'
                                    format = '%s';
                            end
                            format = ['%s - ',format,'\n'];
                            for i = 1:numel(data_par)
                                 fprintf(format,datestr(data_par(i).dateMAT,'yyyy-mm-dd HH:MM:SS.FFF'), data_par(i).value ); 
                            end 
    
                            fprintf('\n++++++++++++++\n\n');
                        end
                        fprintf('+++++++++++++++++++++++++++++++++++++++++\n');
                    end
                    if any(ismember(varargin,'plot'))
                         for i=1:numel(data) 
                            data_par = data(i).data;
                            figure
                            title(sprintf('%s-%s: %s -> %s\n', data(i).subsystem, data(i).type, data(i).name, data(i).description ))
                            plot([data_par.dateMAT], [data_par.value], '-x' );
                            xlabel('time');
                            ylabel(data(i).unit);
                            datetick('x');
                            grid on
                        end
                    end
                end
    
            end
            function t = MatMust_searchTMparFromName(obj, namequery, dataproviders, varargin)
                if obj.loginstatus
                    query = 'metadata/treesearch';
                    obj.GEToptions.HeaderFields = {'Authorization', obj.tockenstruct.token};
                    if isa(dataproviders,'cell')
                        dataproviders = char(join(dataproviders,','));
                    end
                    try
                        response = webread([obj.url,obj.urlsub, query], 'field','name','text',namequery,'dataproviders', dataproviders ,obj.GEToptions);
                        t = response ;
                    catch ME
                        fprintf('+++ WARNING\n%s - %s\n', ME.identifier, ME.message)
                    end
                    if nargin > 3
                        if strcmp(varargin{1},'dump')
                            fprintf('+++++++++++++ PARAMETERS QUERY RESULT LIST +++++++++++++\n');
                            found = false;
                            for i = 1:numel(t)
                                if numel(t(i).children) > 0
                                    found = true;
                                    fprintf('\n\n\t+++++++++++++ %s LIST +++++++++++++\n', t(i).type);
                                    for j = 1:numel(t(i).children)
                                        par = t(i).children(j); 
                                        fprintf('\ttitle: %s\n\tname: %s\n\tdescription: %s\n\tid: %s\n', par.title, par.sortingField, par.label, par.id);
                                        fprintf('  +++++++++++++++++\n');
                                    end
                                end
                            end 
                            if ~found,  fprintf('++++                  Nothing found                 ++++\n'); end
                            fprintf('++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n');
                        end
                    end
                end
            end
            function [exit] = MatMust_login(obj, varargin )
                % login 
                %   https://bepicolombo.esac.esa.int/webclient-must/mustlink/api-docs/index.html#/Authentication/login
                % input:
                %   "username": "string",
                %   "password": "string",
                %   "maxDuration": false
                % output: 
                %   "token": "string",
                %   "issuer": "string",
                %   "subject": "string",
                %   "issuedAt": "string",
                %   "expiresAt": "string",
                %   "expirationDate": "yyyy-mm-ddTHH:MM:SS.FFFZ"
               
                if ~obj.loginstatus
                    % validation input
                    input_validation = struct('username', 'char', 'password', 'char', 'maxDuration', 'logical' );
    
                    exit = false;
                    if(nargin > 0)
                        inputs = varargin{1};
                        e = obj.MatMust_inputvalidator(inputs, input_validation);
                        if ~e
                            exit = false;
                            return
                        else
                            exit = true;
                        end
                    else
                        exit = false;
                        return
                    end
    
                    json_POST = inputs;
                    query = 'auth/login/';
                    obj.tockenstruct = webwrite([obj.url,obj.urlsub, query],json_POST,obj.POSToptions); 
                    if isfield(obj.tockenstruct,'token')
                        disp('+++ Succesfully logged in +++')
                        obj.loginstatus = true;
                    else
                        disp('+++ Error in loggin +++')
                        obj.loginstatus = false;
                    end
    
    
                    %store into a file
                    if obj.loginstatus
                        f = fopen(obj.tockenfilename, 'w');
                        text = jsonencode(obj.tockenstruct);
                        fprintf(f,'%s',text);
                        fclose(f);
                    end
                    
                    %store the UserAvailable Projects in class Property
                    MatMust_getUserLinkedProjects(obj);
                
                else
                   exit = true;
                   disp('user already loggen in')
                end
                
            end       
            function exit = MatMust_inputvalidator(obj, input, validation)
                % For now, every validation_input field is treated as required
                % TODO: set add a subfield required at validataion json
                exit = false;
                val_fields = fields(validation);
                for i = 1:numel(val_fields)
                    val_field = validation.(val_fields{i});
                    if isfield(input, val_fields{i})
                        if isa(input.(val_fields{i}), val_field)
                            exit = true;
                        else
                            exit = false;
                        end
                    else
                        exit = false;
                    end
                end
      
            end
        end
        
    end