Skip to content
  1. Oct 22, 2018
  2. Sep 05, 2018
    • Grégory Mantelet's avatar
      [UWS,TAP] Fix the configuration file for the UPLOAD feature. · 7bd91a1c
      Grégory Mantelet authored
      The property `upload_default_db_limit` has been deprecated. Indeed, in the
      current state of the TAP protocol, this makes no sense: the user can not change
      the limit size (in bytes or rows) for uploaded tables.
      
      The property `upload_max_file_size` has been deprecated. It is actually
      duplicated: `upload_max_db_limit`, if expressed in bytes already lets put a
      limit on the maximum size of an uploaded table/file.
      
      The property `upload_max_request_size` has been added. It lets set a maximum
      size for a whole HTTP Multipart Request. By default it is set to 250MB.
      
      The default value of `upload_max_db_size` is now 1 million rows.
      
      The UPLOAD feature is still disabled by default (i.e. `upload_enabled=false`).
      7bd91a1c
    • Grégory Mantelet's avatar
      [UWS,TAP] Fix destruction of a job: error files were never deleted. · 02a4a7f5
      Grégory Mantelet authored
      This bug occurred "just" due to an un-desired inverted test, since UWS-1.1 is
      implemented (UWSLib-4.3 and TAPLib-2.2).
      02a4a7f5
  3. Aug 21, 2018
  4. Aug 20, 2018
  5. Aug 10, 2018
    • Grégory Mantelet's avatar
      [TAP] Return a clear error message in case of duplicated items in UPLOADs. · e447a487
      Grégory Mantelet authored
      Before correction, if two uploaded tables have been submitted by the user with
      the same name, or if one uploaded table contained duplicated column names, an
      obscure error message coming from the database was returned to the user.
      
      Now, duplicated items (tables and columns) are searched before ingestion in the
      database. When one is detected, an error is immediately returned to the user and
      the query is aborted.
      e447a487
    • Grégory Mantelet's avatar
      [TAP] Fix disk space consumption with UPLOAD for synchronous jobs. · 25f373f6
      Grégory Mantelet authored
      Files uploaded by the user when creating/executing a synchronous job were never
      deleted after the job execution.
      
      The same problem applied for the tables already uploaded in the database (in
      `TAP_UPLOAD`) when an error occurred before the end of the UPLOAD process.
      
      Now, in case of error when uploading one or more files, or in case of success of
      the job, all uploaded files and their corresponding database tables are deleted
      after the end of the job.
      25f373f6
    • Grégory Mantelet's avatar
  6. Aug 07, 2018
  7. Jul 30, 2018
    • Grégory Mantelet's avatar
      [UWS] Fix recurrent ConcurrentModificationException during service overload. · ded9cead
      Grégory Mantelet authored
      This error occurred generally during the backup process while trying to
      backup the job list of a specific user. If several of his jobs were running
      and changing state during the backup process, this
      ConcurrentModificationException was thrown. This generally happens when the same
      user submits a lot of shorts jobs in the same time.
      
      This exception was due to a non thread-safe usage of
      UWSParameters.additionalParams. To fix this issue, instead of creating it as a
      normal HashMap, it is now created as a ConcurrentHashMap.
      
      The same modification has also been applied to UWSParameters.params. In addition
      of the replacement of HashMap into ConcurrentHashMap, all `synchronized` blocks
      have been removed....there should not be needed any more.
      ded9cead
  8. Jul 27, 2018
    • Grégory Mantelet's avatar
      [UWS,TAP] Fix the weekly log file rotation. · 1f4bc6b1
      Grégory Mantelet authored
      When enabled, it was generating a file each minute on the day before the
      specified day of week.
      
      For instance: if the log rotation frequency was `W 1 0 0` (so, weekly on Sunday
      at 00:00). The rotation was performed on Saturday midnight. But, because of a
      bad index correction, the rotation kept going on the whole day of Saturday.
      Since the rotated file is suffixed by the timestamp with hours and minutes
      (no seconds), it actually generated a new log file for each minute of the
      saturday. Of course, each time the file contained only one line (or 2 with some
      luck)...which is pretty useless.
      1f4bc6b1
  9. Jul 04, 2018
  10. Jul 02, 2018
    • Grégory Mantelet's avatar
      [UWS,TAP] Fix the backup file writing. · 474da7f4
      Grégory Mantelet authored
      Instead of writing the new backup content in the final backup file directly,
      write it first in a temporary file and then change the files name.
      
      This fix prevents incomplete backup files (particularly in case of one backup
      file per user) when stopping/restarting by force.
      474da7f4
  11. May 18, 2018
  12. Apr 23, 2018
  13. Apr 09, 2018
  14. Mar 07, 2018
    • gmantele's avatar
      Fix UDF parsing from the configuration file. · 96df1d5a
      gmantele authored
      The end of the description of a UDF was not detected when this UDF was followed
      by another UDF definition. This was due to an incorrect double quote escape in
      the regular expression of a UDF's definition.
      96df1d5a
  15. Mar 02, 2018
    • gmantele's avatar
      [UWS] Fix Javadoc · a959db1f
      gmantele authored
      a959db1f
    • gmantele's avatar
      [UWS] Fix the quote serialization. · 855a805b
      gmantele authored
      The UWS-1.x standard defines the quote as an ISO-8601 date. UWS-Lib stores it
      as a number of seconds (i.e. estimated job duration).
      
      This fix ensures that this integer/long quote value is returned as a date.
      
      Note: The backup and restoration processes are not affected by this change.
            The backup file format is still the same: a quote stored as a long value.
      855a805b
  16. Feb 26, 2018
    • gmantele's avatar
      [UWS] Support the blocking behavior described in PR-UWS-1.1. · d2e5d98a
      gmantele authored
      It is possible to choose how the blocking mechanism should behave
      (e.g. what the max. waiting period, how many requests can be blocked
      in the same time, what happen when the blocking times out, ...).
      
      Indeed, the policy to apply must actually be an extension of the interface
      BlockingPolicy. Already two implementations are provided in the library
      (LimitedBlockingPolicy and UserLimitedBlockingPolicy), but a custom policy
      can perfectly be created and apply to a UWS service.
      
      By default, no policy is set. In such case, the service will block the time
      specified by the user, which may be -1 (i.e. wait indefinitely). A
      BlockingPolicy can help controlling the waiting/blocking process and protect
      the resources of the server.
      d2e5d98a
    • gmantele's avatar
      [UWS] Add PHASE, AFTER and LAST filters on a JobList. · f08018cb
      gmantele authored
      - PHASE: list only jobs in the specified PHASE. If this parameter is repeated
               jobs matching any of the specified phases will be returned.
      - AFTER: list jobs created after the specified ISO-8601 date (included).
               If this parameter is repeated, only the most recent date is retained.
      - LAST: list the N-th most recently created jobs, ordered by descending
              creation time
      
      These filter parameters are additive: their constraints are joint as with an
      AND operator (except for PHASE parameters ; see above).
      
      If no filter is specified, all jobs EXCEPT the ARCHIVED ones are listed. The
      only way to list ARCHIVED jobs is to use PHASE=ARCHIVED (with or without other
      filter parameters).
      
      The filtering API has been made in a generic manner so that it is easily
      possible to create and add new filters. See the interface JobFilter and the
      class JobListRefined for more details.
      f08018cb
    • gmantele's avatar
      [UWS] Add 3 job destruction policies: ALWAYS_DELETE (default), ARCHIVE_ON_DATE · f3954c71
      gmantele authored
      and ALWAYS_ARCHIVE.
      
      When archiving a job, its former phase is stored in jobInfo under the name
      'oldPhase' if no jobInfo is already set.
      
      Archiving a job means that all input files and results are destroyed ; the
      error summary and jobInfo (even if it is a file) are kept.
      
      Each archive operation ends with a log message ; in ERROR if failed or in
      INFO if successful.
      
      This commit also includes the following things:
      
          - reformat on 80 characters width the Javadoc of all modified classes
      
      	- fix a bug with the phase transitions: since it is not possible any more
      	  to go from PENDING to EXECUTING directly, UWSJob.start(...) must first
      	  ensure to be on QUEUED phase. This bug has also been fixed in TAPJob.
      	  Similarly, before going into ARCHIVED phase the job must be set into
      	  ABORTED phase if not already in a final phase.
      f3954c71
    • gmantele's avatar
      [UWS] Add the new Execution Phase `ARCHIVED` and check phase transitions. · 0467dbb1
      gmantele authored
      A JUnit test case has been added in order to check that all possible phase
      transitions are respecting the UWS-1.1 standard. However, there is anyway
      a bit more freedom for some of them:
      
          - it is possible to go to and come from UNKNOWN at any time, whatever is
      	  the source or target phase.
      
      	- it is possible to go to ERROR or ABORTED from the phases HELD and
      	  SUSPENDED. This fact was not specified in the State Machine figure of the
      	  UWS standard but the following sentence at section
      	  "2.1.3 Execution Phase" (page 7) should allow that:
      	  "At any time before the COMPLETED phase a job may either be ABORTED or
      	   may suffer an ERROR."
      
      	- the UWS-1.1 document has an inconsistency about the HELD phase. At
      	  section "2.1.3 Execution Phase" (page 7), the following sentence implies
      	  that it is only possible to go to HELD from PENDING (because it would not
      	  be possible to queue the job). And so, when PHASE=RUN is sent by the UWS
            client, if now possible, the job should go in phase QUEUED. However the
      	  State Machine figure suggests that it is possible to go to HELD only from
      	  EXECUTING and that a PHASE=RUN would make the job go back to EXECUTING (if
      	  now possible). Because of this inconsistency, the UWSLibrary made possible
      	  the following transitions: PENDING/EXECUTING->HELD->QUEUED/EXECUTING.
      
      (note: a figure illustrating the phase transitions supported by the
             UWSLibrary-4.3 has been created in the directory `img` of the
      	   UWS-Tutorial website under the file name `state_machine.png`...which of
      	   course will be visible only when uwslib-4.3 will be released)
      
      Besides, this commit also include almost a full rewriting of the Javadoc of
      JobPhase and ExecutionPhase. The Javadoc of UWSJob has just been reformated
      so that comments do not exceed 80 (+2) characters. This reformating aims to
      improve the human reading of the Javadoc while looking at the source files ;
      however this should not affect much the HTML version of the Javadoc.
      0467dbb1
    • gmantele's avatar
      [UWS,TAP] Add the property 'creationTime' to a UWS (and so TAP as well) job. · a8d98f64
      gmantele authored
      This property is displayed in the XML and JSON serialization of a job item.
      It is also backuped and restored from a UWS backup file.
      
      /!\ WARNING: Small modification of the API! Since creationTime must be
                   set by the UWS service at creation of a job, this date must also
                   be imported at backup restoration. Hence the modification of the
                   interface uws.service.UWSFactory and its implementations
                   (particularly tap.AbstractTAPFactory). Similarly the class
                   uws.job.UWSJob and tap.job.TAPJob have underwent exactly the same
                   modification in one of their constructors.
      
      This commit also adds the new optional jobRef items: creationTime, runID and
      ownerID in the XML and JSON serializations.
      a8d98f64
    • gmantele's avatar
      [UWS] Add the UWS protocol version as attributes of the nodes <uws>, <jobs> and · 15f2fb81
      gmantele authored
      <job>.
      
      This commit also:
      
          - set the version of the UWS protocol to 1.1 (cf UWS.VERSION)
      	- add a static constant for the standardID (cf UWS.STANDARD_ID)
      	- add a xlink:type and xlink:href for jobListRef in the uws XML
      	- set the @version Javadoc attribute of all modified classes to 4.3
      	  (scheduled version of the UWSLibrary supporting UWS-1.1)
      15f2fb81
    • gmantele's avatar
      [UWS] Ensure optional MIME-type and size are set in the XML and JSON · 404db993
      gmantele authored
      serialization of all results if such information are available.
      
      This commit also fix few comments about the result's XML serialization.
      
      The processing of xlink:type of a result reference is made similar as the one of
      jobRef.
      404db993
    • gmantele's avatar
      [TAP] Add an optional parameter to a UDF property: the UDF description. · f6a089c1
      gmantele authored
      Although the Java code allowed the specification of a description of a User
      Defined Function, it was not possible to set one in the UDFs listed in the
      configuration file.
      f6a089c1
  17. Feb 22, 2018
  18. Feb 08, 2018
  19. Jan 31, 2018
  20. Jan 12, 2018
    • gmantele's avatar
      [TAP] Complete the bug fix a0e5c56f for MySQL in JDBCConnection · f496d64e
      gmantele authored
      Now, JDBCConnection is fully capable to test the existence of schemas, tables
      and columns of a MySQL database. Thanks to @zonia3000 for solving this issue.
      f496d64e
    • gmantele's avatar
      [ADQL] Fix the parsing and translation of a concatenation expression. · e4f38c95
      gmantele authored
      * The parsing did not allow unsigned numerics and SQL SET functions as
        specified in the ADQL 2.0 grammar
      
      * It was even forbidden to put a column whose the type is not String.
      
      * The translation of a concatenation expression was always prefixed by the
        ADQLList's name: CONCAT_STR. Of course, no database likes that...
      
      Regarding this last point, this commit fixes the GitHub issue #54
      e4f38c95
  21. Jan 11, 2018
  22. Jan 08, 2018