Newer
Older
package tap.db;
/*
* This file is part of TAPLibrary.
*
* TAPLibrary is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* TAPLibrary is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with TAPLibrary. If not, see <http://www.gnu.org/licenses/>.
*
gmantele
committed
* Copyright 2012-2015 - UDS/Centre de Données astronomiques de Strasbourg (CDS),
* Astronomisches Rechen Institut (ARI)
*/
import java.sql.Connection;
import java.sql.DatabaseMetaData;
gmantele
committed
import java.sql.Driver;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
import java.text.ParseException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
gmantele
committed
import java.util.Properties;
import tap.data.DataReadException;
import tap.data.ResultSetTableIterator;
import tap.data.TableIterator;
import tap.log.TAPLog;
import tap.metadata.TAPColumn;
import tap.metadata.TAPForeignKey;
import tap.metadata.TAPMetadata;
import tap.metadata.TAPMetadata.STDSchema;
import tap.metadata.TAPMetadata.STDTable;
import tap.metadata.TAPSchema;
import tap.metadata.TAPTable;
import tap.metadata.TAPTable.TableType;
import uws.ISO8601Format;
import uws.service.log.UWSLog.LogLevel;
import adql.db.DBColumn;
import adql.db.DBType;
import adql.db.DBType.DBDatatype;
import adql.db.STCS;
import adql.db.STCS.Region;
import adql.query.ADQLQuery;
import adql.query.IdentifierField;
import adql.translator.ADQLTranslator;
import adql.translator.JDBCTranslator;
import adql.translator.TranslationException;
* <p>This {@link DBConnection} implementation is theoretically able to deal with any DBMS JDBC connection.</p>
*
* <p><i>Note:
* "Theoretically", because its design has been done using information about Postgres, SQLite, Oracle, MySQL and Java DB (Derby).
* Then it has been really tested successfully with Postgres and SQLite.
* </i></p>
*
* <h3>Deal with different DBMS features</h3>
*
* <p>Update queries are taking into account whether the following features are supported by the DBMS:</p>
* <ul>
* <li><b>data definition</b>: when not supported, no update operation will be possible.
* All corresponding functions will then throw a {@link DBException} ;
* only {@link #executeQuery(ADQLQuery)} will be possibly called.</li>
*
* <li><b>transactions</b>: when not supported, no transaction is started or merely used.
* It means that in case of update failure, no rollback will be possible
* and that already done modification will remain in the database.</li>
*
* <li><b>schemas</b>: when the DBMS does not have the notion of schema (like SQLite), no schema creation or dropping will be obviously processed.
* Besides, if not already done, database name of all tables will be prefixed by the schema name.</li>
*
* <li><b>batch updates</b>: when not supported, updates will just be done, "normally, one by one.
* In one word, there will be merely no optimization.
* Anyway, this feature concerns only the insertions into tables.</li>
*
* <li><b>case sensitivity of identifiers</b>: the case sensitivity of quoted identifier varies from the used DBMS. This {@link DBConnection}
* implementation is able to adapt itself in function of the way identifiers are stored and
* researched in the database. How the case sensitivity is managed by the DBMS is the problem
* of only one function (which can be overwritten if needed): {@link #equals(String, String, boolean)}.</li>
* </ul>
*
* <p><i><b>Warning</b>:
* All these features have no impact at all on ADQL query executions ({@link #executeQuery(ADQLQuery)}).
* </i></p>
*
* <h3>Datatypes</h3>
*
* <p>
* All datatype conversions done while fetching a query result (via a {@link ResultSet})
* are done exclusively by the returned {@link TableIterator} (so, here {@link ResultSetTableIterator}).
* </p>
*
* <p>
* However, datatype conversions done while uploading a table are done here by the function
* {@link #convertTypeToDB(DBType)}. This function uses first the conversion function of the translator
* ({@link JDBCTranslator#convertTypeToDB(DBType)}), and then {@link #defaultTypeConversion(DBType)}
* if it fails.
* </p>
*
* <p>
* In this default conversion, all typical DBMS datatypes are taken into account, <b>EXCEPT the geometrical types</b>
* (POINT and REGION). That's why it is recommended to use a translator in which the geometrical types are supported
* and managed.
* </p>
* <h3>Fetch size</h3>
*
* <p>
* The possibility to specify a "fetch size" to the JDBC driver (and more exactly to a {@link Statement}) may reveal
* very helpful when dealing with large datasets. Thus, it is possible to fetch rows by block of a size represented
* by this "fetch size". This is also possible with this {@link DBConnection} thanks to the function {@link #setFetchSize(int)}.
* </p>
*
* <p>
* However, some JDBC driver or DBMS may not support this feature. In such case, it is then automatically disabled by
* {@link JDBCConnection} so that any subsequent queries do not attempt to use it again. The {@link #supportsFetchSize}
* is however reset to <code>true</code> when {@link #setFetchSize(int)} is called.
* </p>
*
* <p><i>Note 1:
* The "fetch size" feature is used only for SELECT queries executed by {@link #executeQuery(ADQLQuery)}. In all other functions,
* results of SELECT queries are fetched with the default parameter of the JDBC driver and its {@link Statement} implementation.
* </i></p>
*
* <p><i>Note 2:
* By default, this feature is disabled. So the default value of the JDBC driver is used.
* To enable it, a simple call to {@link #setFetchSize(int)} is enough, whatever is the given value.
* </i></p>
*
gmantele
committed
* @author Grégory Mantelet (CDS;ARI)
* @version 2.0 (06/2015)
* @since 2.0
public class JDBCConnection implements DBConnection {
/** DBMS name of PostgreSQL used in the database URL. */
protected final static String DBMS_POSTGRES = "postgresql";
/** DBMS name of SQLite used in the database URL. */
protected final static String DBMS_SQLITE = "sqlite";
/** DBMS name of MySQL used in the database URL. */
protected final static String DBMS_MYSQL = "mysql";
/** DBMS name of Oracle used in the database URL. */
protected final static String DBMS_ORACLE = "oracle";
gmantele
committed
/** Name of the database column giving the database name of a TAP column, table or schema. */
protected final static String DB_NAME_COLUMN = "dbname";
/** Connection ID (typically, the job ID). It lets identify the DB errors linked to the Job execution in the logs. */
protected final String ID;
/** JDBC connection (created and initialized at the creation of this {@link JDBCConnection} instance). */
protected final Connection connection;
/** The translator this connection must use to translate ADQL into SQL. It is also used to get information about the case sensitivity of all types of identifier (schema, table, column). */
protected final JDBCTranslator translator;
/** Object to use if any message needs to be logged. <i>note: this logger may be NULL. If NULL, messages will never be printed.</i> */
protected final TAPLog logger;
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
/* JDBC URL MANAGEMENT */
/** JDBC prefix of any database URL (for instance: jdbc:postgresql://127.0.0.1/myDB or jdbc:postgresql:myDB). */
public final static String JDBC_PREFIX = "jdbc";
/** Name (in lower-case) of the DBMS with which the connection is linked. */
protected final String dbms;
/* DBMS SUPPORTED FEATURES */
/** Indicate whether the DBMS supports transactions (start, commit, rollback and end). <i>note: If no transaction is possible, no transaction will be used, but then, it will never possible to cancel modifications in case of error.</i> */
protected boolean supportsTransaction;
/** Indicate whether the DBMS supports the definition of data (create, update, drop, insert into schemas and tables). <i>note: If not supported, it will never possible to create TAP_SCHEMA from given metadata (see {@link #setTAPSchema(TAPMetadata)}) and to upload/drop tables (see {@link #addUploadedTable(TAPTable, TableIterator)} and {@link #dropUploadedTable(TAPTable)}).</i> */
protected boolean supportsDataDefinition;
/** Indicate whether the DBMS supports several updates in once (using {@link Statement#addBatch(String)} and {@link Statement#executeBatch()}). <i>note: If not supported, every updates will be done one by one. So it is not really a problem, but just a loss of optimization.</i> */
protected boolean supportsBatchUpdates;
/** Indicate whether the DBMS has the notion of SCHEMA. Most of the DBMS has it, but not SQLite for instance. <i>note: If not supported, the DB table name will be prefixed by the DB schema name followed by the character "_". Nevertheless, if the DB schema name is NULL, the DB table name will never be prefixed.</i> */
protected boolean supportsSchema;
/* CASE SENSITIVITY SUPPORT */
/** Indicate whether UNquoted identifiers will be considered as case INsensitive and stored in mixed case by the DBMS. <i>note: If FALSE, unquoted identifiers will still be considered as case insensitive for the researches, but will be stored in lower or upper case (in function of {@link #lowerCaseUnquoted} and {@link #upperCaseUnquoted}). If none of these two flags is TRUE, the storage case will be though considered as mixed.</i> */
protected boolean supportsMixedCaseUnquotedIdentifier;
/** Indicate whether the unquoted identifiers are stored in lower case in the DBMS. */
protected boolean lowerCaseUnquoted;
/** Indicate whether the unquoted identifiers are stored in upper case in the DBMS. */
protected boolean upperCaseUnquoted;
/** Indicate whether quoted identifiers will be considered as case INsensitive and stored in mixed case by the DBMS. <i>note: If FALSE, quoted identifiers will be considered as case sensitive and will be stored either in lower, upper or in mixed case (in function of {@link #lowerCaseQuoted}, {@link #upperCaseQuoted} and {@link #mixedCaseQuoted}). If none of these three flags is TRUE, the storage case will be mixed case.</i> */
protected boolean supportsMixedCaseQuotedIdentifier;
/** Indicate whether the quoted identifiers are stored in lower case in the DBMS. */
protected boolean lowerCaseQuoted;
/** Indicate whether the quoted identifiers are stored in mixed case in the DBMS. */
protected boolean mixedCaseQuoted;
/** Indicate whether the quoted identifiers are stored in upper case in the DBMS. */
protected boolean upperCaseQuoted;
/* FETCH SIZE */
/** Special fetch size meaning that the JDBC driver is free to set its own guess for this value. */
public final static int IGNORE_FETCH_SIZE = 0;
/** Default fetch size.
* <i>Note 1: this value may be however ignored if the JDBC driver does not support this feature.</i>
* <i>Note 2: by default set to {@link #IGNORE_FETCH_SIZE}.</i> */
public final static int DEFAULT_FETCH_SIZE = IGNORE_FETCH_SIZE;
/** <p>Indicate whether the last fetch size operation works.</p>
* <p>By default, this attribute is set to <code>false</code>, meaning that the "fetch size" feature is
* disabled. To enable it, a simple call to {@link #setFetchSize(int)} is enough, whatever is the given value.</p>
* <p>If just once this operation fails, the fetch size feature will be always considered as unsupported in this {@link JDBCConnection}
* until the next call of {@link #setFetchSize(int)}.</p> */
protected boolean supportsFetchSize = false;
/** <p>Fetch size to set in the {@link Statement} in charge of executing a SELECT query.</p>
* <p><i>Note 1: this value must always be positive. If negative or null, it will be ignored and the {@link Statement} will keep its default behavior.</i></p>
* <p><i>Note 2: if this feature is enabled (i.e. has a value > 0), the AutoCommit will be disabled.</i></p> */
protected int fetchSize = DEFAULT_FETCH_SIZE;
* <p>Creates a JDBC connection to the specified database and with the specified JDBC driver.
* This connection is established using the given user name and password.<p>
*
* <p><i><u>note:</u> the JDBC driver is loaded using <pre>Class.forName(driverPath)</pre> and the connection is created with <pre>DriverManager.getConnection(dbUrl, dbUser, dbPassword)</pre>.</i></p>
*
* <p><i><b>Warning:</b>
* This constructor really creates a new SQL connection. Creating a SQL connection is time consuming!
* That's why it is recommended to use a pool of connections. When doing so, you should use the other constructor of this class
* ({@link #JDBCConnection(Connection, JDBCTranslator, String, TAPLog)}).
* </i></p>
*
* @param driverPath Full class name of the JDBC driver.
* @param dbUrl URL to the database. <i><u>note</u> This URL may not be prefixed by "jdbc:". If not, the prefix will be automatically added.</i>
* @param dbUser Name of the database user.
* @param dbPassword Password of the given database user.
* @param translator {@link ADQLTranslator} to use in order to get SQL from an ADQL query and to get qualified DB table names.
* @param connID ID of this connection. <i>note: may be NULL ; but in this case, logs concerning this connection will be more difficult to localize.</i>
* @param logger Logger to use in case of need. <i>note: may be NULL ; in this case, error will never be logged, but sometimes DBException may be raised.</i>
*
* @throws DBException If the driver can not be found or if the connection can not merely be created (usually because DB parameters are wrong).
*/
public JDBCConnection(final String driverPath, final String dbUrl, final String dbUser, final String dbPassword, final JDBCTranslator translator, final String connID, final TAPLog logger) throws DBException{
this(createConnection(driverPath, dbUrl, dbUser, dbPassword), translator, connID, logger);
}
/**
* Create a JDBC connection by wrapping the given connection.
*
* @param conn Connection to wrap.
* @param translator {@link ADQLTranslator} to use in order to get SQL from an ADQL query and to get qualified DB table names.
* @param connID ID of this connection. <i>note: may be NULL ; but in this case, logs concerning this connection will be more difficult to localize.</i>
* @param logger Logger to use in case of need. <i>note: may be NULL ; in this case, error will never be logged, but sometimes DBException may be raised.</i>
public JDBCConnection(final Connection conn, final JDBCTranslator translator, final String connID, final TAPLog logger) throws DBException{
if (conn == null)
throw new NullPointerException("Missing SQL connection! => can not create a JDBCConnection object.");
if (translator == null)
throw new NullPointerException("Missing ADQL translator! => can not create a JDBCConnection object.");
this.connection = conn;
this.translator = translator;
this.ID = connID;
this.logger = logger;
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
// Set the supporting features' flags + DBMS type:
try{
DatabaseMetaData dbMeta = connection.getMetaData();
dbms = getDBMSName(dbMeta.getURL());
supportsTransaction = dbMeta.supportsTransactions();
supportsBatchUpdates = dbMeta.supportsBatchUpdates();
supportsDataDefinition = dbMeta.supportsDataDefinitionAndDataManipulationTransactions();
supportsSchema = dbMeta.supportsSchemasInTableDefinitions();
lowerCaseUnquoted = dbMeta.storesLowerCaseIdentifiers();
upperCaseUnquoted = dbMeta.storesUpperCaseIdentifiers();
supportsMixedCaseUnquotedIdentifier = dbMeta.supportsMixedCaseIdentifiers();
lowerCaseQuoted = dbMeta.storesLowerCaseQuotedIdentifiers();
mixedCaseQuoted = dbMeta.storesMixedCaseQuotedIdentifiers();
upperCaseQuoted = dbMeta.storesUpperCaseQuotedIdentifiers();
supportsMixedCaseQuotedIdentifier = dbMeta.supportsMixedCaseQuotedIdentifiers();
}catch(SQLException se){
throw new DBException("Unable to access to one or several DB metadata (url, supportsTransaction, supportsBatchUpdates, supportsDataDefinitionAndDataManipulationTransactions, supportsSchemasInTableDefinitions, storesLowerCaseIdentifiers, storesUpperCaseIdentifiers, supportsMixedCaseIdentifiers, storesLowerCaseQuotedIdentifiers, storesMixedCaseQuotedIdentifiers, storesUpperCaseQuotedIdentifiers and supportsMixedCaseQuotedIdentifiers) from the given Connection!");
}
}
/**
* Extract the DBMS name from the given database URL.
*
* @param dbUrl JDBC URL to access the database. <b>This URL must start with "jdbc:" ; otherwise an exception will be thrown.</b>
*
* @return The DBMS name as found in the given URL.
*
* @throws DBException If NULL has been given, if the URL is not a JDBC one (starting with "jdbc:") or if the DBMS name is missing.
*/
protected static final String getDBMSName(String dbUrl) throws DBException{
if (dbUrl == null)
throw new DBException("Missing database URL!");
if (!dbUrl.startsWith(JDBC_PREFIX + ":"))
throw new DBException("This DBConnection implementation is only able to deal with JDBC connection! (the DB URL must start with \"" + JDBC_PREFIX + ":\" ; given url: " + dbUrl + ")");
dbUrl = dbUrl.substring(5);
int indSep = dbUrl.indexOf(':');
if (indSep <= 0)
throw new DBException("Incorrect database URL: " + dbUrl);
return dbUrl.substring(0, indSep).toLowerCase();
}
/**
gmantele
committed
* Create a {@link Connection} instance using the given database parameters.
* The path of the JDBC driver will be used to load the adequate driver if none is found by default.
*
* @param driverPath Path to the JDBC driver.
* @param dbUrl JDBC URL to connect to the database. <i><u>note</u> This URL may not be prefixed by "jdbc:". If not, the prefix will be automatically added.</i>
* @param dbUser Name of the user to use to connect to the database.
* @param dbPassword Password of the user to use to connect to the database.
*
* @return A new DB connection.
*
* @throws DBException If the driver can not be found or if the connection can not merely be created (usually because DB parameters are wrong).
*
gmantele
committed
* @see DriverManager#getDriver(String)
* @see Driver#connect(String, Properties)
*/
private final static Connection createConnection(final String driverPath, final String dbUrl, final String dbUser, final String dbPassword) throws DBException{
gmantele
committed
// Normalize the DB URL:
gmantele
committed
String url = dbUrl.startsWith(JDBC_PREFIX) ? dbUrl : (JDBC_PREFIX + dbUrl);
gmantele
committed
// Select the JDBDC driver:
Driver d;
gmantele
committed
try{
gmantele
committed
d = DriverManager.getDriver(dbUrl);
}catch(SQLException e){
try{
// ...load it, if necessary:
if (driverPath == null)
throw new DBException("Missing JDBC driver path! Since the required JDBC driver is not yet loaded, this path is needed to load it.");
Class.forName(driverPath);
// ...and try again:
d = DriverManager.getDriver(dbUrl);
}catch(ClassNotFoundException cnfe){
throw new DBException("Impossible to find the JDBC driver \"" + driverPath + "\" !", cnfe);
}catch(SQLException se){
throw new DBException("No suitable JDBC driver found for the database URL \"" + dbUrl + "\" and the driver path \"" + driverPath + "\"!", se);
}
}
// Build a connection to the specified database:
gmantele
committed
try{
gmantele
committed
Properties p = new Properties();
gmantele
committed
if (dbUser != null)
p.setProperty("user", dbUser);
if (dbPassword != null)
p.setProperty("password", dbPassword);
gmantele
committed
Connection con = d.connect(url, p);
return con;
gmantele
committed
}catch(SQLException se){
throw new DBException("Impossible to establish a connection to the database \"" + url + "\"!", se);
gmantele
committed
@Override
gmantele
committed
public final String getID(){
return ID;
}
/**
* <p>Get the JDBC connection wrapped by this {@link JDBCConnection} object.</p>
*
* <p><i>Note:
* This is the best way to get the JDBC connection in order to properly close it.
* </i></p>
*
* @return The wrapped JDBC connection.
*/
public final Connection getInnerConnection(){
return connection;
/* ********************* */
/* INTERROGATION METHODS */
/* ********************* */
gmantele
committed
@Override
public TableIterator executeQuery(final ADQLQuery adqlQuery) throws DBException{
String sql = null;
ResultSet result = null;
gmantele
committed
try{
// 1. Translate the ADQL query into SQL:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "TRANSLATE", "Translating ADQL: " + adqlQuery.toADQL().replaceAll("(\t|\r?\n)+", " "), null);
sql = translator.translate(adqlQuery);
// 2. Create the statement and if needed, configure it for the given fetch size:
if (supportsFetchSize && fetchSize > 0){
try{
connection.setAutoCommit(false);
}catch(SQLException se){
supportsFetchSize = false;
if (logger != null)
logger.logDB(LogLevel.WARNING, this, "RESULT", "Fetch size unsupported!", null);
}
}
Statement stmt = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
if (supportsFetchSize){
try{
stmt.setFetchSize(fetchSize);
}catch(SQLException se){
supportsFetchSize = false;
if (logger != null)
logger.logDB(LogLevel.WARNING, this, "RESULT", "Fetch size unsupported!", null);
}
}
// 3. Execute the SQL query:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "EXECUTE", "SQL query: " + sql.replaceAll("(\t|\r?\n)+", " "), null);
result = stmt.executeQuery(sql);
// 4. Return the result through a TableIterator object:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "RESULT", "Returning result (" + (supportsFetchSize ? "fetch size = " + fetchSize : "all in once") + ").", null);
return createTableIterator(result, adqlQuery.getResultingColumns());
gmantele
committed
}catch(SQLException se){
close(result);
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "EXECUTE", "Unexpected error while EXECUTING SQL query!", null);
throw new DBException("Unexpected error while executing a SQL query: " + se.getMessage(), se);
}catch(TranslationException te){
close(result);
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "TRANSLATE", "Unexpected error while TRANSLATING ADQL into SQL!", null);
throw new DBException("Unexpected error while translating ADQL into SQL: " + te.getMessage(), te);
}catch(DataReadException dre){
close(result);
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "RESULT", "Unexpected error while reading the query result!", null);
throw new DBException("Impossible to read the query result, because: " + dre.getMessage(), dre);
}
}
/**
* Create a {@link TableIterator} instance which lets reading the given result table.
*
* @param rs Result of an SQL query.
* @param resultingColumns Metadata corresponding to each columns of the result.
*
* @return A {@link TableIterator} instance.
*
* @throws DataReadException If the metadata (columns count and types) can not be fetched
* or if any other error occurs.
*/
protected TableIterator createTableIterator(final ResultSet rs, final DBColumn[] resultingColumns) throws DataReadException{
return new ResultSetTableIterator(rs, translator, dbms, resultingColumns);
}
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
/* *********************** */
/* TAP_SCHEMA MANIPULATION */
/* *********************** */
/**
* Tell when, compared to the other TAP standard tables, a given standard TAP table should be created.
*
* @param table Standard TAP table.
*
* @return An index between 0 and 4 (included) - 0 meaning the first table to create whereas 4 is the last one.
* -1 is returned if NULL is given in parameter of if the standard table is not taken into account here.
*/
protected int getCreationOrder(final STDTable table){
if (table == null)
return -1;
switch(table){
case SCHEMAS:
return 0;
case TABLES:
return 1;
case COLUMNS:
return 2;
case KEYS:
return 3;
case KEY_COLUMNS:
return 4;
default:
return -1;
/* ************************************ */
/* GETTING TAP_SCHEMA FROM THE DATABASE */
/* ************************************ */
/**
* <p>In this implementation, this function is first creating a virgin {@link TAPMetadata} object
* that will be filled progressively by calling the following functions:</p>
* <ol>
* <li>{@link #loadSchemas(TAPTable, TAPMetadata, Statement)}</li>
* <li>{@link #loadTables(TAPTable, TAPMetadata, Statement)}</li>
* <li>{@link #loadColumns(TAPTable, List, Statement)}</li>
* <li>{@link #loadKeys(TAPTable, TAPTable, List, Statement)}</li>
* </ol>
*
* <p><i>Note:
* If schemas are not supported by this DBMS connection, the DB name of all tables will be set to NULL
* and the DB name of all tables will be prefixed by the ADQL name of their respective schema.
* </i></p>
*
* @see tap.db.DBConnection#getTAPSchema()
*/
gmantele
committed
@Override
public TAPMetadata getTAPSchema() throws DBException{
// Build a virgin TAP metadata:
TAPMetadata metadata = new TAPMetadata();
// Get the definition of the standard TAP_SCHEMA tables:
gmantele
committed
TAPSchema tap_schema = TAPMetadata.getStdSchema(supportsSchema);
// LOAD ALL METADATA FROM THE STANDARD TAP TABLES:
Statement stmt = null;
gmantele
committed
try{
// create a common statement for all loading functions:
stmt = connection.createStatement();
// load all schemas from TAP_SCHEMA.schemas:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "LOAD_TAP_SCHEMA", "Loading TAP_SCHEMA.schemas.", null);
loadSchemas(tap_schema.getTable(STDTable.SCHEMAS.label), metadata, stmt);
// load all tables from TAP_SCHEMA.tables:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "LOAD_TAP_SCHEMA", "Loading TAP_SCHEMA.tables.", null);
List<TAPTable> lstTables = loadTables(tap_schema.getTable(STDTable.TABLES.label), metadata, stmt);
// load all columns from TAP_SCHEMA.columns:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "LOAD_TAP_SCHEMA", "Loading TAP_SCHEMA.columns.", null);
loadColumns(tap_schema.getTable(STDTable.COLUMNS.label), lstTables, stmt);
// load all foreign keys from TAP_SCHEMA.keys and TAP_SCHEMA.key_columns:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "LOAD_TAP_SCHEMA", "Loading TAP_SCHEMA.keys and TAP_SCHEMA.key_columns.", null);
loadKeys(tap_schema.getTable(STDTable.KEYS.label), tap_schema.getTable(STDTable.KEY_COLUMNS.label), lstTables, stmt);
gmantele
committed
}catch(SQLException se){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to create a Statement!", se);
throw new DBException("Can not create a Statement!", se);
}finally{
close(stmt);
return metadata;
/**
* <p>Load into the given metadata all schemas listed in TAP_SCHEMA.schemas.</p>
*
* <p><i>Note:
* If schemas are not supported by this DBMS connection, the DB name of the loaded schemas is set to NULL.
* </i></p>
*
* @param tableDef Definition of the table TAP_SCHEMA.schemas.
* @param metadata Metadata to fill with all found schemas.
* @param stmt Statement to use in order to interact with the database.
*
* @throws DBException If any error occurs while interacting with the database.
*/
protected void loadSchemas(final TAPTable tableDef, final TAPMetadata metadata, final Statement stmt) throws DBException{
ResultSet rs = null;
gmantele
committed
try{
gmantele
committed
// Determine whether the dbName column exists:
/* note: if the schema notion is not supported by this DBMS, the column "dbname" is ignored. */
boolean hasDBName = supportsSchema && isColumnExisting(tableDef.getDBSchemaName(), tableDef.getDBName(), DB_NAME_COLUMN, connection.getMetaData());
// Build the SQL query:
StringBuffer sqlBuf = new StringBuffer("SELECT ");
sqlBuf.append(translator.getColumnName(tableDef.getColumn("schema_name")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("description")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("utype")));
gmantele
committed
if (hasDBName)
sqlBuf.append(", ").append(DB_NAME_COLUMN);
sqlBuf.append(" FROM ").append(translator.getTableName(tableDef, supportsSchema)).append(';');
// Execute the query:
rs = stmt.executeQuery(sqlBuf.toString());
// Create all schemas:
while(rs.next()){
gmantele
committed
String schemaName = rs.getString(1), description = rs.getString(2), utype = rs.getString(3), dbName = (hasDBName ? rs.getString(4) : null);
// create the new schema:
TAPSchema newSchema = new TAPSchema(schemaName, nullifyIfNeeded(description), nullifyIfNeeded(utype));
if (dbName != null && dbName.trim().length() > 0)
newSchema.setDBName(dbName);
// add the new schema inside the given metadata:
metadata.addSchema(newSchema);
}
gmantele
committed
}catch(SQLException se){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to load schemas from TAP_SCHEMA.schemas!", se);
throw new DBException("Impossible to load schemas from TAP_SCHEMA.schemas!", se);
}finally{
close(rs);
/**
* <p>Load into the corresponding metadata all tables listed in TAP_SCHEMA.tables.</p>
*
* <p><i>Note:
* Schemas are searched in the given metadata by their ADQL name and case sensitively.
* If they can not be found a {@link DBException} is thrown.
* </i></p>
*
* <p><i>Note:
* If schemas are not supported by this DBMS connection, the DB name of the loaded
* {@link TAPTable}s is prefixed by the ADQL name of their respective schema.
* </i></p>
*
* @param tableDef Definition of the table TAP_SCHEMA.tables.
* @param metadata Metadata (containing already all schemas listed in TAP_SCHEMA.schemas).
* @param stmt Statement to use in order to interact with the database.
*
* @return The complete list of all loaded tables. <i>note: this list is required by {@link #loadColumns(TAPTable, List, Statement)}.</i>
*
* @throws DBException If a schema can not be found, or if any other error occurs while interacting with the database.
*/
protected List<TAPTable> loadTables(final TAPTable tableDef, final TAPMetadata metadata, final Statement stmt) throws DBException{
ResultSet rs = null;
gmantele
committed
// Determine whether the dbName column exists:
boolean hasDBName = isColumnExisting(tableDef.getDBSchemaName(), tableDef.getDBName(), DB_NAME_COLUMN, connection.getMetaData());
// Build the SQL query:
StringBuffer sqlBuf = new StringBuffer("SELECT ");
sqlBuf.append(translator.getColumnName(tableDef.getColumn("schema_name")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("table_name")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("table_type")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("description")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("utype")));
gmantele
committed
if (hasDBName)
sqlBuf.append(", ").append(DB_NAME_COLUMN);
sqlBuf.append(" FROM ").append(translator.getTableName(tableDef, supportsSchema)).append(';');
// Execute the query:
rs = stmt.executeQuery(sqlBuf.toString());
// Create all tables:
ArrayList<TAPTable> lstTables = new ArrayList<TAPTable>();
while(rs.next()){
gmantele
committed
String schemaName = rs.getString(1), tableName = rs.getString(2), typeStr = rs.getString(3), description = rs.getString(4), utype = rs.getString(5), dbName = (hasDBName ? rs.getString(6) : null);
// get the schema:
TAPSchema schema = metadata.getSchema(schemaName);
if (schema == null){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to find the schema of the table \"" + tableName + "\": \"" + schemaName + "\"!", null);
throw new DBException("Impossible to find the schema of the table \"" + tableName + "\": \"" + schemaName + "\"!");
}
gmantele
committed
// If the table name is qualified, check its prefix (it must match to the schema name):
int endPrefix = tableName.indexOf('.');
if (endPrefix >= 0){
if (endPrefix == 0)
throw new DBException("Incorrect table name syntax: \"" + tableName + "\"! Missing schema name (before '.').");
else if (endPrefix == tableName.length() - 1)
throw new DBException("Incorrect table name syntax: \"" + tableName + "\"! Missing table name (after '.').");
else if (schemaName == null)
throw new DBException("Incorrect schema prefix for the table \"" + tableName.substring(endPrefix + 1) + "\": this table is not in a schema, according to the column \"schema_name\" of TAP_SCHEMA.tables!");
else if (!tableName.substring(0, endPrefix).trim().equalsIgnoreCase(schemaName))
throw new DBException("Incorrect schema prefix for the table \"" + schemaName + "." + tableName.substring(tableName.indexOf('.') + 1) + "\": " + tableName + "! Mismatch between the schema specified in prefix of the column \"table_name\" and in the column \"schema_name\".");
}
// resolve the table type (if any) ; by default, it will be "table":
TableType type = TableType.table;
if (typeStr != null){
try{
type = TableType.valueOf(typeStr.toLowerCase());
}catch(IllegalArgumentException iae){}
}
// create the new table:
TAPTable newTable = new TAPTable(tableName, type, nullifyIfNeeded(description), nullifyIfNeeded(utype));
gmantele
committed
newTable.setDBName(dbName);
// add the new table inside its corresponding schema:
schema.addTable(newTable);
lstTables.add(newTable);
}
return lstTables;
}catch(SQLException se){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to load tables from TAP_SCHEMA.tables!", se);
throw new DBException("Impossible to load tables from TAP_SCHEMA.tables!", se);
}finally{
close(rs);
/**
* <p>Load into the corresponding tables all columns listed in TAP_SCHEMA.columns.</p>
*
* <p><i>Note:
* Tables are searched in the given list by their ADQL name and case sensitively.
* If they can not be found a {@link DBException} is thrown.
* </i></p>
*
* @param tableDef Definition of the table TAP_SCHEMA.columns.
* @param lstTables List of all published tables (= all tables listed in TAP_SCHEMA.tables).
* @param stmt Statement to use in order to interact with the database.
*
* @throws DBException If a table can not be found, or if any other error occurs while interacting with the database.
*/
protected void loadColumns(final TAPTable tableDef, final List<TAPTable> lstTables, final Statement stmt) throws DBException{
ResultSet rs = null;
gmantele
committed
// Determine whether the dbName column exists:
boolean hasDBName = isColumnExisting(tableDef.getDBSchemaName(), tableDef.getDBName(), DB_NAME_COLUMN, connection.getMetaData());
// Build the SQL query:
StringBuffer sqlBuf = new StringBuffer("SELECT ");
sqlBuf.append(translator.getColumnName(tableDef.getColumn("table_name")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("column_name")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("description")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("unit")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("ucd")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("utype")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("datatype")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("size")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("principal")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("indexed")));
sqlBuf.append(", ").append(translator.getColumnName(tableDef.getColumn("std")));
gmantele
committed
if (hasDBName)
sqlBuf.append(", ").append(DB_NAME_COLUMN);
sqlBuf.append(" FROM ").append(translator.getTableName(tableDef, supportsSchema)).append(';');
// Execute the query:
rs = stmt.executeQuery(sqlBuf.toString());
// Create all tables:
while(rs.next()){
gmantele
committed
String tableName = rs.getString(1), columnName = rs.getString(2), description = rs.getString(3), unit = rs.getString(4), ucd = rs.getString(5), utype = rs.getString(6), datatype = rs.getString(7), dbName = (hasDBName ? rs.getString(12) : null);
int size = rs.getInt(8);
boolean principal = toBoolean(rs.getObject(9)), indexed = toBoolean(rs.getObject(10)), std = toBoolean(rs.getObject(11));
// get the table:
TAPTable table = searchTable(tableName, lstTables.iterator());
if (table == null){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to find the table of the column \"" + columnName + "\": \"" + tableName + "\"!", null);
throw new DBException("Impossible to find the table of the column \"" + columnName + "\": \"" + tableName + "\"!");
}
// resolve the column type (if any) ; by default, it will be "VARCHAR" if unknown or missing:
DBDatatype tapDatatype = null;
// ...try to resolve the datatype in function of all datatypes declared by the TAP standard.
if (datatype != null){
try{
tapDatatype = DBDatatype.valueOf(datatype.toUpperCase());
}catch(IllegalArgumentException iae){}
}
// ...build the column type:
DBType type;
if (tapDatatype == null)
type = new DBType(DBDatatype.VARCHAR);
else
type = new DBType(tapDatatype, size);
// create the new column:
TAPColumn newColumn = new TAPColumn(columnName, type, nullifyIfNeeded(description), nullifyIfNeeded(unit), nullifyIfNeeded(ucd), nullifyIfNeeded(utype));
newColumn.setPrincipal(principal);
newColumn.setIndexed(indexed);
newColumn.setStd(std);
gmantele
committed
newColumn.setDBName(dbName);
// add the new column inside its corresponding table:
table.addColumn(newColumn);
}
}catch(SQLException se){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to load columns from TAP_SCHEMA.columns!", se);
throw new DBException("Impossible to load columns from TAP_SCHEMA.columns!", se);
}finally{
close(rs);
/**
* <p>Load into the corresponding tables all keys listed in TAP_SCHEMA.keys and detailed in TAP_SCHEMA.key_columns.</p>
*
* <p><i>Note:
* Tables and columns are searched in the given list by their ADQL name and case sensitively.
* If they can not be found a {@link DBException} is thrown.
* </i></p>
*
* @param keysDef Definition of the table TAP_SCHEMA.keys.
* @param keyColumnsDef Definition of the table TAP_SCHEMA.key_columns.
* @param lstTables List of all published tables (= all tables listed in TAP_SCHEMA.tables).
* @param stmt Statement to use in order to interact with the database.
*
* @throws DBException If a table or a column can not be found, or if any other error occurs while interacting with the database.
*/
protected void loadKeys(final TAPTable keysDef, final TAPTable keyColumnsDef, final List<TAPTable> lstTables, final Statement stmt) throws DBException{
ResultSet rs = null;
PreparedStatement keyColumnsStmt = null;
// Prepare the query to get the columns of each key:
StringBuffer sqlBuf = new StringBuffer("SELECT ");
sqlBuf.append(translator.getColumnName(keyColumnsDef.getColumn("key_id")));
sqlBuf.append(", ").append(translator.getColumnName(keyColumnsDef.getColumn("from_column")));
sqlBuf.append(", ").append(translator.getColumnName(keyColumnsDef.getColumn("target_column")));
gmantele
committed
sqlBuf.append(" FROM ").append(translator.getTableName(keyColumnsDef, supportsSchema));
sqlBuf.append(" WHERE ").append(translator.getColumnName(keyColumnsDef.getColumn("key_id"))).append(" = ?").append(';');
keyColumnsStmt = connection.prepareStatement(sqlBuf.toString());
// Build the SQL query to get the keys:
sqlBuf.delete(0, sqlBuf.length());
sqlBuf.append("SELECT ").append(translator.getColumnName(keysDef.getColumn("key_id")));
sqlBuf.append(", ").append(translator.getColumnName(keysDef.getColumn("from_table")));
sqlBuf.append(", ").append(translator.getColumnName(keysDef.getColumn("target_table")));
sqlBuf.append(", ").append(translator.getColumnName(keysDef.getColumn("description")));
sqlBuf.append(", ").append(translator.getColumnName(keysDef.getColumn("utype")));
gmantele
committed
sqlBuf.append(" FROM ").append(translator.getTableName(keysDef, supportsSchema)).append(';');
// Execute the query:
rs = stmt.executeQuery(sqlBuf.toString());
// Create all foreign keys:
while(rs.next()){
String key_id = rs.getString(1), from_table = rs.getString(2), target_table = rs.getString(3), description = rs.getString(4), utype = rs.getString(5);
// get the two tables (source and target):
TAPTable sourceTable = searchTable(from_table, lstTables.iterator());
if (sourceTable == null){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to find the source table of the foreign key \"" + key_id + "\": \"" + from_table + "\"!", null);
throw new DBException("Impossible to find the source table of the foreign key \"" + key_id + "\": \"" + from_table + "\"!");
}
TAPTable targetTable = searchTable(target_table, lstTables.iterator());
if (targetTable == null){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to find the target table of the foreign key \"" + key_id + "\": \"" + target_table + "\"!", null);
throw new DBException("Impossible to find the target table of the foreign key \"" + key_id + "\": \"" + target_table + "\"!");
}
// get the list of columns joining the two tables of the foreign key:
HashMap<String,String> columns = new HashMap<String,String>();
ResultSet rsKeyCols = null;
try{
keyColumnsStmt.setString(1, key_id);
rsKeyCols = keyColumnsStmt.executeQuery();
while(rsKeyCols.next())
columns.put(rsKeyCols.getString(1), rsKeyCols.getString(2));
}catch(SQLException se){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to load key columns from TAP_SCHEMA.key_columns for the foreign key: \"" + key_id + "\"!", se);
throw new DBException("Impossible to load key columns from TAP_SCHEMA.key_columns for the foreign key: \"" + key_id + "\"!", se);
}finally{
close(rsKeyCols);
}
// create and add the new foreign key inside the source table:
try{
sourceTable.addForeignKey(key_id, targetTable, columns, nullifyIfNeeded(description), nullifyIfNeeded(utype));
}catch(Exception ex){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to create the foreign key \"" + key_id + "\" because: " + ex.getMessage(), ex);
throw new DBException("Impossible to create the foreign key \"" + key_id + "\" because: " + ex.getMessage(), ex);
}
}
}catch(SQLException se){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "LOAD_TAP_SCHEMA", "Impossible to load columns from TAP_SCHEMA.columns!", se);
throw new DBException("Impossible to load columns from TAP_SCHEMA.columns!", se);
}finally{
close(rs);
close(keyColumnsStmt);
/* ********************************** */
/* SETTING TAP_SCHEMA IN THE DATABASE */
/* ********************************** */
/**
* <p>This function is just calling the following functions:</p>
* <ol>
* <li>{@link #mergeTAPSchemaDefs(TAPMetadata)}</li>
* <li>{@link #startTransaction()}</li>
* <li>{@link #resetTAPSchema(Statement, TAPTable[])}</li>
* <li>{@link #createTAPSchemaTable(TAPTable, Statement)} for each standard TAP_SCHEMA table</li>
* <li>{@link #fillTAPSchema(TAPMetadata)}</li>
* <li>{@link #createTAPTableIndexes(TAPTable, Statement)} for each standard TA_SCHEMA table</li>
* <li>{@link #commit()} or {@link #rollback()}</li>
* <li>{@link #endTransaction()}</li>
* </ol>
*
* <p><i><b>Important note:
* If the connection does not support transactions, then there will be merely no transaction.
* Consequently, any failure (exception/error) will not clean the partial modifications done by this function.
* </i></p>
*
* @see tap.db.DBConnection#setTAPSchema(tap.metadata.TAPMetadata)
*/
gmantele
committed
@Override
public void setTAPSchema(final TAPMetadata metadata) throws DBException{
Statement stmt = null;
// A. GET THE DEFINITION OF ALL STANDARD TAP TABLES:
TAPTable[] stdTables = mergeTAPSchemaDefs(metadata);
startTransaction();
// B. RE-CREATE THE STANDARD TAP_SCHEMA TABLES:
stmt = connection.createStatement();
// 1. Ensure TAP_SCHEMA exists and drop all its standard TAP tables:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "CLEAN_TAP_SCHEMA", "Cleaning TAP_SCHEMA.", null);
resetTAPSchema(stmt, stdTables);
// 2. Create all standard TAP tables:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "CREATE_TAP_SCHEMA", "Creating TAP_SCHEMA tables.", null);
for(TAPTable table : stdTables)
createTAPSchemaTable(table, stmt);
// C. FILL THE NEW TABLE USING THE GIVEN DATA ITERATOR:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "CREATE_TAP_SCHEMA", "Filling TAP_SCHEMA tables.", null);
fillTAPSchema(metadata);
// D. CREATE THE INDEXES OF ALL STANDARD TAP TABLES:
if (logger != null)
logger.logDB(LogLevel.INFO, this, "CREATE_TAP_SCHEMA", "Creating TAP_SCHEMA tables' indexes.", null);
for(TAPTable table : stdTables)
createTAPTableIndexes(table, stmt);
commit();
}catch(SQLException se){
if (logger != null)
logger.logDB(LogLevel.ERROR, this, "CREATE_TAP_SCHEMA", "Impossible to SET TAP_SCHEMA in DB!", se);
rollback();
throw new DBException("Impossible to SET TAP_SCHEMA in DB!", se);
}finally{
close(stmt);
endTransaction();
}
}
/**
* <p>Merge the definition of TAP_SCHEMA tables given in parameter with the definition provided in the TAP standard.</p>
*
* <p>
* The goal is to get in output the list of all standard TAP_SCHEMA tables. But it must take into account the customized
* definition given in parameter if there is one. Indeed, if a part of TAP_SCHEMA is not provided, it will be completed here by the
* definition provided in the TAP standard. And so, if the whole TAP_SCHEMA is not provided at all, the returned tables will be those
* of the IVOA standard.
* </p>
*
* <p><i><b>Important note:</b>
* If the TAP_SCHEMA definition is missing or incomplete in the given metadata, it will be added or completed automatically
* by this function with the definition provided in the IVOA TAP standard.
* </i></p>
*
* <p><i>Note:
* Only the standard tables of TAP_SCHEMA are considered. The others are skipped (that's to say: never returned by this function ;
* however, they will stay in the given metadata).
* </i></p>
*
* <p><i>Note:
* If schemas are not supported by this DBMS connection, the DB name of schemas is set to NULL and
* the DB name of tables is prefixed by the schema name.
* </i></p>
* @param metadata Metadata (with or without TAP_SCHEMA schema or some of its table). <i>Must not be NULL</i>