SET DATADIR TO [ <directory> ]This command is used to specify a directory where database tables, memos, indexes, and dictionary files are located. When a table is being opened this directory is searched first before the current directory and the file search path to locate the table and its associated files. This allows the database tables to be relocated to a different file system without the need to change an existing application.
Recital is a proven and cost-effective database solution that will help reduce the cost of your database and application software infrastructure substantially. As an added benefit, Recital can run many legacy applications with little to no change as it understands FoxBASE, FoxPRO and Clipper languages as a subset of it's overall capability.
In this article Barry Mavin, CEO and Chief Software Architect for Recital, details how to use the Client Drivers provided with the Recital Database Server to work with local or remote server-side OLE DB data sources.
Overview
The Recital Universal .NET Data Provider provides connectivity to the Recital Database Server running on any supported platform (Windows, Linux, Unix, OpenVMS) using the RecitalConnection object.
The Recital Universal JDBC Driver provides the same functionality for java applications.
The Recital Universal ODBC Driver provides the same functionality for applications that use ODBC.
Each of the above Client Drivers use a connection string to describe connections parameters.
The basic format of a connection string consists of a series of keyword/value pairs separated by semicolons. The equal sign (=) connects each keyword and its value.
The following table lists the valid names for keyword/values.
| Name | Default | Description |
|---|---|---|
|
Data Source
|
The name or network address of the instance of the Recital Database Server which to connect to. | |
| Directory | The target directory on the remote server where data to be accessed resides. This is ignored when a Database is specified. | |
|
Encrypt |
false | When true, DES3 encryption is used for all data sent between the client and server. |
| Initial Catalog -or- Database |
The name of the database on the remote server. | |
| Password -or- Pwd |
The password used to authenticate access to the remote server. | |
| User ID | The user name used to authenticate access to the remote server. | |
|
Connection Pooling |
false | Enable connection pooling to the server. This provides for one connection to be shared. |
| Logging | false | Provides for the ability to log all server requests for debugging purposes |
| Rowid | true | When Rowid is true (the default) a column will be post-fixed to each SELECT query that is a unique row identifier. This is used to provide optimised UPDATE and DELETE operations. If you use the RecitalSqlGrid, RecitalSqlForm, or RecitalSqlGridForm components then this column is not visible but is used to handle updates to the underlying data source. |
| Logfile | The name of the logfile for logging | |
| Gateway |
Opens an SQL gateway(Connection) to a foreign SQL data source on the remote server. Using Gateways, you can transparently access the following local or remote data sources:
The gateway can be specified in several formats: servertype@nodename:username/password-database e.g. oracle@nodename:username/password-database mysql@nodename:username/password-database postgresql@nodename:username/password-database -or- odbc:odbc_data_source_name_on_server oledb:oledb_connection_string_on_server jdbc:jdbc_driver_path_on_server;jdbc:Recital:args |
To connect to a server-side OLE DB data source, you use the gateway=value key/value pair in the following way.
gateway=oledb:oledb_connection_string_on_server
ImportantWhen specifying the connection string be sure to quote the gateway= with "...".
You can find examples of connection strings for most ODBC and OLE DB data sources by clicking here.
Example in C# using the Recital Universal .NET Data Provider:
////////////////////////////////////////////////////////////////////////
// include the references below
using System.Data;
using Recital.Data;
////////////////////////////////////////////////////////////////////////
// The following code example creates an instance of a DataAdapter that
// uses a Connection to the Recital Database Server, and a gateway to
// the SQL server Northwind database. It then populates a DataTable
// in a DataSet with the list of customers. The SQL statement and
// Connection arguments passed to the DataAdapter constructor are used
// to create the SelectCommand property of the DataAdapter.
public DataSet SelectCustomers()
{
string gateway = "oledb:Provider=sqloledb;Initial Catalog=Northwind;
Data Source=localhost;Integrated Security=SSPI";
RecitalConnection swindConn = new
RecitalConnection("Data Source=localhost;gateway=\""+gateway+"\");
RecitalCommand selectCMD = new
RecitalCommand("SELECT CustomerID, CompanyName FROM Customers", swindConn);
selectCMD.CommandTimeout = 30;
RecitalDataAdapter custDA = new RecitalDataAdapter();
custDA.SelectCommand = selectCMD;
swindConn.Open();
DataSet custDS = new DataSet();
custDA.Fill(custDS, "Customers");
swindConn.Close();
return custDS;
}
Example in Java using the Recital Universal JDBC Driver:
////////////////////////////////////////////////////////////////////////
// standard imports required by the JDBC driver
import java.sql.*;
import java.io.*;
import java.net.URL;
import java.math.BigDecimal;
import Recital.sql.*;
////////////////////////////////////////////////////////////////////////
// The following code example creates a Connection to the Recital
// Database Server, and a gateway to the SQL server Northwind database.
// It then retrieves all the customers.
public void SelectCustomers()
{
// setup the Connection URL for JDBC
String gateway = "oledb:Provider=sqloledb;Initial Catalog=Northwind;
Data Source=localhost;Integrated Security=SSPI";
String url = "jdbc:Recital:Data Source=localhost;gateway=\""+gateway+"\";
// load the Recital Universal JDBC Driver
new RecitalDriver();
// create the connection
Connection con = DriverManager.getConnection(url);
// create the statement
Statement stmt = con.createStatement();
// perform the SQL query
ResultSet rs = stmt.executeQuery("SELECT CustomerID, CompanyName FROM Customers");
// fetch the data
while (rs.next())
{
String CompanyID = rs.getString("CustomerID");
String CompanyName = rs.getString("CompanyName");
// do something with the data...
}
// Release the statement
stmt.close();
// Disconnect from the server
con.close();
} In this article Barry Mavin, CEO and Chief Software Architect for Recital provides details on how to use the Recital Universal .NET Data Provider with the Recital Database Server.
Overview
A data provider in the .NET Framework serves as a bridge between an application and a data source. A data provider is used to retrieve data from a data source and to reconcile changes to that data back to the data source.
Each .NET Framework data provider has a DataAdapter object: the .NET Framework Data Provider for OLE DB is the OleDbDataAdapter object, the .NET Framework Data Provider for SQL Server is the SqlDataAdapter object, the .NET Framework Data Provider for ODBC is the OdbcDataAdapter object, and the .NET Framework Data Provider for the Recital Database Server is the RecitalDataAdapter object.
The Recital Universal .NET Data Provider can access any data sources supported by the Recital Database Server. It is not restricted to only access Recital data. It can be used to access server-side ODBC, JDBC and OLE DB data sources also.
Core classes of the Data Provider
The Connection, Command, DataReader, and DataAdapter objects represent the core elements of the .NET Framework data provider model. The Recital Universal .NET Data Provider is plug compatible with the .NET Framework Data Provider for SQL Server. All SQL Server classes are prefixed with "Sql" e.g. SqlDataAdaptor. To use the Recital Universal Data Adaptor, simply change the "Sql" prefix to "Recital" e.g. RecitalDataAdaptor.
The following table describes these objects.
| Object | Description |
|---|---|
| RecitalConnection | Establishes a connection to a specific data source. |
| RecitalCommand | Executes a command against a data source. |
| RecitalDataReader | Reads a forward-only, read-only stream of data from a data source. |
| RecitalDataAdapter | Populates a DataSet and resolves updates with the data source. |
Along with the core classes listed in the preceding table, a .NET Framework data provider also contains the classes listed in the following table.
| Object | Description |
|---|---|
| RecitalTransaction | Enables you to enlist commands in transactions at the data source. |
| RecitalCommandBuilder | A helper object that will automatically generate command properties of a DataAdapter or will derive parameter information from a stored procedure and populate the Parameters collection of a Command object. |
| RecitalParameter | Defines input, output, and return value parameters for commands and stored procedures. |
The Recital Universal .NET Data Provider provides connectivity to the Recital Database Server running on any supported platform (Windows, Linux, Unix, OpenVMS) using the RecitalConnection object. The Recital Universal .NET Data Provider supports a connection string format that is similar to the SQL Server connection string format.
The basic format of a connection string consists of a series of keyword/value pairs separated by semicolons. The equal sign (=) connects each keyword and its value.
The following table lists the valid names for keyword values within the ConnectionString property of the RecitalConnection class.
| Name | Default | Description |
|---|---|---|
| Data Source -or- Server -or- Servername -or- Nodename |
The name or network address of the instance of the Recital Database Server which to connect to. | |
| Directory | The target directory on the remote server where data to be accessed resides. This is ignored when a Database is specified. | |
| Encrypt -or- Encryption |
false | When true, DES3 encryption is used for all data sent between the client and server. |
| Initial Catalog -or- Database |
The name of the database on the remote server. | |
| Password -or- Pwd |
The password used to authenticate access to the remote server. | |
| User ID -or- uid -or- User -or- Username |
The user name used to authenticate access to the remote server. | |
| Connection Pooling -or- Pool |
false | Enable connection pooling to the server. This provides for one connection to be shared. |
| Logging | false | Provides for the ability to log all server requests for debugging purposes |
| Rowid | true | When Rowid is true (the default) a column will be post-fixed to each SELECT query that is a unique row identifier. This is used to provide optimised UPDATE and DELETE operations. If you use the RecitalSqlGrid, RecitalSqlForm, or RecitalSqlGridForm components then this column is not visible but is used to handle updates to the underlying data source. |
| Logfile | The name of the logfile for logging | |
| Gateway |
Opens an SQL gateway(Connection) to a foreign SQL data source on the remote server.
The gateway can be specified in several formats: |
Populating a DataSet from a DataAdaptor
The ADO.NET DataSet is a memory-resident representation of data that provides a consistent relational programming model independent of the data source. The DataSet represents a complete set of data including tables, constraints, and relationships among the tables. Because the DataSet is independent of the data source, a DataSet can include data local to the application, as well as data from multiple data sources. Interaction with existing data sources is controlled through the DataAdapter.
A DataAdapter is used to retrieve data from a data source and populate tables within a DataSet. The DataAdapter also resolves changes made to the DataSet back to the data source. The DataAdapter uses the Connection object of the .NET Framework data provider to connect to a data source and Command objects to retrieve data from and resolve changes to the data source.
The SelectCommand property of the DataAdapter is a Command object that retrieves data from the data source. The InsertCommand, UpdateCommand, and DeleteCommand properties of the DataAdapter are Command objects that manage updates to the data in the data source according to modifications made to the data in the DataSet.
The Fill method of the DataAdapter is used to populate a DataSet with the results of the SelectCommand of the DataAdapter. Fill takes as its arguments a DataSet to be populated, and a DataTable object, or the name of the DataTable to be filled with the rows returned from the SelectCommand.
The Fill method uses the DataReader object implicitly to return the column names and types used to create the tables in the DataSet, as well as the data to populate the rows of the tables in the DataSet. Tables and columns are only created if they do not already exist; otherwise Fill uses the existing DataSet schema.
Examples in C#:
////////////////////////////////////////////////////////////////////////
// include the references below
using System.Data;
using Recital.Data;
////////////////////////////////////////////////////////////////////////
// The following code example creates an instance of a DataAdapter that
// uses a Connection to the Recital Database Server Southwind database
// and populates a DataTable in a DataSet with the list of customers.
// The SQL statement and Connection arguments passed to the DataAdapter
// constructor are used to create the SelectCommand property of the DataAdapter.
public DataSet SelectCustomers()
{
RecitalConnection swindConn = new
RecitalConnection("Data Source=localhost;Initial Catalog=southwind");
RecitalCommand selectCMD = new
RecitalCommand("SELECT CustomerID, CompanyName FROM Customers", swindConn);
selectCMD.CommandTimeout = 30;
RecitalDataAdapter custDA = new RecitalDataAdapter();
custDA.SelectCommand = selectCMD;
swindConn.Open();
DataSet custDS = new DataSet();
custDA.Fill(custDS, "Customers");
swindConn.Close();
return custDS;
}
////////////////////////////////////////////////////////////////////////
// The following example uses the RecitalCommand, RecitalDataAdapter and
// RecitalConnection, to select records from a database, and populate a
// DataSet with the selected rows. The filled DataSet is then returned.
// To accomplish this, the method is passed an initialized DataSet, a
// connection string, and a query string that is a SQL SELECT statement
public DataSet SelectRecitalRows(DataSet dataset, string connection, string query)
{
RecitalConnection conn = new RecitalConnection(connection);
SqlDataAdapter adapter = new RecitalDataAdapter();
adapter.SelectCommand = new RecitalCommand(query, conn);
adapter.Fill(dataset);
return dataset;
} open database southwind
// open child table
use order_details order orderid in 0
// open parent table
use orders order orderid in 0
set relation to orderid into order_details
do while not eof()
? orders.orderid, order_details.productid
skip
enddo
The code above will display the productid from the first related record, but you will often want to display information from all the related records in the child or detail table as in an SQL Left Outer Join.
open database southwind
select orders.orderid, order_details.productid;
from orders left outer join order_details;
on orders.orderid = order_details.orderid
In this case, we can use a second nested DO WHILE loop, for example:
open database southwind
use order_details order orderid in 0
use orders order orderid in 0
set relation to orderid into order_details
do while not eof()
// Display first or 0 child record
? orders.orderid, order_details.productid
// Display any additional child records
do while not eof(order_details)
? orders.orderid, order_details.productid
skip in order_details
enddo
skip
enddo
Or we can use the SET SKIP command. The SET SKIP command can be used with DISPLAY, LIST and REPORT and automatically skips through all the related records in the child table.
open database southwind
use order_details order orderid in 0
use orders order orderid in 0
set relation to orderid into order_details
set skip on
set skip to order_details
list orders.orderid, order_details.productid
LIST and DISPLAY offer a number of keyword options to allow you to configure the display output. REPORT offers full column based report design.
A quick tip for optimizing TCP performance on linux.
edit /etc/sysctl.conf add the lines:
If using gigabit ethernet:
net.ipv4.tcp_mem= 98304 131072 196608
net.ipv4.tcp_window_scaling=1
net.core.wmem_default = 65536
net.core.rmem_default = 65536
net.core.wmem_max=8388608
To reload these use:
# sysctl -p
If using infiniband:
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=0
net.ipv4.tcp_rmem=10000000 10000000 10000000
net.ipv4.tcp_wmem=10000000 10000000 10000000
net.ipv4.tcp_mem=10000000 10000000 10000000
net.core.rmem_max=524287
net.core.wmem_max=524287
net.core.rmem_default=524287
net.core.wmem_default=524287
net.core.optmem_max=524287
net.core.netdev_max_backlog=300000
If when your attempt to create device meta-data fails this is drbd preventing you from corrupting a file system present on the target partition.
$ drbdadm create-md drbd0
v08 Magic number not found
md_offset 30005817344
al_offset 30005784576
bm_offset 30004867072
Found ext2 filesystem which uses 190804004 kB
current configuration leaves usable 29301628 kB
Device size would be truncated, which
would corrupt data and result in
'access beyond end of device' errors.
You need to either
* use external meta data (recommended)
* shrink that filesystem first
* zero out the device (destroy the filesystem)
Operation refused.
Command 'drbdmeta /dev/drbd0 v08 /dev/sda4 internal create-md' terminated with exit code 40
drbdadm aborting
Once you have confirmed that the file system present on the target partition is no longer required at the prompt type the following:
Replace /dev/sdaX with the block device you are targeting.
dd if=/dev/zero of=/dev/sdaX bs=1M count=128
Once this has completed the drbdadm create-md drbd0 command will complete with a "success."
$ drbdadm create-md drbd0
v08 Magic number not found
v07 Magic number not found
v07 Magic number not found
v08 Magic number not found
Writing meta data...
initialising activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
$
The REQUIRE() statement includes and executes the contents of the specified file at the current program execution level.
When a file is included, the code it contains inherits the variable scope of the line on which the include occurs. Any variables, procedures, functions or classes declared in the included file will be available at the current program execution level.
The REQUIRE_ONCE() statement is identical to the REQUIRE() statement except that Recital will check to see if the file as already been included and if so ignore the command.
The full syntax is:
REQUIRE( expC ) REQUIRE_ONCE( expC ) e.g. REQUIRE_ONCE( "myapp/myglobals.prg" )
This article looks at After Image Journaling and audit trails in Recital using SET JOURNAL and associated commands.
Overview
After Image Journaling, used in conjunction with a structured backup policy is an effective disaster recovery solution. Any transaction that takes place on a table that is being journaled is logged in the journal file. In the event of a disk crash or similar event in which the table is lost, the journaled transactions can be reapplied to the latest backup copy of the table. Alternatively or additionally, the journal can be used to provide an audit trail to all modifications made to the table data.
NOTE: Recital also provides Before Image Journaling via BEGIN TRANSACTION / END TRANSACTION blocks, allowing unsuccessful transactions to be rolled back to a set saved state.
SET JOURNAL and RECOVER
Regular backups are an essential routine for any system, but in high-transaction environments restoration of the latest backup can still mean a major loss of data. After image journaling can successfully be used as part of your disaster recovery strategy to minimize data loss and down time. Recital after image journaling functionality is based on the use of the SET JOURNAL and RECOVER commands.
SET JOURNAL
SET JOURNAL TO [<.dbj filename> | ()] SET JOURNAL ON | OFF | ( )
The SET JOURNAL command is used to enable the After Image Journaling and audit trail for the active table. The TO <.dbj filename> clause associates the specified transaction journal file with the active table. If the journal file does not exist, it will be created. The filename can be substituted with a <expC>, enclosed in round brackets, which returns a valid filename. If no file extension is specified, ‘.dbj’ is used. When specifying a journal file, it is recommended that the journal file is stored on a different disk than that which the table is stored on, so that if a fatal disk error occurs, then the journal file will not be lost along with the table.
//Enable journaling for the southwind!orders table open database southwind use orders set journal to /journals/ord_journ
The <.dbj filename> is a standard table. It contains seven
fields that are specific to a journal file, followed by the first
249 fields of the associated table.
The first seven fields in the journal are:
|
Field |
Type |
Display |
Storage |
Description |
|
AUD_DATE |
Date |
8 | 10 * |
4 |
The date on which the transaction was performed. |
|
AUD_TIME |
Character |
8 |
8 |
The time at which the transaction was performed, in the format HH:MM:SS. |
|
AUD_TERM |
Character |
12 |
12 |
The name of the terminal from which the transaction was performed |
|
AUD_UID |
Short |
5 |
2 |
The ID of the user who performed the transaction. |
|
AUD_GID |
Short |
5 |
2 |
The group ID of the user who performed the transaction. |
|
AUD_CMD |
Short |
4 |
2 |
The command number of the transaction performed from the command table below |
|
AUD_RECNO |
Integer |
7 |
4 |
The record number in the associated table which the transaction was performed on. |
* Dependent on SET CENTURY setting.
The AUD_CMD Command Reference Numbers are as follows:
|
Command |
Number |
|
DELETE |
14 |
|
RECALL |
36 |
|
REPLACE |
41 |
|
BROWSE |
6 |
|
CHANGE |
8 |
|
EDIT |
17 |
|
INSERT |
26 |
|
APPEND |
5 |
|
READ |
35 |
Since journal files are standard Recital tables, you can use standard Recital commands such as the REPORT command to print audit trails, transaction logs, etc.
//Enable journaling for the southwind!orders table open database southwind use orders set journal to /journals/ord_journ //.. transactions close data //View journaled records use /journals/ord_journ.dbj
Click image to display full size
Fig 1: Journal Record Example.
The SET JOURNAL TO command without a <.dbj filename> specified closes the active journal file and no further journaling will take place on the active table until the SET JOURNAL TO <.dbj filename> is reissued.
The journaling features are mainly used with shared tables. It should be noted that there is an overhead in enabling transaction journaling, as records updated in a table are also written to the journal file. When records are appended into a journal file, locking is automatically performed so that multiple users can update the journal concurrently. The associated table must be opened shareable for this to occur. Each table can have a journal file associated with it.
The SET JOURNAL ON | OFF command enables or disables transaction journaling. This command is primarily used in applications where journaling can be disabled for a certain class of operations. By default, SET JOURNAL is ON, but no journal files are set.
NOTE: Only the first 249 fields of a table can be journaled: subsequent fields are ignored. The maximum number of fields in a Recital table is 256.
RECOVER
RECOVER FROM <.dbj filename> | ()
The RECOVER command uses the journal file to reapply lost transactions to a previous backup of the data after a fatal error such as a disk head crash. The FROM clause specifies the journal file to use. The file name can be substituted with an <expC>, enclosed in round brackets, which returns a valid filename. If no file extension is specified, then ‘.dbj’ is assumed.
Regular backups are essential to the successful use of After Image Journaling. It is also very important to reinitialize the journal file after each backup: either open the journal file as you would a normal table and use the ZAP command, or delete the file completely. If a fatal error occurs, such as a disk head crash, the table and index files must be restored from a backup, then the RECOVER command executed. RECOVER will reapply' all of the transactions in the journal file to the table, and update the indexes. After the RECOVER command has completed, you can continue with normal processing.
//Create a backup of the southwind!orders table //...backup table and associated files //Reinitialize the journal file erase /journals/ord_journ.dbj //Enable journaling for the southwind!orders table open database southwind use orders set journal to /journals/ord_journ //.. transactions //Restore the backup of the southwind!orders table //...restore //Open the restored backup open database southwind use orders //Reapply the transactions using the journal recover from /journals/ord_journ.dbj //Now, enable the journal file again or //restart with a new backup
Journaling Memo Fields
By default, memo fields - variable length text fields - are not journaled due to the possible storage overhead of multiple copies of potentially large blocks of text. But, if memo journaling is required, the SET MEMOJOURNAL ON command can be used to enable this.
SET MEMOJOURNAL
SET MEMOJOURNAL ON | OFF | ()
The SET MEMOJOURNAL command causes memo fields to be journaled when journaling is set on a table. This command allows the optional logical expression <expL> to be evaluated. If a value of .T. is returned, MEMOJOURNAL is set ON. If a value of .F. is returned, MEMOJOURNAL is set OFF. By default SET MEMOJOURNAL is OFF.
Like a normal Recital table, the journal holds only a pointer to a data block in an associated memo file, not the actual memo data itself. The journal's memo file has a file extension of .dbm rather than the standard Recital .dbt. Therefore, if the journal is being opened as a table, in order to view the journal's memo data, the SET MEMOEXT command should be used.
//Enable journaling for the southwind!suppliers table open database southwind use suppliers set journal to /journals/sup_journ //.. transactions close data //Set filename extension for memo file set memoext to '.dbm' //View journaled records use /journals/sup_journ.dbj
Summary
The After Image Journaling enabled by the SET JOURNAL and RECOVER commands can be used in conjunction with a strict backup regime to minimize data loss in cases where tables become damaged or irretrievable. Journal files can be accessed like standard Recital tables and provide detailed information about the transactions applied to a table, so can be used for auditing purposes.