A quick tip for optimizing TCP performance on linux.
edit /etc/sysctl.conf add the lines:
If using gigabit ethernet:
net.ipv4.tcp_mem= 98304 131072 196608
net.ipv4.tcp_window_scaling=1
net.core.wmem_default = 65536
net.core.rmem_default = 65536
net.core.wmem_max=8388608
To reload these use:
# sysctl -p
If using infiniband:
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=0
net.ipv4.tcp_rmem=10000000 10000000 10000000
net.ipv4.tcp_wmem=10000000 10000000 10000000
net.ipv4.tcp_mem=10000000 10000000 10000000
net.core.rmem_max=524287
net.core.wmem_max=524287
net.core.rmem_default=524287
net.core.wmem_default=524287
net.core.optmem_max=524287
net.core.netdev_max_backlog=300000
./configure CFLAGS='-arch x86_64' APXSLDFLAGS='-arch x86_64' --with-apxs=/usr/sbin/apxsThen you must pass the these additional flags to the apxs command in order to generate a Universal Binary shared module.
-Wl,-dynamic -Wl,'-arch ppc' -Wl,'-arch ppc64' -Wl,'-arch i386' -Wl,'-arch x86_64' -Wc,-dynamic -Wc,'-arch ppc' -Wc,'-arch ppc64' -Wc,'-arch i386' -Wc,'-arch x86_64'If you then do a file command on the shared module it should return;
$ file mod_recital.so mod_recital2.2.so: Mach-O universal binary with 4 architectures mod_recital2.2.so (for architecture ppc7400): Mach-O bundle ppc mod_recital2.2.so (for architecture ppc64): Mach-O 64-bit bundle ppc64 mod_recital2.2.so (for architecture i386): Mach-O bundle i386 mod_recital2.2.so (for architecture x86_64): Mach-O 64-bit bundle x86_64The apache module files are stored in the /usr/libexec/apache2/ directory on a default apache install on the Mac and the configuration file is /private/etc/apache2/httpd.conf
Key features of the Recital database include:
- SQL-92 and a broad subset of ANSI SQL 99, as well as extensions
- Cross-platform support
- Stored procedures
- Triggers
- Cursors
- Updatable Views
- System Tables
- Query caching
- High-performance
- Single-User and Multi-User
- Multi-Process
- ACID Transactions
- Referential Integrity
- Cascading Updates and Deletes
- Multi-table Joins
- Row-level Locking
- BLOBs (Binary Large Objects)
- UDFs (User Defined Functions)
- OLTP (On-Line Transaction Processing)
- Drivers for ODBC, JDBC, and .NET
- Sub-SELECTs (i.e. nested SELECTs)
- Embedded database library
- Database timelines providing data undo functionality
- Fault tolerant clustering support
- Hot backup
VMware products, such as ESX, Workstation, Server, and Fusion, come with a built-in VNC server to access guests.
This allows you to connect to the guest without having a VNC server installed in the guest - useful if a server doesn't exist for the guest or if you need access some time when a server would not work (say during the boot process). It's also good in conjunction with Headless Mode.
The VNC server is set up on a per-VM basis, and is disabled by default. To enable it, add the following lines to the .vmx:
RemoteDisplay.vnc.enabled = "TRUE" RemoteDisplay.vnc.port = "5901"
You can set a password with RemoteDisplay.vnc.key; details for how to calculate the obfuscated value given a plaintext password are in Compute hashed password for use with RemoteDisplay.vnc.key.
If you want more than one VM set up in this manner, make sure they have unique port numbers. To connect, use a VNC client pointing at host-ip-address:port. If you connect from a different computer, you may have to open a hole in the OS X firewall. If you use Leopard's Screen Sharing.app on the same computer as Fusion, don't use port 5900 since Screen Sharing refuses to connect to that.
Recital implements SQL-92 and most of the SQL-99 standard for SQL, but also provides lower level navigational data access for performing high transaction throughput. It is the choice of the application developer whether to use SQL, navigational data access, or a combination of both depending upon the type of application being developed.
This is a good primer for getting familiar with using Infiniband with Redhat/Centos Linux.
http://people.redhat.com/dledford/infiniband_get_started.html
Getting Started with InfiniBand
The first step to using a new infiniband based network is to get the right packages installed. These are the infiniband related packages we ship and what they are there for (Note, the Fedora packages have not all been built or pushed to the repos yet, so their mention here is as a "Coming soon" variety, not an already done variety):
System Requirements:
- Minimum memory: 4MB
- Minimum Diskspace: ~20MB
It would appear that gigabit LAN is not! In fact it often runs at the same speed as 100Mbps LAN. Let's look at why exactly.
After configuring your network you can use the ifconfig command to see what speeds the LAN is connected. Even though 1000Mbps is reported by the card, the reality is that the overall throughtput may well be ~100Mpbs. You can try copying a large file using scp to demonstrate this.
As it turns out, in order to use a gigabit LAN you need to use CAT6 cables. CAT5 and CAT5E are not good enough. End result, the ethernet cards throttle back the speed to reduce dropped packets and errors.
You can find a good article here titled Squeeze Your Gigabit NIC for Top Performance. After tuning up the TCP parameters i found that it made no dfifference. The principal reasons behind low gigabit ethernet performance can be summed up as follows.
- Need to use CAT6 cables
- Slow Disk speed
- Limitations of the PCI bus which the gigabit ethernet cards use
You can get an idea about the disk speed using the hdparm command:
Display the disk partitions and choose the main linux partition which has the / filesystem.
# fdisk -l
Then get disk cache and disk read statistics:
# hdparm -Tt /dev/sda0
On my desktop system the sata disk perfomance is a limiting factor. These were the results:
/dev/sda1:
Timing cached reads: 9984 MB in 2.00 seconds = 4996.41 MB/sec
Timing buffered disk reads: 84 MB in 3.13 seconds = 58.49 MB/sec
Well, that equates to a raw disk read speed of 58.49 * 8 = 467Mbps which is half the speed of a gigabit LAN.
So.. NAS storage with lots of memory looks to be the way to go... If you use the right cables!
There is a good article on the gluster website here which gives some good information regarding file system optimization suitable for a HA Recital cluster solution.
This article looks at the range of client access mechanisms for Windows that can be used with the Recital C-ISAM Bridge and details bridge configuration and usage.
Overview
Just because the format of data is regarded as 'legacy' does not make that data in any way obsolete. Modern client interfaces can not only extend the life of long-term data, but also provide different ways to analyse and gain advantage from that data.
Recital Corporation provides a range of solutions to interface to Informix compliant C-ISAM data on Linux or UNIX from Windows clients.
.NET
Click image to display full size
Fig 1: Recital Mirage .NET application accessing the C-ISAM Demo table.
Recital offers two alternative ways to access C-ISAM data using Microsoft .NET:
The Recital .NET Data Provider is a managed Data Provider written in C# that provides full compatibility with the Microsoft SQLserver and OLE DB data providers that ship with the .NET framework. It is fully integrated with the Visual Studio .NET IDE supporting data binding and automatic code generation using the form designer. The Recital .NET Data Provider works in conjunction with the Recital Database Server for Linux or UNIX to access C-ISAM data.
Recital Mirage .NET is a complete solution for migrating, developing and deploying 4GL database applications. Recital Mirage .NET works in conjunction with the Recital Mirage .NET Application Server for Linux or UNIX to access C-ISAM data.
JDBC
Click image to display full size
Fig 2: Java™ Swing JTable accessing the C-ISAM Demo table via the Recital JDBC Driver.
The Recital JDBC Driver is an all Java Type 4 JDBC 3.0 Driver, allowing you to access C-ISAM data from Java applets and applications. The Recital JDBC Driver works in conjunction with the Recital Database Server for Linux or UNIX to access C-ISAM data.
ODBC
Click image to display full size
Fig 3: Microsoft® Office Excel 2003 Pivot Chart and Pivot Table accessing the C-ISAM Demo table via the Recital ODBC Driver.
The Recital ODBC Driver is an ODBC 3.0 Driver, allowing you to access C-ISAM data from your preferred ODBC based Windows applications. You can develop your own applications in languages such as C++ or Visual Basic, manipulate the data in a spreadsheet package or word processor document and design charts, graphs and reports. The Recital ODBC Driver works in conjunction with the Recital Database Server for Linux or UNIX to access C-ISAM data.
Configuring the Recital C-ISAM Bridge
Data access is achieved through a C-ISAM Bridge. This requires the creation of an empty Recital table that has the same structure as the C-ISAM file and of a RecitalC-ISAM Bridge file.
On Linux and UNIX, Recital Terminal Developer and the Recital Database Server come complete with an example C-ISAM data file, C-ISAM index and Recital C-ISAM bridge that can be used for testing and as a template for configuring your own C-ISAM bridges. The Recital Database Server also includes a bridge creation ini file.
Step 1:
Create a Recital table with the same structure as the C-ISAM file. The fields/columns in the structure file must exactly match the data type and length of those in the C-ISAM file. The Recital table will have one byte more in total record length due to the Recital record deletion marker.
To create the table, use the SQL CREATE TABLE command or the Recital Terminal Developer CREATE worksurface. The SQL CREATE TABLE command can be called directly:
SQLExecDirect: In: hstmt = 0x00761BE8, szSqlStr = "CREATE TABLE cisamdemo.str (DD Char(4) DESCRIPTION "Dd...", cbSqlStr = -3 Return: SQL_SUCCESS=0
or be included in a 4GL program:
// createtab.prg CREATE TABLE cisamdemo.str; (DD Char(4) DESCRIPTION "Dd",; CONFIRM Char(6) DESCRIPTION "Confirm",; PROCDATE Char(6) DESCRIPTION "Procdate",; CONTROL Char(5) DESCRIPTION "Control",; DOLLARS Decimal(13,2) DESCRIPTION "Dollars",; DEALER Char(5) DESCRIPTION "Dealer",; TERRITORY Char(2) DESCRIPTION "Territory",; WOREP Char(12) DESCRIPTION "Worep",; CURRTRAN Char(3) DESCRIPTION "Currtran",; TRADDATE Char(6) DESCRIPTION "Traddate",; CITY Char(10) DESCRIPTION "City",; ACCOUNT Char(11) DESCRIPTION "Account",; PRETRAN Char(2) DESCRIPTION "Pretran",; AFSREP Char(14) DESCRIPTION "Afsrep",; REPKEY Char(9) DESCRIPTION "Repkey",; BRANCH Char(3) DESCRIPTION "Branch",; WODEALER Char(5) DESCRIPTION "Wodealer",; BANKCODE Char(2) DESCRIPTION "Bankcode",; COMMRATE Decimal(6,4) DESCRIPTION "Commrate",; NEWREP Char(1) DESCRIPTION "Newrep",; SETTLE Char(1) DESCRIPTION "Settle",; POSTDATE Char(6) DESCRIPTION "Postdate") if file("cisamdemo.str") return .T. else return .F. endif // end of createtab.prg
Server-side 4GL programs can be called by all clients, e.g. from a Java class with a JDBC connection:
//--------------------------------- //-- create_str.java -- //--------------------------------- import java.sql.*; import java.io.*; import Recital.sql.*; public class create_str { public static void main(String argv[]) { try { new RecitalDriver(); String url = "jdbc:Recital:" + "SERVERNAME=cisamserver;" + "DIRECTORY=/usr/recital/data/southwind;" + "USERNAME=user;" + "PASSWORD=password"; Connection con = DriverManager.getConnection(url, "user", "pass"); Statement stmt = con.createStatement(); CallableStatement sp = con.prepareCall("{call createtab}"); boolean res = sp.execute(); String outParam = sp.getString(1); System.out.println("Returned "+outParam); sp.close(); con.close(); } catch (Exception e) { System.out.flush(); System.err.flush(); DriverManager.println("Driver exception: " + e.getMessage()); e.printStackTrace(); } try { System.out.println("Press any key to continue..."); System.in.read(); } catch(IOException ie) { ; } } }
The table should be given a ‘.str’ file extension (rather than the default ‘.dbf’) to signify that this is a structure file only.
Please see the end of this article for information on matching Informix and Recital data types
Fig 4: Recital CREATE/MODIFY STRUCTURE worksurface for character mode table creation.
Step 2: Creating the Bridge File
If you have Recital installed on the server platform, the Bridge File can be created using the CREATE BRIDGE worksurface. The corresponding command to modify the bridge file is MODIFY BRIDGE <bridge file>. This is the cisamdemo.dbf bridge file in the CREATE/MODIFY BRIDGE WORKSURFACE:
> modify bridge cisamdemo.dbf
Fig 5: Recital CREATE BRIDGE/MODIFY BRIDGE worksurface for bridge creation.
For Recital Database Server clients, the Bridge File can be created using the Recital/SQL CREATE BRIDGE command:
Recital/SQL CREATE BRIDGE:
CREATE BRIDGE cisamdemo.dbf; TYPE "CISAM"; EXTERNAL "cisamdemo.dat"; METADATA "cisamdemo.str"; ALIAS "cisamdemo"
or:
CREATE BRIDGE cisamdemo.dbf; AS "type=CISAM;external=cisamdemo.dat;metadata=cisamdemo.str;alias=cisamdemo"
The examples above assume that the C-ISAM file, the bridge file and the Recital structure file are all in the current working directory. Full path information can be specified for the <externalname> and the <databasename>. For added flexibility, environment variables can be used to determine the path at the time the bridge is opened. Environment variables can be included for either or both the <externalname> and the <databasename>. A colon should be specified between the environment variable and the file name.
e.g.
CREATE BRIDGE cisamdemo.dbf; TYPE "CISAM"; EXTERNAL "DB_DATADIR:cisamdemo"; METADATA "DB_MIRAGE_PATH:cisamdemo.str"; ALIAS "cisamdemo"
Recital CREATE BRIDGE/MODIFY BRIDGE worksurface:
Fig 6: Recital CREATE BRIDGE/MODIFY BRIDGE worksurface - using environment variables.
Using the Bridge
The Bridge can now be used. To access the C-ISAM file, use the ‘alias’ specified in the Bridge definition.
SQL:
SELECT * FROM cisamdemo
Recital/4GL:
use cisamdemo
Indexes
The cisamdemo.dat file included in the Recital distributions for Linux and UNIX has three associated index keys in the cisamdemo.idx file:
Select area: 1 Database in use: cisamdemo Alias: cisamdemo Number of records: 4 Current record: 2 File Type: CISAM (C-ISAM) Master Index: [cisamdemo.idx key #1] Key: DD+CONFIRM+PROCDATE+CONTROL Type: Character Len: 21 (Unique) Index: [cisamdemo.idx key #2]
Key: DD+SUBSTR(CONFIRM,2,5)+TRADDATE+STR(DOLLARS,13,2) +CURRTRAN+ACCOUNT Type: Character Len: 42 Index: [cisamdemo.idx key #3] Key: DEALER+BRANCH+AFSREP+SUBSTR(PROCDATE,5,2) +SUBSTR(CONTROL,2,4) Type: Character Len: 28
The Recital C-ISAM bridge makes full use of the C-ISAM indexes. SQL SELECT statements with WHERE clauses are optimized based on any of the existing indexes when possible. The following ODBC SELECT call makes use of key #3 rather than sequentially searching through the data file.
SQLExecDirect: In: hstmt = 0x00761BE8, szSqlStr = "select * from cisamdemo where dealer+branch+afsrep=' 00161 595-7912", cbSqlStr = -3 Return: SQL_SUCCESS=0 Get Data All: "DD", "CONFIRM", "PROCDATE", "CONTROL", "DOLLARS", "DEALER", "TERRITORY", "WOREP", "CURRTRAN", "TRADDATE", "CITY", "ACCOUNT", "PRETRAN", "AFSREP", "REPKEY", "BRANCH", "WODEALER", "BANKCODE", "COMMRATE", "NEWREP", "SETTLE", "POSTDATE" "0159", " 15522", "930312", "13356", 4992.60, "00161", "19", "", "210", "930305", "", "70000100009", "", "595-7912", "930315791", "", "", "59", 0.0000, "1", "", "930315" 1 row fetched from 22 columns.
Using the Recital/4GL, the primary index is set as the master index when the bridge is first opened. Any secondary indexes can be selected using the SET ORDER TO <expN> command. The Recital/4GL SEEK or FIND commands and SEEK() function can be used to search in the current master index.
> SET ORDER TO 3 Master index: [cisamdemo.idx key #3] > SEEK “00161 595-7912”
Appendix 1: Data Types
Informix |
Recital |
Byte |
Numeric |
Char |
Character |
Character |
Character |
Date |
Date |
Datetime |
Character |
Decimal |
Numeric |
Double Precision |
Float |
Float |
Real |
16 Bit Integer |
Short |
Integer |
Numeric |
Interval |
Character |
32 Bit Long |
Integer |
Money |
Numeric |
Numeric |
Numeric |
Real |
Numeric |
Smallfloat |
Numeric |
Smallint |
Numeric |
Text |
Unsupported |
Varchar |
Character |
Appendix 2: C-ISAM RDD Error Messages
The following errors relate to the use of the Recital CISAM Replaceable Database Driver (RDD). They can be received as an ‘errno <expN>’ on Recital error messages:
ERRNO() |
Error Description |
100 |
Duplicate record |
101 |
File not open |
102 |
Invalid argument |
103 |
Invalid key description |
104 |
Out of file descriptors |
105 |
Invalid ISAM file format |
106 |
Exclusive lock required |
107 |
Record claimed by another user |
108 |
Key already exists |
109 |
Primary key may not be used |
110 |
Beginning or end of file reached |
111 |
No match was found |
112 |
There is no “current” established |
113 |
Entire file locked by another user |
114 |
File name too long |
115 |
Cannot create lock file |
116 |
Memory allocation request failed |
117 |
Bad custom collating |
118 |
Duplicate primary key allowed |
119 |
Invalid transaction identifier |
120 |
Exclusively locked in a transaction |
121 |
Internal error in journaling |
122 |
Object not locked |