Difference between revisions of "Recital Replication Getting Started"

From Recital Documentation Wiki
Jump to: navigation, search
 
(13 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
=Recital Replication Summary=  
 
=Recital Replication Summary=  
There are two types of replication in Recital;
+
Replication in Recital is a Master to Slave(s) configuration, where the updates are performed on one master server and published for slaves to subscribe.  
* Master to Slave(s); where the updates are performed on one master server and published for slaves to subscribe.
+
* Peer to Peer; where the updates are performed on many servers and the updates published for each server to subscribe.
+
  
 
== Master to Slave Replication (MSR) ==
 
== Master to Slave Replication (MSR) ==
Line 12: Line 10:
 
* Analytic - live data can be created on the master, while the analysis of the information can take place on the slave without affecting the performance of the master.  
 
* Analytic - live data can be created on the master, while the analysis of the information can take place on the slave without affecting the performance of the master.  
 
* Long-distance data distribution - if a branch office would like to work with a copy of your main data, you can use replication to create a local copy of the data for their use without requiring permanent access to the master.  
 
* Long-distance data distribution - if a branch office would like to work with a copy of your main data, you can use replication to create a local copy of the data for their use without requiring permanent access to the master.  
 
== Peer to Peer Replication (PPR) ==
 
This type of replication enables data from one or more Recital database servers to be published for replication for one or more additional Recital database servers to subscribe. Replication is asynchronous - your replication subscription service does not need to be connected permanently to receive updates from the publication service. Which means that updates can occur over long-distance connections and even temporary solutions such as a dial-up service. Depending on the configuration, you can replicate all databases, selected databases and even selected tables within a database.
 
 
=== Target uses for PPR in Recital include:  ===
 
* Scale-out solutions - spreading the load among multiple servers to improve performance.
 
  
 
= Enabling Replication  =
 
= Enabling Replication  =
 
In order for transactions to be published in the queue table, replication must be set on first. This can be done by adding the ''SET REPLICATION ON'' command into a local or systemwide configuration file. The config.db stored in the ''conf'' directory which is in the root recital installation directory is used for systemwide or you can update the config.db in the local directory. You may also add the command to any 4GL program file.  
 
In order for transactions to be published in the queue table, replication must be set on first. This can be done by adding the ''SET REPLICATION ON'' command into a local or systemwide configuration file. The config.db stored in the ''conf'' directory which is in the root recital installation directory is used for systemwide or you can update the config.db in the local directory. You may also add the command to any 4GL program file.  
 
  
 
When replication is turned on all tables already open and all tables opened afterwards are flagged for replication. You may turn replication off to disable replication, but if you want to filter out tables from replication it's better to use the allow and deny access control lists. Index files associated with the tables are also updated. Multiple index tag files are handled automatically. Single index files are sorted by name and then added onto the table name when stored with the transaction.
 
When replication is turned on all tables already open and all tables opened afterwards are flagged for replication. You may turn replication off to disable replication, but if you want to filter out tables from replication it's better to use the allow and deny access control lists. Index files associated with the tables are also updated. Multiple index tag files are handled automatically. Single index files are sorted by name and then added onto the table name when stored with the transaction.
 
  
 
To turn replication on you must have installed the Replication service on the system already. If this hasn''t been done then a ''You must configure the replication service first with 'dbservice RRS''' error will be returned.  
 
To turn replication on you must have installed the Replication service on the system already. If this hasn''t been done then a ''You must configure the replication service first with 'dbservice RRS''' error will be returned.  
 
  
 
The first time a table is flagged for replication all rows must be updated with a sequence number, if the table contains a large number of row this process may take some time. Once it's been the table is flagged so this process will not be done again. All new rows inserted after this point will automatically contain their own unique sequence number. For the life of a table these numbers will never be duplicated.  
 
The first time a table is flagged for replication all rows must be updated with a sequence number, if the table contains a large number of row this process may take some time. Once it's been the table is flagged so this process will not be done again. All new rows inserted after this point will automatically contain their own unique sequence number. For the life of a table these numbers will never be duplicated.  
 
  
 
== Access Control Lists ==
 
== Access Control Lists ==
Line 62: Line 50:
 
=== replication.allow ===
 
=== replication.allow ===
 
This file describes the names of the tables which will be replicated if REPLICATION is SET ON. If this list is not empty then only tables matching will be replicated.
 
This file describes the names of the tables which will be replicated if REPLICATION is SET ON. If this list is not empty then only tables matching will be replicated.
 
  
 
=== replication.deny ===
 
=== replication.deny ===
Line 68: Line 55:
  
 
= Starting Replication Services =
 
= Starting Replication Services =
There are two replication services that can be used in the Recital, a subscriber and a publisher.
 
 
  
 
== Subscriber ==
 
== Subscriber ==
The subscriber service is used by any system that needs to update it's databases from published data. In a Master Slave configuration each slave would start a subscriber service and to connect to the published data on the Master server. In a Peer to Peer configuration each peer would start a subscriber service and connect to the published data on the publication server.
+
The subscriber service is used by any system that needs to update its databases from published data. In a Master Slave configuration each slave would start a subscriber service and to connect to the published data on the Master server.  
 
+
 
+
== Publisher ==
+
The publication service is only used in a Peer to Peer configuration and only the master publisher would run this service. This service will retrieve the data from the replication queue tables on each peer. This service checks for conflicts before publishing the data for subscriber services.
+
 
+
  
 
== Recital Database Server ==
 
== Recital Database Server ==
In order for either of these services to work the Recital Database Server must be installed and running on the system publishing the data. In a master slave configuration this would be the master server. In a Peer to Peer configuration a Recital Database Server must be running each peer.
+
The Recital Database Server must be installed and running on the system publishing the data.  
 
+
  
 
== ODBC Data Source ==
 
== ODBC Data Source ==
 
In order for the replication services to connect to the Recital Database Server you must define a ODBC data source in the odbc.ini file. The Data Source name format is ''Recital Replication Service on'' plus the node name. The Driver name is Recital and the Database format is ''ODBC:RECITAL:'' The following options can also be added;
 
In order for the replication services to connect to the Recital Database Server you must define a ODBC data source in the odbc.ini file. The Data Source name format is ''Recital Replication Service on'' plus the node name. The Driver name is Recital and the Database format is ''ODBC:RECITAL:'' The following options can also be added;
 
== recitalreplication ==
 
The replication services are administrated with the recitalreplication command. See [[recitalreplication]] for full details.
 
 
  
 
= Starting Master Slave Configuration  =
 
= Starting Master Slave Configuration  =
 
A Master Slave configuration in Recital works with the published data stored in the queue table of the replication database. The Recital engine on the master system (the source of the database changes) writes all updates, inserts, deletes, recalls, packs and zaps into the queue table of the replication database. These transactions are stored in a XML format in the queue table.  
 
A Master Slave configuration in Recital works with the published data stored in the queue table of the replication database. The Recital engine on the master system (the source of the database changes) writes all updates, inserts, deletes, recalls, packs and zaps into the queue table of the replication database. These transactions are stored in a XML format in the queue table.  
 
  
 
Slaves are configured to subscribe to the master and to execute the published events on the slave's local databases. The Master is dumb in this scenario. Once replication has been enabled, all statements are published in the replication queue. If required, you can configure the Master to only publish events that apply to particular databases or tables
 
Slaves are configured to subscribe to the master and to execute the published events on the slave's local databases. The Master is dumb in this scenario. Once replication has been enabled, all statements are published in the replication queue. If required, you can configure the Master to only publish events that apply to particular databases or tables
 
  
 
Each subscribed slave will get a copy of the changes published since the last time it connected. Slaves keep a record of the position within the published queue that they have read and processed. This means that multiple slaves can be connected to the master and executing different parts of the same published data. Because the slaves control this process, individual slaves can be connected and disconnected from the server without affecting the master's operation. Also, because each slave remembers its position within the published queue, it is possible for slaves to be disconnected, reconnect and then 'catch up' by continuing from the recorded position.  
 
Each subscribed slave will get a copy of the changes published since the last time it connected. Slaves keep a record of the position within the published queue that they have read and processed. This means that multiple slaves can be connected to the master and executing different parts of the same published data. Because the slaves control this process, individual slaves can be connected and disconnected from the server without affecting the master's operation. Also, because each slave remembers its position within the published queue, it is possible for slaves to be disconnected, reconnect and then 'catch up' by continuing from the recorded position.  
 
  
 
Both the master and each slave must be configured with a unique id. In addition, the slave must be configured with information about the master host name, RTQ database name and position within that file.  
 
Both the master and each slave must be configured with a unique id. In addition, the slave must be configured with information about the master host name, RTQ database name and position within that file.  
 
 
== Starting MSR ==
 
You should start the Master Slave Replication processes in the following order;
 
 
 
=== Starting Master  ===
 
On the system specified as the Master you must start the Recital Database Server first with the command;
 
 
<pre>
 
# dbserver start
 
</pre>
 
 
=== Staring Slaves ===
 
On the systems specified as Slaves you must start the Subscriber Service with the command;
 
 
<pre>
 
# dbservice subscriber start
 
</pre>
 
 
== Checking status ==
 
Once you have started the subscriber services on the Slaves you can check there current status with the command;
 
 
<pre>
 
# dbservice subscriber status
 
</pre>
 
 
An example out put for the Slave Subscriber Service would be;
 
 
<code><pre>
 
The subscription server is running, last SYNCNUM processed was 0100000000000002 on node jazz
 
</pre></code>
 
 
== Starting PPR ==
 
On each Peer system you must start the Recital Database Server first with the command;
 
 
<pre>
 
# dbserver start
 
</pre>
 
 
The you should start the Peer to Peer Replication processes in the following order;
 
 
 
=== Starting Master Publication service ===
 
On the system specified as the Master you must start the Publication Service first with the command;
 
 
<pre>
 
# dbservice publication start
 
</pre>
 
 
When the publication service starts it will attempt to connect to all the subscribed peer servers. If they are not on line, then it will set the error status on in the peer table for the server it can't connect to. It will try to establish a connection to unconnected servers each time it wakes to process transactions. So when the server comes back online it will connect and process all waiting transaction since it last connected.
 
  
 
=== Staring Subscriber Service ===
 
=== Staring Subscriber Service ===
On each Peer system you must start the Subscription Service with the command;
 
 
<pre>
 
# dbservice subscription start
 
</pre>
 
 
 
The subscriber service will sleep the specified number of seconds between each set of transaction. Then it will connect to the master publisher and retrieve all published data since it last connected. For each transaction that it processes it will attempt to open the table in the required mode. For ZAP and PACK transactions it must open the table exclusively, for all others it will open it shared. If it can't find the table specified via the path or can't open it in the required mode an error will be returned. The Subscriber Service will perform all required locks on tables so existing Recital users can coexist.
 
The subscriber service will sleep the specified number of seconds between each set of transaction. Then it will connect to the master publisher and retrieve all published data since it last connected. For each transaction that it processes it will attempt to open the table in the required mode. For ZAP and PACK transactions it must open the table exclusively, for all others it will open it shared. If it can't find the table specified via the path or can't open it in the required mode an error will be returned. The Subscriber Service will perform all required locks on tables so existing Recital users can coexist.
  
== Checking status ==
+
=== Starting replication services ===
Once you have started the subscriber services on the peers you can check there current status with the command;
+
The replication services are administrated with the recitalreplication command. See [[recitalreplication]] for full details.
 
+
<code><pre>
+
dbservice subscriber status
+
</pre></code>
+
 
+
An example out put for the Peer Subscriber Service would be;
+
 
+
<code><pre>
+
Recital Replication Service on pubserv
+
Subscriber service jazz (02) is running on a 10 second delay.
+
Version 9.5.1 Compiled on Wed Feb 11 10:21:50 EST 2009
+
Connected to publication pubserv in a Peer to Peer environment.
+
Last SYNCNUM processed was 0100000000000002 at %s
+
</pre></code>
+
 
+
You can check the status of the publication server y;
+
 
+
<code><pre>
+
dbservice publisher status
+
</pre></code>
+
 
+
An example out put for the Master Publisher Service would be;
+
 
+
<code><pre>
+
Publication service pubserv is running on a 10 second delay.
+
Version 9.5.1 Compiled on Wed Feb 11 10:21:50 EST 2009
+
 
+
Peer Name: jazz Publication ID: 02 Status: Connected SYNCNUM last update
+
Publication 0100000000003856 2009-02-11 15:59:42
+
Subscription 0100000000003856 2009-01-30 16:21:03
+
Peer 0200000000001B9D 2009-02-11 15:59:42
+
 
+
Peer Name: port052 Publication ID: Unknown Status: Connected
+
SYNCNUM last update
+
Publication 0100000000003856 2009-02-11 15:59:42
+
Subscription
+
Peer 0100000000001CB9 2009-02-11 15:59:42
+
</pre></code>
+

Latest revision as of 15:00, 8 June 2010

Recital Replication Summary

Replication in Recital is a Master to Slave(s) configuration, where the updates are performed on one master server and published for slaves to subscribe.

Master to Slave Replication (MSR)

This type of replication enables data from one Recital database server (called the master) to be published for replication for one or more Recital database servers (slaves) to subscribe. Replication is asynchronous - your replication slaves do not need to be connected permanently to receive updates published on the master. They just need to connect to the publisher when updating the subscription. Which means that updates can occur over long-distance connections and even temporary solutions such as a dial-up service. Depending on the configuration, you can replicate all databases, selected databases and even selected tables within a database.

Target uses for MSR in Recital include:

  • Scale-out solutions - spreading the load among multiple slaves to improve performance. In this environment, all writes and updates must take place on the master server. Reads, however, may take place on one or more slaves. This model can improve the performance of writes (since the master is dedicated to updates), while dramatically increasing read speed across an increasing number of slaves.
  • Data security - because data is replicated to the slave, and the slave can pause the replication process, it is possible to run backup services on the slave without corrupting the corresponding master data.
  • Analytic - live data can be created on the master, while the analysis of the information can take place on the slave without affecting the performance of the master.
  • Long-distance data distribution - if a branch office would like to work with a copy of your main data, you can use replication to create a local copy of the data for their use without requiring permanent access to the master.

Enabling Replication

In order for transactions to be published in the queue table, replication must be set on first. This can be done by adding the SET REPLICATION ON command into a local or systemwide configuration file. The config.db stored in the conf directory which is in the root recital installation directory is used for systemwide or you can update the config.db in the local directory. You may also add the command to any 4GL program file.

When replication is turned on all tables already open and all tables opened afterwards are flagged for replication. You may turn replication off to disable replication, but if you want to filter out tables from replication it's better to use the allow and deny access control lists. Index files associated with the tables are also updated. Multiple index tag files are handled automatically. Single index files are sorted by name and then added onto the table name when stored with the transaction.

To turn replication on you must have installed the Replication service on the system already. If this hasnt been done then a You must configure the replication service first with 'dbservice RRS error will be returned.

The first time a table is flagged for replication all rows must be updated with a sequence number, if the table contains a large number of row this process may take some time. Once it's been the table is flagged so this process will not be done again. All new rows inserted after this point will automatically contain their own unique sequence number. For the life of a table these numbers will never be duplicated.

Access Control Lists

If you wish to limit which tables are replicated you may do this with access control lists. Access control lists are defined in two files stored in the conf directory which is in the root recital installation directory. Comment lines can be added to these file by starting the line with # symbol. The access control specification is "directory path" or [database name!]tablename, where an optional database name may be specified. Wild cards may also be specified with ''*'' for all text matches and ''%'' for single character matches. Name expansion from environment variables can also be used.

Example controls
Example Description
southwind!* This would include all tables in the southwind database.
recital* This would include all tables in all databases or directories that start with recital.
/tmp/* This would include all tables in the directory "/tmp/"
${DB_TMPDIR}/* This would include all tables in the directory expanded from the environment variable DB_TMPDIR


replication.allow

This file describes the names of the tables which will be replicated if REPLICATION is SET ON. If this list is not empty then only tables matching will be replicated.

replication.deny

This file describes the names of the tables which will NOT be replicated if REPLICATION is SET ON. If this list is not empty then any tables matching will be not be replicated.

Starting Replication Services

Subscriber

The subscriber service is used by any system that needs to update its databases from published data. In a Master Slave configuration each slave would start a subscriber service and to connect to the published data on the Master server.

Recital Database Server

The Recital Database Server must be installed and running on the system publishing the data.

ODBC Data Source

In order for the replication services to connect to the Recital Database Server you must define a ODBC data source in the odbc.ini file. The Data Source name format is Recital Replication Service on plus the node name. The Driver name is Recital and the Database format is ODBC:RECITAL: The following options can also be added;

Starting Master Slave Configuration

A Master Slave configuration in Recital works with the published data stored in the queue table of the replication database. The Recital engine on the master system (the source of the database changes) writes all updates, inserts, deletes, recalls, packs and zaps into the queue table of the replication database. These transactions are stored in a XML format in the queue table.

Slaves are configured to subscribe to the master and to execute the published events on the slave's local databases. The Master is dumb in this scenario. Once replication has been enabled, all statements are published in the replication queue. If required, you can configure the Master to only publish events that apply to particular databases or tables

Each subscribed slave will get a copy of the changes published since the last time it connected. Slaves keep a record of the position within the published queue that they have read and processed. This means that multiple slaves can be connected to the master and executing different parts of the same published data. Because the slaves control this process, individual slaves can be connected and disconnected from the server without affecting the master's operation. Also, because each slave remembers its position within the published queue, it is possible for slaves to be disconnected, reconnect and then 'catch up' by continuing from the recorded position.

Both the master and each slave must be configured with a unique id. In addition, the slave must be configured with information about the master host name, RTQ database name and position within that file.

Staring Subscriber Service

The subscriber service will sleep the specified number of seconds between each set of transaction. Then it will connect to the master publisher and retrieve all published data since it last connected. For each transaction that it processes it will attempt to open the table in the required mode. For ZAP and PACK transactions it must open the table exclusively, for all others it will open it shared. If it can't find the table specified via the path or can't open it in the required mode an error will be returned. The Subscriber Service will perform all required locks on tables so existing Recital users can coexist.

Starting replication services

The replication services are administrated with the recitalreplication command. See recitalreplication for full details.