Local (in-cluster) write forwarding allows your applications to issue read/write transactions directly on an Aurora Replica. These transactions are then forwarded to the writer DB instance to be committed. You can use local write forwarding when your applications require read-after-write consistency, which is the ability to read the latest write in a transaction.

Read replicas receive updates asynchronously from the writer. Without write forwarding, you have to transact any reads that require read-after-write consistency on the writer DB instance. Or you have to develop complex custom application logic to take advantage of multiple read replicas for scalability. Your applications must fully split all read and write traffic, maintaining two sets of database connections to send the traffic to the correct endpoint. This development overhead complicates application design when the queries are part of a single logical session, or transaction, within the application. Moreover, because replication lag can differ among read replicas, it’s difficult to achieve global read consistency across all instances in the database.

Write forwarding avoids the need to split those transactions or send them exclusively to the writer, which simplifies application development. This new capability makes it easy to achieve read scale for workloads that need to read the latest write in a transaction and aren’t sensitive to write latency.

Local write forwarding is different from global write forwarding, which forwards writes from a secondary DB cluster to the primary DB cluster in an Aurora global database. You can use local write forwarding in a DB cluster that is part of an Aurora global database. For more information, see Using write forwarding in an Amazon Aurora global database.

Local write forwarding requires Aurora MySQL version 3.04 or higher.

Enabling local write forwarding

By default, local write forwarding isn’t enabled for Aurora MySQL DB clusters. You enable local write forwarding at the cluster level, not at the instance level.

Using the AWS Management Console, select the Turn on local write forwarding check box under Read replica write forwarding when you create or modify a DB cluster.

Important

You can also enable local write forwarding for cross-Region read replicas that use binary logging, but write operations aren’t forwarded to the source AWS Region. They’re forwarded to the writer DB instance of the binlog read replica cluster.

Use this method only if you have a use case for writing to the binlog read replica in the secondary AWS Region. Otherwise, you might end up with a “split-brain” scenario where replicated datasets are inconsistent with each other.

We recommend that you use global write forwarding with global databases, rather than local write forwarding on cross-Region read replicas, unless absolutely necessary. For more information, see Using write forwarding in an Amazon Aurora global database.

To determine that you can use write forwarding in a DB cluster, confirm that the cluster has the attribute LocalWriteForwardingStatus set to enabled.

In the AWS Management Console, on the Configuration tab of the details page for the cluster, you see the status Enabled for Local read replica write forwarding.

To see the status of the write forwarding setting for all of your clusters, run the following AWS CLI command.

 

aws rds describe-db-clusters \
--query '*[].{DBClusterIdentifier:DBClusterIdentifier,LocalWriteForwardingStatus:LocalWriteForwardingStatus}'

[
    {
        "LocalWriteForwardingStatus": "enabled",
        "DBClusterIdentifier": "write-forwarding-test-cluster-1"
    },
    {
        "LocalWriteForwardingStatus": "disabled",
        "DBClusterIdentifier": "write-forwarding-test-cluster-2"
    },
    {
        "LocalWriteForwardingStatus": "requested",
        "DBClusterIdentifier": "test-global-cluster-2"
    },
    {
        "LocalWriteForwardingStatus": "null",
        "DBClusterIdentifier": "aurora-mysql-v2-cluster"
    }
]

A DB cluster can have the following values for LocalWriteForwardingStatus:

  • disabled – Write forwarding is disabled.
  • disabling – Write forwarding is in the process of being disabled.
  • enabled – Write forwarding is enabled.
  • enabling – Write forwarding is in the process of being enabled.
  • null – Write forwarding isn’t available for this DB cluster.
  • requested – Write forwarding has been requested, but is not yet active.

Application and SQL compatibility with write forwarding

You can use the following kinds of SQL statements with write forwarding:

  • Data manipulation language (DML) statements, such as INSERTDELETE, and UPDATE. There are some restrictions on the properties of these statements that you can use with write forwarding, as described following.
  • SELECT ... LOCK IN SHARE MODE and SELECT FOR UPDATE statements.
  • PREPARE and EXECUTE statements.

Certain statements aren’t allowed or can produce stale results when you use them in a DB cluster with write forwarding. Thus, the EnableLocalWriteForwarding setting is disabled by default for DB clusters. Before enabling it, check to make sure that your application code isn’t affected by any of these restrictions.

The following restrictions apply to the SQL statements you use with write forwarding. In some cases, you can use the statements on DB clusters with write forwarding enabled. This approach works if write forwarding isn’t enabled within the session by the aurora_replica_read_consistency configuration parameter. If you try to use a statement when it’s not allowed because of write forwarding, then you will see an error message similar to the following:

 

ERROR 1235 (42000): This version of MySQL doesn't yet support 'operation with write forwarding'. 
Data definition language (DDL)
Connect to the writer DB instance to run DDL statements. You can’t run them from reader DB instances.
Updating a permanent table using data from a temporary table
You can use temporary tables on DB clusters with write forwarding enabled. However, you can’t use a DML statement to modify a permanent table if the statement refers to a temporary table. For example, you can’t use an INSERT ... SELECT statement that takes the data from a temporary table.
XA transactions
You can’t use the following statements on a DB cluster when write forwarding is enabled within the session. You can use these statements on DB clusters that don’t have write forwarding enabled, or within sessions where the aurora_replica_read_consistency setting is empty. Before enabling write forwarding within a session, check if your code uses these statements.

 

XA {START|BEGIN} xid [JOIN|RESUME]
XA END xid [SUSPEND [FOR MIGRATE]]
XA PREPARE xid
XA COMMIT xid [ONE PHASE]
XA ROLLBACK xid
XA RECOVER [CONVERT XID]
LOAD statements for permanent tables
You can’t use the following statements on a DB cluster with write forwarding enabled.

 

LOAD DATA INFILE 'data.txt' INTO TABLE t1;
LOAD XML LOCAL INFILE 'test.xml' INTO TABLE t1;
Plugin statements
You can’t use the following statements on a DB cluster with write forwarding enabled.

 

INSTALL PLUGIN example SONAME 'ha_example.so';
UNINSTALL PLUGIN example;
SAVEPOINT statements
You can’t use the following statements on a DB cluster when write forwarding is enabled within the session. You can use these statements on DB clusters that don’t have write forwarding enabled, or within sessions where the aurora_replica_read_consistency setting is blank. Check if your code uses these statements before enabling write forwarding within a session.

 

SAVEPOINT t1_save;
ROLLBACK TO SAVEPOINT t1_save;
RELEASE SAVEPOINT t1_save;

Isolation levels for write forwarding

In sessions that use write forwarding, you can only use the REPEATABLE READ isolation level. Although you can also use the READ COMMITTED isolation level with Aurora Replicas, that isolation level doesn’t work with write forwarding. For information about the REPEATABLE READ and READ COMMITTED isolation levels, see Aurora MySQL isolation levels.

Read consistency for write forwarding

You can control the degree of read consistency on a DB cluster. The read consistency level determines how long the DB cluster waits before each read operation to ensure that some or all changes are replicated from the writer. You can adjust the read consistency level to make sure that all forwarded write operations from your session are visible in the DB cluster before any subsequent queries. You can also use this setting to make sure that queries on the DB cluster always see the most current updates from the writer. This setting also applies to queries submitted by other sessions or other clusters. To specify this type of behavior for your application, choose a value for the aurora_replica_read_consistency DB parameter or DB cluster parameter.

Important

Always set the aurora_replica_read_consistency DB parameter or DB cluster parameter when you want to forward writes. If you don’t, then Aurora doesn’t forward writes. This parameter has an empty value by default, so choose a specific value when you use this parameter. The aurora_replica_read_consistency parameter only affects DB clusters or instances that have write forwarding enabled.

As you increase the consistency level, your application spends more time waiting for changes to be propagated between DB instances. You can choose the balance between fast response time and making sure that changes made in other DB instances are fully available before your queries run.

You can specify the following values for the aurora_replica_read_consistency parameter:

  • EVENTUAL – Results of write operations in the same session aren’t visible until the write operation is performed on the writer DB instance. The query doesn’t wait for the updated results to be available. Thus it might retrieve the older data or the updated data, depending on the timing of the statements and the amount of replication lag. This is the same consistency as for Aurora MySQL DB clusters that don’t use write forwarding.
  • SESSION – All queries that use write forwarding see the results of all changes made in that session. The changes are visible regardless of whether the transaction is committed. If necessary, the query waits for the results of forwarded write operations to be replicated.
  • GLOBAL – A session sees all committed changes across all sessions and instances in the DB cluster. Each query might wait for a period that varies depending on the amount of session lag. The query proceeds when the DB cluster is up-to-date with all committed data from the writer, as of the time that the query began.

For information about the configuration parameters involved in write forwarding, see Configuration parameters for write forwarding.

Note

You can also use aurora_replica_read_consistency as a session variable, for example:

 

mysql> set aurora_replica_read_consistency = 'session';

Examples of using write forwarding

The following examples show the effects of the aurora_replica_read_consistency parameter on running INSERT statements followed by SELECT statements. The results can differ, depending on the value of aurora_replica_read_consistency and the timing of the statements.

To achieve higher consistency, you might wait briefly before issuing the SELECT statement. Or Aurora can automatically wait until the results finish replicating before proceeding with SELECT.

For information on setting DB parameters, see Working with parameter groups.

Example with aurora_replica_read_consistency set to EVENTUAL

Running an INSERT statement, immediately followed by a SELECT statement, returns a value for COUNT(*) with the number of rows before the new row is inserted. Running the SELECT again a short time later returns the updated row count. The SELECT statements don’t wait.

 

mysql> select count(*) from t1;
+----------+
| count(*) |
+----------+
|        5 |
+----------+
1 row in set (0.00 sec)

mysql> insert into t1 values (6); select count(*) from t1;
+----------+
| count(*) |
+----------+
|        5 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from t1;
+----------+
| count(*) |
+----------+
|        6 |
+----------+
1 row in set (0.00 sec)
Example with aurora_replica_read_consistency set to SESSION

SELECT statement immediately after an INSERT waits until the changes from the INSERT statement are visible. Subsequent SELECT statements don’t wait.

 

mysql> select count(*) from t1;
+----------+
| count(*) |
+----------+
|        6 |
+----------+
1 row in set (0.01 sec)

mysql> insert into t1 values (6); select count(*) from t1; select count(*) from t1;
Query OK, 1 row affected (0.08 sec)
+----------+
| count(*) |
+----------+
|        7 |
+----------+
1 row in set (0.37 sec)
+----------+
| count(*) |
+----------+
|        7 |
+----------+
1 row in set (0.00 sec)

With the read consistency setting still set to SESSION, introducing a brief wait after performing an INSERT statement makes the updated row count available by the time the next SELECT statement runs.

 

mysql> insert into t1 values (6); select sleep(2); select count(*) from t1;
Query OK, 1 row affected (0.07 sec)
+----------+
| sleep(2) |
+----------+
|        0 |
+----------+
1 row in set (2.01 sec)
+----------+
| count(*) |
+----------+
|        8 |
+----------+
1 row in set (0.00 sec)
Example with aurora_replica_read_consistency set to GLOBAL

Each SELECT statement waits for all data changes, as of the start time of the statement, to be visible before performing the query. The wait time for each SELECT statement varies, depending on the amount of replication lag.

 

mysql> select count(*) from t1;
+----------+
| count(*) |
+----------+
|        8 |
+----------+
1 row in set (0.75 sec)

mysql> select count(*) from t1;
+----------+
| count(*) |
+----------+
|        8 |
+----------+
1 row in set (0.37 sec)

mysql> select count(*) from t1;
+----------+
| count(*) |
+----------+
|        8 |
+----------+
1 row in set (0.66 sec)

Running multipart statements with write forwarding

A DML statement might consist of multiple parts, such as an INSERT ... SELECT statement or a DELETE ... WHERE statement. In this case, the entire statement is forwarded to the writer DB instance and run there.

Transactions with write forwarding

If the transaction access mode is set to read only, write forwarding isn’t used. You can specify the access mode for the transaction by using the SET TRANSACTION statement or the START TRANSACTION statement. You can also specify the transaction access mode by changing the value of the transaction_read_only session variable. You can change this session value only while you’re connected to a DB cluster that has write forwarding enabled.

If a long-running transaction doesn’t issue any statement for a substantial period of time, it might exceed the idle timeout period. This period has a default of one minute. You can set the aurora_fwd_writer_idle_timeout parameter to increase it up to one day. A transaction that exceeds the idle timeout is canceled by the writer instance. The next subsequent statement you submit receives a timeout error. Then Aurora rolls back the transaction.

This type of error can occur in other cases when write forwarding becomes unavailable. For example, Aurora cancels any transactions that use write forwarding if you restart the DB cluster or if you disable write forwarding.

When a writer instance in a cluster using local write forwarding is restarted, any active, forwarded transactions and queries on reader instances using local write forwarding are automatically closed. After the writer instance is available again, you can retry these transactions.

Configuration parameters for write forwarding

The Aurora DB parameter groups include settings for the write forwarding feature. Details about these parameters are summarized in the following table, with usage notes after the table.

ParameterScopeTypeDefault valueValid values
aurora_fwd_writer_idle_timeoutClusterUnsigned integer601–86,400
aurora_fwd_writer_max_connections_pctClusterUnsigned long integer100–90
aurora_replica_read_consistencyCluster or instanceEnum” (null)EVENTUALSESSIONGLOBAL

To control incoming write requests, use these settings:

  • aurora_fwd_writer_idle_timeout – The number of seconds the writer DB instance waits for activity on a connection that’s forwarded from a reader instance before closing it. If the session remains idle beyond this period, Aurora cancels the session.
  • aurora_fwd_writer_max_connections_pct – The upper limit on database connections that can be used on a writer DB instance to handle queries forwarded from reader instances. It’s expressed as a percentage of the max_connections setting for the writer. For example, if max_connections is 800 and aurora_fwd_master_max_connections_pct or aurora_fwd_writer_max_connections_pct is 10, then the writer allows a maximum of 80 simultaneous forwarded sessions. These connections come from the same connection pool managed by the max_connections setting.This setting applies only on the writer when it has write forwarding enabled. If you decrease the value, existing connections aren’t affected. Aurora takes the new value of the setting into account when attempting to create a new connection from a DB cluster. The default value is 10, representing 10% of the max_connections value.
Note

Because aurora_fwd_writer_idle_timeout and aurora_fwd_writer_max_connections_pct are DB cluster parameters, all DB instances in each cluster have the same values for these parameters.

For more information about aurora_replica_read_consistency, see Read consistency for write forwarding.

For more information on DB parameter groups, see Working with parameter groups.

Amazon CloudWatch metrics and Aurora MySQL status variables for write forwarding

The following Amazon CloudWatch metrics and Aurora MySQL status variables apply when you use write forwarding on one or more DB clusters. These metrics and status variables are all measured on the writer DB instance.

CloudWatch metricAurora MySQL status variableUnitDescription
ForwardingWriterDMLLatencyMillisecondsAverage time to process each forwarded DML statement on the writer DB instance.

It doesn’t include the time for the DB cluster to forward the write request, or the time to replicate changes back to the writer.

ForwardingWriterDMLThroughputCount per secondNumber of forwarded DML statements processed each second by this writer DB instance.
ForwardingWriterOpenSessionsAurora_fwd_writer_open_sessionsCountNumber of forwarded sessions on the writer DB instance.
Aurora_fwd_writer_dml_stmt_countCountTotal number of DML statements forwarded to this writer DB instance.
Aurora_fwd_writer_dml_stmt_durationMicrosecondsTotal duration of DML statements forwarded to this writer DB instance.
Aurora_fwd_writer_select_stmt_countCountTotal number of SELECT statements forwarded to this writer DB instance.
Aurora_fwd_writer_select_stmt_durationMicrosecondsTotal duration of SELECT statements forwarded to this writer DB instance.

The following CloudWatch metrics and Aurora MySQL status variables are measured on each reader DB instance in a DB cluster with write forwarding enabled.

CloudWatch metricAurora MySQL status variableUnitDescription
ForwardingReplicaDMLLatencyMillisecondsAverage response time of forwarded DMLs on replica.
ForwardingReplicaDMLThroughputCount per secondNumber of forwarded DML statements processed each second.
ForwardingReplicaOpenSessionsAurora_fwd_replica_open_sessionsCountNumber of sessions that are using write forwarding on a reader DB instance.
ForwardingReplicaReadWaitLatencyMillisecondsAverage wait time that a SELECT statement on a reader DB instance waits to catch up to the writer.

The degree to which the reader DB instance waits before processing a query depends on the aurora_replica_read_consistency setting.

ForwardingReplicaReadWaitThroughputCount per secondTotal number of SELECT statements processed each second in all sessions that are forwarding writes.
ForwardingReplicaSelectLatencyMillisecondsForwarded SELECT latency, averaged over all forwarded SELECT statements within the monitoring period.
ForwardingReplicaSelectThroughputCount per secondForwarded SELECT throughput per second average within the monitoring period.
Aurora_fwd_replica_dml_stmt_countCountTotal number of DML statements forwarded from this reader DB instance.
Aurora_fwd_replica_dml_stmt_durationMicrosecondsTotal duration of all DML statements forwarded from this reader DB instance.
Aurora_fwd_replica_errors_session_limitCountNumber of sessions rejected by the primary cluster due to one of the following error conditions:

  • writer full
  • Too many forwarded statements in progress.
Aurora_fwd_replica_read_wait_countCountTotal number of read-after-write waits on this reader DB instance.
Aurora_fwd_replica_read_wait_durationMicrosecondsTotal duration of waits due to the read consistency setting on this reader DB instance.
Aurora_fwd_replica_select_stmt_countCountTotal number of SELECT statements forwarded from this reader DB instance.
Aurora_fwd_replica_select_stmt_durationMicrosecondsTotal duration of SELECT statements forwarded from this reader DB instance.

Identifying forwarded transactions and queries

You can use the information_schema.aurora_forwarding_processlist table to identify forwarded transactions and queries. For more information on this table, see information_schema.aurora_forwarding_processlist.

The following example shows all forwarded connections on a writer DB instance.

 

mysql> select * from information_schema.AURORA_FORWARDING_PROCESSLIST where IS_FORWARDED=1 order by REPLICA_SESSION_ID;

+-----+----------+--------------------+----------+---------+------+--------------+--------------------------------------------+--------------+--------------------+---------------------------------+----------------------+----------------+
| ID  | USER     | HOST               | DB       | COMMAND | TIME | STATE        | INFO                                       | IS_FORWARDED | REPLICA_SESSION_ID | REPLICA_INSTANCE_IDENTIFIER     | REPLICA_CLUSTER_NAME | REPLICA_REGION |
+-----+----------+--------------------+----------+---------+------+--------------+--------------------------------------------+--------------+--------------------+---------------------------------+---------------------------------------+
| 648 | myuser   | IP_address:port1 | sysbench | Query | 0 | async commit | UPDATE sbtest58 SET k=k+1 WHERE id=4802579 | 1 | 637 | my-db-cluster-instance-2 | my-db-cluster | us-west-2 | | 650 | myuser | IP_address:port2 | sysbench | Query | 0 | async commit | UPDATE sbtest54 SET k=k+1 WHERE id=2503953 | 1 | 639 | my-db-cluster-instance-2 | my-db-cluster | us-west-2 | +-----+----------+--------------------+----------+---------+------+--------------+--------------------------------------------+--------------+--------------------+---------------------------------+----------------------+----------------+

On the forwarding reader DB instance, you can see the threads associated with these writer DB connections by running SHOW PROCESSLIST. The REPLICA_SESSION_ID values on the writer, 637 and 639, are the same as the Id values on the reader.

 

mysql> select @@aurora_server_id;

+---------------------------------+
| @@aurora_server_id              |
+---------------------------------+
| my-db-cluster-instance-2        |
+---------------------------------+
1 row in set (0.00 sec)

mysql> show processlist;

+-----+----------+--------------------+----------+---------+------+--------------+---------------------------------------------+
| Id  | User     | Host               | db       | Command | Time | State        | Info                                        |
+-----+----------+--------------------+----------+---------+------+--------------+---------------------------------------------+
| 637 | myuser   | IP_address:port1 | sysbench | Query | 0 | async commit | UPDATE sbtest12 SET k=k+1 WHERE id=4802579 | | 639 | myuser | IP_address:port2 | sysbench | Query | 0 | async commit | UPDATE sbtest61 SET k=k+1 WHERE id=2503953 | +-----+----------+--------------------+----------+---------+------+--------------+---------------------------------------------+ 12 rows in set (0.00 sec)