Unhandled Exception Logging Causes Large Log Files

Under high load conditions, the system experiences a significant increase in log file size due to unhandled exceptions being repeatedly logged. Specifically, when a WriterThread encounters an exception (e.g., NotServingRegionException), the exception is stored in the controller but is neither removed nor updated. Consequently, within the RegionReplicaReplicationEndpoint.replicate method, a while loop responsible for replication checks for errors in each iteration. While it successfully logs the caught exception via checkForErrors() method, it fails to address or resolve the exception. This results in a continuous cycle of exception logging without any corrective action, leading to disproportionately large log file sizes (~2GB in approximately 5 minutes).

Impact: Elevated log file sizes hinder log management and monitoring processes. Increased disk usage due to large log files may affect system performance and scalability.

incorrect exception handling

https://issues.apache.org/jira/browse/HBASE-23601

7 2 1 1 Apr. 4, 2024, 2:50 PM

Authors

Launch on Chameleon

Launching this artifact will open it within Chameleon’s shared Jupyter experiment environment, which is accessible to all Chameleon users with an active allocation.

Download Archive

Download an archive containing the files of this artifact.

Version Stats

5 2 1