Billed volume storage on our Aurora RDS MySQL cluster went up from 70GB to 1200GB within few hours and it is just not getting down.
AWS premium support seems to be clueless. They made us increase version of Aurora RDS MySQL 5.7 from 2.09.1 to 2.10.0 saying that there is some bug in currently used version and that space should become free on reboot. We did the upgrade, manually rebooted the cluster post upgrading but it made no difference to billed volume storage.
Space actually used by our application database is 69GB including indexes. Free space in this database is 15GB so total used space should be ~85GB. There are no binary logs, temporary tables, we are also not using replicas (this is a single node cluster).
SELECT table_schema "DataBase Name", sum( data_length + index_length ) / 1024 / 1024 / 1024 "Occupied Space in GB", sum( data_free )/ 1024 / 1024 / 1024 "Free Space in GB", sum( data_length + index_length + data_free ) / 1024 / 1024 / 1024 "Total Database Size in GB" FROM information_schema.TABLES GROUP BY table_schema;
| DataBase Name | Occupied Space in GB | Free Space in GB | Total Database Size in GB |
+--------------------+----------------------+------------------+---------------------------+
| information_schema | 0.000198364258 | 0.875976562500 | 0.876174926758 |
| app920 | 69.161712646484 | 15.512695312500 | 84.674407958984 |
| mysql | 0.019073486328 | 925.045898437500 | 925.064971923828 |
| performance_schema | 0.000000000000 | 0.000000000000 | 0.000000000000 |
| sys | 0.000015258789 | 21.512695312500 | 21.512710571289 |
+--------------------+----------------------+------------------+---------------------------+
5 rows in set (0.05 sec)
Timing of the sudden spike on 26th June matches with maintenance window that we set for this cluster (it is our night time when there is no traffic). We suspect something went wrong during the maintenance window. Our application makes no use of internal databases like mysql. There were also no scheme changes made by us.
We want to understand what made Volume Bytes Used become so high here and how to prevent it?
