November News From the Mozilla DB Team

You may have noticed that I stopped posting the “weekly news” from the Mozilla DB Team. After going through the Operations Report Card and applying it to DBAs in OurSQL Podcast numbers 111, 112, 114, 115 and 116, I started thinking that the updates were really more like metrics, and it would better serve my own purposes better to do the updates monthly.

The purposes of doing this type of blog post are:
0) Answering “So what does a DBA do, anyway?”
1) Answering “DBA? At Mozilla? Does Firefox have a database? Why does Mozilla have databases, and what for?”
2) Showing what the DB team does for Mozilla, so that folks will understand that “just keeping things working” is actually a lot of work. It also helps compile yearly reviews of accomplishments.

We are also starting to get some metrics information. This month we started easy – number of MySQL and Postgres machines, number of unique databases (mysql, information_schema, performance_schema and test are ignored, and duplicates, like the same database on a master and 2 slaves, are ignored), and version information.

As of today, we have 9 unique databases across 8 Postgres servers in 4 clusters, with 6 being Postgres 9.0 and 2 being Postgres 9.2 – we are currently upgrading all our Postgres databases to 9.2 and expect that by the end of December all servers will be using 9.2.

We have 427 unique databases across 98 MySQL DB machines in 20 clusters, with 3 being MySQL 5.0, 71 being MySQL 5.1 (mostly Percona’s patched 5.1), and 24 being MariaDB 5.5.

And in the last week of October and the month of November, we have:

• Documented 4 more of our Nagios checks.
  
• Started to upgrade Postgres databases to Postgres 9.2
  
• Decommissioned a legacy database cluster for Firefox themes.
  
• Built a new database cluster (complete with monitoring and backups) for a new Sentry implementation.
  
• Upgraded X machines for general operating system updating purposes and to ensure that be2net drivers are up-to-date; out-of-date drivers can (and have!) caused servers to crash. (b1-db1, b1-db2, addons1, addons2, 3,4,5,
  
• Upgraded MySQL 5.0 to MySQL 5.1 across X clusters and Y machines (a01, b02, b2, 7)
  
• Did a quarterly purge of Crash Stats data.
  
• Had to re-sync 6 slaves when a transaction rolled back on the master, but some of the tables modified were MyISAM. So the master had data in some tables that was out of sync with the slaves.
  
• Assisted in converting to use UTC timestamps in the Elmo database behind the Mozilla localization portal and the Bugzilla Anthropology Project, prompting a blog post on converting timezone-specific times in MySQL.
  
• Decommissioned a legacy “production generic” database cluster that had over 60 databases on it.
  
• Built a 5th database backup instance due to growing backup needs.
  
• Changed binary log format to MIXED on our JIRA installation due to JIRA requirements and an upgrade to MySQL 5.1 issuing warnings that MySQL 5.0 had not.
  
• Added checksums to the database cluster that runs Input and Firefox about:home snippets.
  
• Archived and dropped the database behind Rock Your Firefox.
  
• Exported Bugzilla data for a research project. Did you know if you are doing academic research, you can get a copy of Mozilla’s public Bugzilla data?
  
• Gave read-only database access to a developer behind the Datazilla project.
  
• Updated the email list for vouched Mozillians.
  
• Backfilled missing crash-stats data after some failed cron scripts.
  
• Cleared some junk data from the Datazilla databases.
  
• Added new custom fields to our implementation of Bugzilla for upcoming release versions: Firefox 20, Thunderbird 20, Thunderbird ESR 20 and seamonkey 217.
  
• Added 10 new machines to the Graphs database, and added new sets of machines for Mozilla ESR 17 and Thunderbird ESR 17.
  
• Gave read-only database access to the two main leads of the Air Mozilla project.
  
• Debugged and turned off a 10-second timeout in our load balancing pool that was causing Postgres monitors and processors to lose connection to their databases.
  
• Discovered that the plugins database actually does better with the query_cache turned on, and tuned its size.
  
• Tweaked tokudb_cache_size and innodb_buffer_pool_size on our Datazilla databases so that less swap would be used.
  
• Created 2 read/write accounts for 2 people to access the development database for Mozillians.
  
• Gave access to Datazilla databases for staging.
  

Wednesday, Nov 28th was my 1-year anniversary at Mozilla. Tomorrow is December! 2012 went by very quickly.