6.3. Optimizing database

The heart of the netflow monitoring is MySQL database. This database consumes most of the memory and utilizes the majority of the CPU and disc space. For this reason optimize your database server. Carefully read MySQL documentation and especially the chapter on "Optimizing the MySQL Server". The MySQL documentation can be obtained from URL http://www.mysql.org/doc/.

In most cases the configuration is in file /etc/mysql/my.cnf or /etc/my.cnf.

On systems with two processors and a 1GB memory use the following configuration:

[mysqld]
skip-innodb         # CFI doesn't use INNO DB engine, so you can disable it.

key_buffer_size     = 260M  # See below.

max_connections     = 30    # See below.
wait_timeout        = 180   # Reduced to prevent idle clients holding connections.
table_cache         = 1024  # See below.

max_allowed_packet  = 16M
sort_buffer         = 12M   # See below.
read_buffer         = 2M    # See below.
read_rnd_buffer     = 2M    # See below.

myisam_sort_buffer  = 32M
bulk_insert_buffer  = 32M

tmp_table_size      = 256M  # See below.
max_heap_table_size = 256M

query_cache_type    = 1     # Enable query caching.
query_cache_limit   = 1M
query_cache_size    = 32M   # See below.

thread_cache        = 20    # See below.
thread_concurrency  = 2     # thread_concurrency = 2 * (number of CPU)

#log                = /var/log/mysql/mysql.log     # Disable queries logging.
#log-bin            = /var/log/mysql/mysql-bin.log # Disable binary query logging.
log-error           = /var/log/mysql/mysql.err     # Enable error logging.

On systems with 8GB of memory we recommend using the following configuration: key_buffer=1500M, myisam_sort_buffer=128M, max_heap_table_size=1250M, tmp_table_size=1250M, sort_buffer=32M, read_buffer=16M, read_rnd_buffer=16M, and table_cache=4096.

[Note]Note
Before MySQL 4.0.2, the only syntax for setting program variables was --set-variable=option=value. This syntax is still recognized, but is deprecated as of MySQL 4.0.2.

Some of the important MySQL variables:

key_buffer_size

  • The value of key_buffer_size is the size of the buffer used with indexes. The larger the buffer, the faster the SQL command will finish and a result will be returned. The rule-of-thumb is to set the key_buffer_size to at least a quarter, but no more than half, of the total amount of memory on the server. Ideally, key_buffer_size will be large enough to contain all the indexes (the total size of all .MYI files on the server).
  • A simple way to check the actual performance of the buffer is to examine four additional variables: key_read_requests, key_reads, key_write_requests, and key_writes.
  • If you divide the value of key_read by the value of key_reads_requests, the result should be less than 0.01. Also, if you divide the value of key_write by the value of key_writes_requests, the result should be less than 1.

max_connections

The number of simultaneous client connections allowed, this is 100 by default. Increasing max_connections value increases the number of file descriptors that MySQL requires. For CFI software we recommend setting max_connections parameter to: number_of_collectors + number_of_max_online_users + (number_of_units * 4) + 10 [reserve] Max_connections value is usually it is more or less 40.

table_cache

The value for table_cache is 64 by default. Each time MySQL accesses a table, it places it in the cache. If the system accesses many tables, it is faster to have these in the cache. MySQL, being multi-threaded, may be running many queries on the table at one time, and each of these will open a table. Examine the value of open_tables at peak times. If you find it stays at the same value as your table_cache value, and the number of opened_tables starts rapidly increasing, you should increase the table_cache if you have enough memory. We recommend setting this value in the range 512 to 4096.

sort_buffer

The sort_buffer value is very useful for speeding up myisamchk operations (this is why it is set much higher in the default configuration files), but it can also be useful everyday when performing large numbers of sorts.

read_buffer

Each thread that does a sequential scan allocates a buffer of this size (in bytes) for each table it scans. If you do many sequential scans, you might want to increase this value. Before MySQL 4.0.3, this variable was named record_buffer. For CFI software we recommend setting this parameter to size of your disk cache. Usually it is value between 1MB and 32MB.

read_rnd_buffer

The read_rnd_buffer is used after a sort, when reading rows in sorted order. If you use many queries with ORDER BY, increasing read_rnd_buffer can improve performance. Remember that, unlike key_buffer_size and table_cache, read_rnd_buffer is allocated for each thread. This read_rnd_buffer was renamed from record_rnd_buffer in MySQL 4.0.3. It defaults to the same size as the read_buffer_size. A rule-of-thumb is to allocate 1KB for each 1MB of memory on the server, for example 1MB on a machine with 1GB memory.

tmp_table_size

"Created_tmp_disk_tables" are the number of implicit temporary tables on disk created while executing statements and "created_tmp_tables" are memory-based. It is more effecient if you go to the memory instead of the disk all the time.

query_cache_size

MySQL 4 provides one feature that can prove very handy - a query cache. In a situation where the database has to repeatedly run the same queries on the same data set, returning the same results each time, MySQL can cache the result set, avoiding the overhead of running through the data over and over. This feature is extremely helpful on busy servers.

thread_cache

If you have a busy server that's getting a lot of quick connections, set your thread cache high enough that the thread's cache created value in SHOW STATUS stops increasing. This action should take some of load off the CPU.

[Note]Note
Don't forget to restart MySQL after making all changes.
[Tip]Tip
MySQLTuner is a high-performance MySQL tuning script written in Perl that will provide you with a snapshot of a MySQL server’s health. Based on statistics gathered, specific recommendations will be provided that will increase a MySQL server’s efficiency and performance. The script gives you automated MySQL tuning that is on the same level as you would receive from a MySQL DBA.
[Tip]Tip
If your collector consumes a lot of CPU you can use another server and move several collectors on to the second unit (server).