~jk - Home

Finding memleaks in win32 with valgrind

2010-06-02T18:45:00Z

I use MacOS X for most of my development work, but part of my job is writing apps that run on all major Unixes and Win32. Most of the time bugs and memleaks are "portable", meaning that they appear on all platforms in the same way and fixing them on your dev-platforms, fixes them everywhere else.

Well, "most of the time".

Today I had the joy of a win32 only mem-leak. We actually ''bisect''ed it down to the locking support for openssl that we just added http://www.openssl.org/docs/crypto/threads.html. But starring at the code didn't reveal anything obvious. So, back to the tools. Let them find it.

Going through the web on the search for "how to find mem-leaks on win32" I came along:

http://wiki.winehq.org/Wine_and_Valgrind

The idea is simple:

use wine to run the win32 app
run wine in valgrind

You get all the benefits of valgrind and can work on the OS you are used to. But not so fast:

wine from MacPorts is 32bit only http://trac.macports.org/browser/trunk/dports/x11/wine-devel/Portfile

... but all your macports (at least on Mac OS X 10.6) dependencies are 64bit only. After rebuilding the dependencies with +universal, wine starts up nicely:

[jan@macbook bin]$ file mysql-proxy.exe 
mysql-proxy.exe: PE32 executable for MS Windows (console) Intel 80386 32-bit
[jan@macbook bin]$ wine mysql-proxy.exe --version
mysql-proxy 0.8.0
  chassis: mysql-proxy 0.8.0
  glib2: 2.20.3
  libevent: 1.4.12-stable
  LUA: Lua 5.1.2
    package.path: Z:\Users\jan\projects\in-targz\mysql-monitor-agent-2.2.0.1717-win32-x86\lib
    package.cpath: Z:\Users\jan\projects\in-targz\mysql-monitor-agent-2.2.0.1717-win32-x86\bin\lua-?.dll
-- modules
  admin: 0.7.0
  proxy: 0.7.0

valgrind fails to build as +universal if built from MacPorts http://trac.macports.org/browser/trunk/dports/devel/valgrind-devel/Portfile

... and ends up as 64bit app ... which can't run a 32bit 'wine'. Back to "build valgrind by hand". Using the Portfile as template I set up the valgrind source tree and built it with '--enable-32bitonly' to get a 32bit valgrind.

Everything in place that I need I did hit a bug in valgrind: https://bugs.kde.org/show_bug.cgi?id=194757

Anyway, some real hints how to track down the mem-leaks on win32:

MySQL 5.5's Semi Sync Replication: the protocol side

2010-04-10T21:20:00Z

I'm preparing the code for my MySQLConf 2010 session "MySQL Proxy meets: Memcache" where I'll present how to replicate from MySQL to memcache by using the MySQL Proxy.

Part of it will be using the semi-sync replication support for MySQL 5.5 to implement a synchronous MySQL-to-Memcache replication. All I need is the network protocol definition for semi-sync ...

The semi-sync replication is a implemented as replication plugin and lives in:

plugins/semisync/

The master sends a binlog event with a indication that it wants to get a reply for it. Compare these two events:

[0000]  28 00 00 01|00|00 00 00 00 04 01 00 00 00 27 00  (.............'.
[0010]  00 00 00 00 00 00 20 00 14 01 00 00 00 00 00 00  ...... .........
[0020]  6c 6f 63 61 6c 2e 30 30 30 30 30 31              local.000001

... and now with semi-sync:

[0000]  2a 00 00 01|00|ef 00|00 00 00 00 04 01 00 00 00  *...............
[0010]  27 00 00 00 00 00 00 00 20 00 14 01 00 00 00 00  '....... .......
[0020]  00 00 6c 6f 63 61 6c 2e 30 30 30 30 30 31        ..local.000001

The 0xef is the semi-sync magic and the 0x00 it follows is the semi-sync flag. As it is 0, the master doesn't expect us to ACK it.

If the slave side is asked to ACK a command it sends back:

/* The layout of a semisync slave reply packet:
   1 byte for the magic num
   8 bytes for the binlog positon
   n bytes for the binlog filename, terminated with a '\0'
*/
#define REPLY_MAGIC_NUM_LEN 1
#define REPLY_BINLOG_POS_LEN 8
#define REPLY_BINLOG_NAME_LEN (FN_REFLEN + 1)
#define REPLY_MAGIC_NUM_OFFSET 0
#define REPLY_BINLOG_POS_OFFSET (REPLY_MAGIC_NUM_OFFSET + REPLY_MAGIC_NUM_LEN)
#define REPLY_BINLOG_NAME_OFFSET (REPLY_BINLOG_POS_OFFSET + REPLY_BINLOG_POS_LEN)

Beware, the binlog filename is 255 chars max: http://bugs.mysql.com/bug.php?id=52748

The master will wait until a slave sent a ACK to its event before it commits the transaction.

[0000]  15 00 00 00|ef|fe 03 00 00 00 00 00 00|6c 6f 63  .............loc
[0010]  61 6c 2e 30 30 30 30 30 32                       al.000002

Import is that you send the packet-id 0x00. Otherwise the master will ignore the packet. The master will send a OK packet as response to your ACK:

[0000]  07 00 00 01|00 00 00 02 00 00 00

MySQL Proxy: 0.8.0 released

2010-01-26T14:30:00Z

MySQL Proxy 0.8.0 has been released at http://dev.mysql.com/downloads/mysql-proxy/

While it took a while to get it out, it contains a lot of good stuff and will make a few more things possible for the future. First of all, it is the first release with a threaded network-backend allowing to use plenty of CPUs.

The other big infrastructure change is mostly for developers: out-of-tree plugins can now be written. Now you can write plugins for the MySQL Proxy against a installed MySQL Proxy. No need to have the MySQL Proxy's source code around.

ChangeLog

added a threaded event handling layer
added support to change the auth-response on the way to the backend
added timing infrastruture
added out-of-tree plugins builds and pkg-config support
fixed handling of broken length encoded data in resultsets
fixed version encoding in win32 (#45996)
fixed too small listen backlog() (#43278)
fixed configure check if flex is really installed (#45766)
fixed handling of >16M packets (#35202)
fixed parsing of invalid IP-addresses (#47273)
fixed off-by-one line-numbers in Lua stackstraces (#47348)

MySQL Proxy: FrosCon 2009

2009-08-18T08:37:00Z

I'll be at the OpenSQLCamp 2009 which is part of the FrosCon this weekend and give a talk about drumroll MySQL Proxy.

http://programm.froscon.org/2009/track/OpenSQLCamp/index.de.html has all the sessions incl. my

MySQL Proxy: a MySQL toolbox

Architecture and Concepts of misuse

It will be an extension of my MySQL Conference talk from this year with binlogs, frm-files, ... and what happens if you expose MySQLs core functionality as libraries and wrap it with a scripting layer.

MySQL Proxy: profiling 0.8

2009-08-06T17:27:00Z

In MySQL Proxy 0.8 we are added a multi-threaded network-subsystem allowing several networks events be processed in parallel. Early benchmarks show that what we have in trunk basicly works.

But the benchmarks weren't as good as we expected. That's the time where you prepare to get dirty.

While Kay went with lockstat to analyze the proxy on solaris and found that g_atomic_int_get() isn't using native code if built with Sun's CC, I attacked the Linux side with oprofile.

After rebuilding the proxy with -fno-omit-frame-pointer I got the information I was looking for from oprofile:

$ opcontrol --vmlinux=...
$ opcontrol --callgraph=5
$ opcontrol --separate=lib,kernel
$ opcontrol --start
$ sysbench ...
$ opcontrol --stop
$ opreport \
    --debug-info \
    --callgraph \
    --symbols \
    --merge tgid \
    '.../mysql-proxy'
....
-------------------------------------------------------------------------------
  1224     20.3966  event.c:468                 libevent-1.4.so.2.1.1    event_base_loop
  4777     79.6034  network-mysqld.c:446        libmysql-proxy.so.0.0.0  plugin_call
102069    2.7567  (no location information)   libpthread-2.5.so        pthread_mutex_lock
  102069   100.000  (no location information)   libpthread-2.5.so        pthread_mutex_lock [self]
...

The global Lua lock, the lock we wanted to remove for 0.9 anyway. Perhaps we can do something simple and not take the lock if there is no Lua script loaded. In 0.9 we will change the implementation and give each connection its own Lua state instead of shared the state between all connections.

MySQL Proxy: Roles

2009-06-25T15:54:00Z

On the MySQL Proxy channel we get questions from time to time if the authentication can be intercepted and replaced data from a external source.

From now on, you can. For example if you want to get data from a external source (like LDAP) or want to implement roles.

Mapping Accounts to "Roles"

There isn't much needed to implement Roles for MySQL with the help of the MySQL Proxy.

mysql.user doesn't contain users, but roles instead
the proxy maps user-accounts to role-accounts with a script like above

It works like this:

$ mysql --user=jan --password=secret --port=4040

proxy looks up username password, finds a role for him
proxy replaces credentials ad hoc
mysql-server sees the role-name and role-password and lets the user in

This works nicely as long as you only have 1:n mapping between roles and users. One user has one role, but one role has many users.

All you need is a storage that has user's password in one of three forms:

cleartext as in the example below
SHA1(cleartext) as you may have in LDAP
SHA1(SHA1(cleartext)) as in mysql.user: SELECT CONCAT("*", SHA1(UNHEX(SHA1("pass")))), PASSWORD("pass");

How the Auth works

The MySQL Proxy has access to all protocol data and exposes it as part of different stages of the authentication to the scripting layer:

---
-- show how to use the mysql.password functions 
function read_auth()
    local c = proxy.connection.client
    local s = proxy.connection.server
    print(("for challenge %q the client sent %q"):format(
            s.scramble_buffer,
            c.scrambled_password
    ))
end

The challenge and response secures the password and is works like this:

SERVER:  scramble_buffer=create_random_string()
         send(scramble_buffer)

CLIENT:  recv(scramble_buffer)
         hash_stage1=sha1("password")
         hash_stage2=sha1(hash_stage1)
         scrambled_password=xor(hash_stage1, sha1(scramble_buffer, hash_stage2)

         send(scrambled_password)

SERVER:  recv(scrambled_password)
         hash_stage1=xor(scrambled_password, sha1(scramble_buffer,hash_stage2))
         candidate_hash2=sha1(hash_stage1)
         check(candidate_hash2==hash_stage2)

That's taken from sql/password.c of the MySQL Server Source. The hash_stage2 is the same as result of the PASSWORD() function without the * and the HEX() and is stored in mysql.user.

Replacing the auth response

Based on the read_auth() from above you can also replace the credentials on the fly:

    ---
    -- map replace:me with root:secret
    --
    if c.username == "replace" and -- the username we want to map
            password.check(
                    s.scramble_buffer,
                    c.scrambled_password,
                    password.hash(password.hash("me")) -- its valid password
            ) then 

            proxy.queries:append(1,
                    proto.to_response_packet({
                            username = "root", 
                            response = password.scramble(s.scramble_buffer, password.hash("secret")),
                            charset  = 8, -- default charset
                            database = c.default_db,
                            max_packet_size = 1 * 1024 * 1024
                    })
            )

            return proxy.PROXY_SEND_QUERY  -- works if you use lp:mysql-proxy r694 or later
    end

http://bazaar.launchpad.net/~mysql-proxy-developers/mysql-proxy/trunk/annotate/head%3A/examples/tutorial-scramble.lua

you can see the full version of it.

Let's see if it works:

$ mysql-proxy --proxy-lua-script=tutorial-scramble.lua
$ mysql --port=4040 --user=replace --password=me
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2702
Server version: 5.1.34-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

root@127.0.0.1 [(none)]>

I logged in as the root user.

Fixing the hostcheck

This can also be used to rewrite the username to something that includes his original ip. Like

...,
username = c.username .. "-" .c.src.address,
...

to make the server see something else than the proxy's client-address if you use the host-field in the mysql.user tables.

MySQL Proxy: Hackathon 2009

2009-06-05T08:25:00Z

At MySQL Conference in Santa Clara this April the idea was brought up, now we do it:

a MySQL Proxy hackathon this summer in Sun Office in Hamburg, Germany.

We want bring together everyone who is interested in

hacking on MySQL Proxy,
sharing ideas how it is used and
how it should develop.

More at http://forge.mysql.com/wiki/MySQL_Proxy_Hackathon_2009

Wormhole SE: JOINs

2009-05-28T16:21:00Z

In my rare spare time work on the http://jan.kneschke.de/2009/5/4/binlog-storage-engine-mysql-proxy-edition and for a few days I was wondering why my index-based JOINs didn't worked. With a index it works, with an index nothing is returned.

I narrowed it down to a simple test-case:

select * from test AS a JOIN test AS b USING (event_pos) WHERE a.event_pos IN (106);
...
1 row in set (0.03 sec)

EXPLAIN
+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------+
| id | select_type | table | type  | possible_keys | key     | key_len | ref   | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------+
|  1 | SIMPLE      | a     | const | PRIMARY       | PRIMARY | 4       | const |    1 |       | 
|  1 | SIMPLE      | b     | const | PRIMARY       | PRIMARY | 4       | const |    1 |       | 
+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------+

vs.

select * from test AS a JOIN test AS b USING (event_pos) WHERE a.event_pos IN (106, 177);
0 row in set (0.03 sec)

EXPLAIN ...
+----+-------------+-------+--------+---------------+---------+---------+------------------+------+-------------+
| id | select_type | table | type   | possible_keys | key     | key_len | ref              | rows | Extra       |
+----+-------------+-------+--------+---------------+---------+---------+------------------+------+-------------+
|  1 | SIMPLE      | a     | range  | PRIMARY       | PRIMARY | 4       | NULL             |    2 | Using where | 
|  1 | SIMPLE      | b     | eq_ref | PRIMARY       | PRIMARY | 4       | test.a.event_pos |    1 |             | 
+----+-------------+-------+--------+---------------+---------+---------+------------------+------+-------------+

A self-JOIN on the PRIMARY KEY with two PK-values that exist.

The EXPLAINs look fine, eq_ref is what we want. But, we get no rows back. Looking at the debug output from MySQL and the wormhole storage engine it all looked fine. The right functions are called, the right return values are given.

At #mysql-dev Sergey Petrunia did take a look at my problem and we stepped through the code until I stumbled over:

static int
join_read_key(JOIN_TAB *tab)
{
    ...
    return table->status ? -1 : 0;
}

(gdb) p table->status
3 (aka STATUS_NO_RECORD)

Scanning the code in sql/sql_select.cc where that comes from:

case JT_EQ_REF:
  table->status=STATUS_NO_RECORD;
  if (tab->select) 
...

Who is supposed to change that value ? Me ? Checking the other storage engines it looks like there are two return codes: the obvious one and the hidden one. Depending on the execution plan (as seen above) one or the other is used. grmpf

Long story, simple fix:

=== modified file 'src/ha_wormhole.cc'
--- src/ha_wormhole.cc  2009-05-25 20:55:14 +0000
+++ src/ha_wormhole.cc  2009-05-28 13:28:06 +0000
@@ -1071,6 +1071,8 @@
        if (0 == wormhole_hook_call(L, 2)) {
                /* no thing pushed */
        } else if (lua_istable(L, -1)) {
+               table->status = 0; /** _WOOT_ ... we have to set it to 0 to make the eq_ref work */
+
                lua_getrow();
                lua_pop(L, 1); /* pop the table */
                DBUG_RETURN(0);

in the storage-engine's index_read() function and my JOINs on index work.

MySQL Proxy: 0.7.1 released

2009-05-23T11:09:00Z

We are happy to announce that MySQL Proxy 0.7.1 is available in a source and binary release for many popular platforms.

This release contains a few minor bugfixes and changes in directory layout over the previous 0.7.0 release.

moved plugins to lib/mysql-proxy/plugins
moved lua modules to lib/mysql-proxy/lua
moved libs to lib/

Please report any problems on http://bugs.mysql.com, our Launchpad discussion mailing list at https://launchpad.net/~mysql-proxy-discuss or on IRC: #mysql-proxy on irc.freenode.net.

Please note that the binary for Windows is currently still the old 0.6.1 release and will be updated soon.

ChangeLog

fixed connection close at COM_REFRESH
fixed detection of the --basedir for relative execs
fixed decoding of binlog.event_type on sparc
fixed build on freebsd and hpux

Binlog Storage Engine - MySQL Proxy Edition

2009-05-04T10:30:00Z

A binlog storage engine was talked about many times already. Being able to JOIN against the binlogs, the just SELECT it, search for entries ... is not possible with current SQL commands in the MySQL Server.

As you have read the previous articles here, I wrote a binlog encoder/decoder for MySQL Proxy and wrapped with with a Lua interface. Some years ago I also wrote a scriptable storage engine which can call lua scripts to return data for SELECTs. Putting the two together gives us: a Binlog Storage Engine.

In the datadir we just create

$ ls -la test/binlog_*.*
-rw-r--r--  1 jan  staff  1342  4 Mai 12:30 test/binlog_000016.lua

mysql> select * from test.binlog_000016;
Query OK, 0 rows affected (0.00 sec)

+----------------------------------+------------+----------------+
| type                             | event_size | next_event_pos |
+----------------------------------+------------+----------------+
| FORMAT_DESCRIPTION_EVENT         |        102 |            106 |
| QUERY_EVENT                      |         71 |            177 |
| QUERY_EVENT                      |         71 |            248 |

That binlog_000016 table is a .lua script that the scriptable storage engine can load:

local binlog = assert(require("mysql.binlog"))

local binlog_table = { }

-- Lua does OOP and and inheritance ...
function binlog_table:new(o)
    o = o or {}
    setmetatable(o, self)
    self.__index = self
    return o
end

As we don't want to mess with CREATE TABLE statements, we just create the .frm files on the fly by providing a discover() function.

-- return a the table structure
function binlog_table:discover()
    return {
            { name = "type", type = 254, length = 32 },
            { name = "event_size", type = 3, length = 32 },
            { name = "next_event_pos", type = 3, length = 32 }
    }
end

After SELECTing from the table the first time the mysql-server will create the .frm files for us automaticly:

$ ls -la test/binlog_*.*
-rw-rw----  1 jan  staff  8652  4 Mai 12:29 test/binlog_000016.frm
-rw-r--r--  1 jan  staff  1342  4 Mai 12:30 test/binlog_000016.lua

The most basic command is get row for a table-scan. Our :next() method sets up the iterator and returns the iterator-function that is called until it returns nil.

-- returns a iterator
function binlog_table:next()
    -- set the iterator
    self.f:seek(4)
    self.f_iter = self.f:next()

    if not self.f_iter then return nil end
    -- return the iterator function
    return function ()
            self.event = self.f_iter()

            if not self.event then return nil end

            return { self.event.type, self.event.event_size, self.event.log_pos }
    end
end

To create our binlog table we just inherit from that binlog_table class and return it:

-- this table is a binlog-table
local tbl = binlog_table:new({ })

function tbl:open()
    self.f = binlog.open("/Users/jan/sandboxes/msb_5_1_35/data/msandbox.000016")

    return (self.f ~= nil)
end

return tbl

The last piece is the storage engine plugin:

$ mysql <<EOF
FLUSH TABLES;
UNINSTALL PLUGIN lua;
EOF
$ make install
$ mysql << EOF
INSTALL PLUGIN lua SONAME 'libwormhole_engine.so';
EOF

If everything works out nicely you should see:

$ mysql 
mysql> show plugins;
+------------+--------+----------------+-----------------------+---------+
| Name       | Status | Type           | Library               | License |
+------------+--------+----------------+-----------------------+---------+
...
| LUA        | ACTIVE | STORAGE ENGINE | libwormhole_engine.so | GPL     |
...

Update

Properly normalized you can now decode the different events by just JOINing the ..._query table based on the event_pos

> select * from test.test_query AS q join test.test AS e using (event_pos) where e.type = "QUERY_EVENT" and event_pos = 106;
+-----------+-------------+-----------+--------------+-----------+-------------+------------+-----------------+------------+
| event_pos | type        | thread_id | query        | server_id | type        | event_size | next_event_posH | timestamp  |
+-----------+-------------+-----------+--------------+-----------+-------------+------------+-----------------+------------+
|       106 | QUERY_EVENT |         2 | FLUSH TABLES |         1 | QUERY_EVENT |         71 |             177 | 1241432054 | 
+-----------+-------------+-----------+--------------+-----------+-------------+------------+-----------------+------------+

And that is supposed to be fast as event_pos is our PRIMARY KEY:

> explain select * from test.test_query AS q join test.test AS e using (event_pos) where e.type = "QUERY_EVENT" and event_pos = 106;
+----+-------------+-------+------+---------------+---------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key     | key_len | ref   | rows | Extra       |
+----+-------------+-------+------+---------------+---------+---------+-------+------+-------------+
|  1 | SIMPLE      | q     | ref  | PRIMARY       | PRIMARY | 5       | const |   10 | Using where |
|  1 | SIMPLE      | e     | ref  | PRIMARY       | PRIMARY | 5       | const |   10 | Using where |
+----+-------------+-------+------+---------------+---------+---------+-------+------+-------------+

On the implementation side the ..._query table just inherits from the base-table and overwrites the discover (for the new table-structure) and the get_row (to return the corresponding information) methods. Everything else stays the same.

local binlog_table = assert(require("wormhole.binlog_table"))

-- this table is a binlog-table
local tbl = binlog_table:new()

function tbl:discover()
    return {
            { name = "event_pos", type = 3, is_nullable = false, is_primary = true },
            { name = "type", type = 254, length = 64 * 3, is_nullable = false },
            { name = "thread_id", type = 3, is_nullable = false },
            { name = "query", type = 254, length = 256, is_nullable = false },
    }
end

function tbl:get_row()
    local e = self.event
    if e.type == "QUERY_EVENT" then
            return { e.log_pos - e.event_size,
                    e.type,
                    e.query.thread_id,
                    e.query.query,
            }
    else
            return { e.log_pos - e.event_size,
                    e.type,
                    nil,
                    nil
            }
    end
end

function tbl:open()
    self.f = binlog.open("/Users/jan/sandboxes/msb_5_1_35/data/msandbox.000016")

    return (self.f ~= nil)
end

return tbl

MySQLs hidden Loader Interface

2009-04-28T11:58:00Z

After getting the http://jan.kneschke.de/2009/4/22/druckbetankung stuff working and discussing the idea with the replication team we figured out that my presented approach has a nice hack-value, but otherwise is perhaps a bit too complicated. The same result can be achieved by a more simplified approach:

turn our input data into a RBR based base64-encoded BINLOG-stmt using binlog
use the mysql client to push the data into the server

The BINLOG stmt is used by mysqlbinlog to pass the RBR-events through the mysql client to the mysql-server. Usually it wraps a FORMAT_DESCRIPTION_EVENT and the TABLE_MAP_EVENT and the *_ROWS_EVENTS. BUT ... it can handle any binlog-event. Just try

$ mysqlbinlog --base64-output=always

to see how that looks like.

A generated example looks like this:

# FORMAT_DESCRIPTION
BINLOG '5hX3SQ8CAAAAZgAAAGYAAAAAAAQANS4xLjI2LW15c3FsLXByb3h5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADmFfdJEzgNAAgAEgAEBAQEEgAAUwAEGggAAAAICAgC';
# DROP TABLE IF EXISTS ...
BINLOG '5hX3SQICAAAAPgAAAKQAAAAAAAEAAAAAAAAABAAAAAB0ZXN0AERST1AgVEFCTEUgSUYgRVhJU1RTIHB1bXA=';
# CREATE TABLE 
BINLOG '5hX3SQICAAAAYgAAAAYBAAAAAAEAAAAAAAAABAAAAAB0ZXN0AENSRUFURSBUQUJMRSBwdW1wICggZjEgVkFSQ0hBUigxNikgTk9UIE5VTEwsIGYyIElOVCwgZjMgSU5UICk=';
# BEGIN
BINLOG '5hX3SQICAAAAKgAAADABAAAAAAEAAAAAAAAABAAAAAB0ZXN0AEJFR0lO';
# INSERT ...
BINLOG '5hX3SQICAAAAUQAAAIEBAAAAAAEAAAAAAAAABAAAAAB0ZXN0AElOU0VSVCBJTlRPIHB1bXAgVkFMVUVTICggJ2l0IHdvcmtzJywgMSwgMiAp';
 # TABLE_MAP_EVENT + WRITE_ROWS_EVENT
BINLOG '5hX3SRMCAAAALwAAALABAAAAAAEAAAAAAAAABHRlc3QABHB1bXAAAw8DAwIQAAY=
5hX3SRcCAAAANwAAAOcBAAAAAAEAAAAAAAkAAwcAA2FiYwIAAAADAAAAAANkZWYDAAAABQAAAA==';
# INSERT ...
BINLOG '5hX3SQICAAAATQAAADQCAAAAAAEAAAAAAAAABAAAAAB0ZXN0AElOU0VSVCBJTlRPIHB1bXAgVkFMVUVTICggJ2RvbmUnLCAxLCAyICk=';
# XID .. (aka COMMIT)
BINLOG '5hX3SRACAAAAGwAAAE8CAAAAALYKAAAAAAAA';

Feel free to execute it in your client. It will create a table test.pump and insert 4 rows into it. 2 with normal INSERT stmts, 2 with a RBR-WRITE_ROWS_EVENT.

To make it a loader interface what is faster then LOAD DATA INFILE or multi-row INSERT you have to convert all your data into WRITE_ROWS_EVENTs and push it. As the data is already in the native form it should/may be faster than the 2 other approaches.

Now the interesting question: how much faster is a WRITE_ROWS_EVENT with BASE64 encoding than a multi-row INSERT ?

Druckbetankung

2009-04-22T18:18:00Z

... or how to misuse RBR to fill a MySQL table in the hopefully fastest way.

In my session yesterday I didn't had time to talk about this. The slide-desk covers it and I still wanted to document the idea at least.

The starting point is "Row Based Replication stores data in internal MySQL row-format." The slave can apply these events without any SQL parsing involved.

The idea is to create a master plugin that

takes a file or something else that you want to INSERT into the mysql-server
creates a binlog-stream using row-based replication events from it
expose the binlog-stream and pretend to be a mysql-master (listen on port :3306 and handle COM_BINLOG_DUMP)
let the MySQL-server connect to master-plugin using CHANGE MASTER TO and START SLAVE and fetch and apply the binlog-stream natively

The final implementation is in replication-tree.

The master plugin takes a lua script when does all the magic and will hide the details. To give you an idea what is going on behind the scenes a few snippets of Lua code, like how the different binlog-events are represented:

    {
            server_id = 2,
            type = "QUERY_EVENT",
            query = {
                    db_name = "test",
                    query = "DROP TABLE IF EXISTS pump"
            }
    },
    {
            server_id = 2,
            type = "QUERY_EVENT",
            query = {
                    db_name = "test",
                    query = "CREATE TABLE pump ( f1 CHAR(16) NOT NULL, f2 INT, f3 INT )"
            }
    },

generates the two QUERY_EVENTs to create the table we want to insert into:

    -- define a table-map event matching our CREATE TABLE from above
    {
            server_id = 2,
            type = "TABLE_MAP_EVENT",
            table_map = {
                    table_id = 1,
                    db_name = "test",
                    table_name = "pump",
                    fields = {
                            { type = proxy.MYSQL_TYPE_STRING, is_nullable = false },
                            { type = proxy.MYSQL_TYPE_LONG, is_nullable = true },
                            { type = proxy.MYSQL_TYPE_LONG }, -- is_null defaults to true
                    },
            }
    },
    -- full the table with 2 rows
    {
            server_id = 2,
            type = "WRITE_ROWS_EVENT",
            rbr = {
                    table_id = 1,
                    flags    = 0,
                    rows = {
                            {
                                    before  = { "abc", 2, 3 },
                            },
                            {
                                    before  = { "def", 3, 5 },
                            },
                    }
            }
    },

adds the two events that represent a INSERT statement for Row-based-replication-speak. The events are passed to the binlog-generator and are pass out as response to a COM_BINLOG_DUMP response:

local f = binlog.new()
f:seek(4)
... 
local event = f:to_event(event_tbl[event_ndx])
return "\000" ... f:append(event)

To give it a test we load the master-plugin and the script

    $ mysql-proxy \
    --plugins=master \
    --master-address=:4041 \
    --master-lua-script=master.lua

and let the MySQL Server connect to it:

STOP SLAVE;
RESET SLAVE;
CHANGE MASTER TO master_host='127.0.0.1', \
    master_port=4041, \
    master_user='root', \
    master_password='secret';
START SLAVE;
-- importing done ... let's verify it
USE test;
SHOW TABLES;
SELECT SLEEP(0.1); -- wait until the slave has caught up
SELECT * FROM pump;

outputs:

+-----+------+------+
| f1  | f2   | f3   |
+-----+------+------+
| abc |    2 |    3 |
| def |    3 |    5 |
+-----+------+------+

MySQL Proxy meets: binlog - the examples

2009-04-21T18:28:00Z

I just pushed the code for my replication changes on launchpad:

$ bzr branch lp:~jan-kneschke/mysql-proxy/replication

The presentation should be available ... soon.

One of the first examples is about filtering binlogs. If you want to remove all ALTER statements from the replication stream (or in this case the binlog files) you can just iterate the binlog and copy everything to a new binlog, but the ALTER statement

local binlog = require("mysql.binlog")
local tokenizer = require("proxy.tokenizer")

local f_in = assert(binlog.open("/tmp/binlog-test.log"))
local f_out = assert(binlog.open("/tmp/binlog-test-filtered.log", "w"))
print("filtering /tmp/binlog-test.log to /tmp/binlog-test-filtered.log")

for event in f_in:next() do
    if event.type == "QUERY_EVENT" then
            local tokens = tokenizer.tokenize(event.query.query)
            local first_token = tokenizer.first_stmt_token(tokens)

            if first_token.token_name == "TK_SQL_ALTER" then
                    print("removing " .. event.query.query)
            else
                    -- log it
                    f_out:append(event)
            end
    else
            f_out:append(event)
    end
end

f_in:close()
f_out:close()

Straight forward.

You can also merge binlogs like it happens in sharded setups:

-- open the files 
local in_files = {
    assert(binlog.open("/tmp/binlog-merge-1.log")),
    assert(binlog.open("/tmp/binlog-merge-2.log"))
}
f_merged = assert(binlog.open("/tmp/binlog-merged.log", "w"))
print("merging to /tmp/binlog-merge.log")

local iterators = {}

-- get the iterators for all the binlogs
for i, f in ipairs(in_files) do
    iterators[i] = f:next()
end

local cur_events = {}
for i, f in ipairs(iterators) do
    cur_events[i] = f()
end

--
-- copy the oldest timestamp from all the binlogs to the out-binlog
repeat
    -- get the event with the oldest timestamp
    local o_ndx
    local o_ts
    for i = 1, #cur_events do
            local event = cur_events[i]
            if event and (not o_ts or event.timestamp < o_ts) then
                    o_ndx = i
                    o_ts = event.timestamp
            end
    end

    -- append the oldest event
    if o_ndx then
            f_merged:append(cur_events[o_ndx])
            cur_events[o_ndx] = iterators[o_ndx]()
    end
until not o_ndx


for i, f in ipairs(in_files) do
    f:close()
end

f_merged:close()

That's just the foundation for more fun stuff like: "synchronized replication". More of that in the session.

MySQL Proxy meets: binlogs - sneak peek II

2009-04-18T16:13:00Z

The slides are written, uploaded and the code snippets work. I'm ready to present.

One of the topics will be merging binlogs which is what multi-master replication is all about on the low level. A common example is sharding where you have several masters which share the same table-structures, but store the data independently. This is great for scaling out, but tricky if you have to run a query over the full dataset.

The logfiles look like:

$ mysqlbinlog binlog-merge-1.log -s
SET TIMESTAMP=1240066015/*!*/;
INSERT INTO tbl VALUES ( 1 )
/*!*/;
SET TIMESTAMP=1240066017/*!*/;
INSERT INTO tbl VALUES ( 3 )
/*!*/;
# End of log file
ROLLBACK /* added by mysqlbinlog */;

and

$ mysqlbinlog binlog-merge-2.log -s
SET TIMESTAMP=1240066016/*!*/;
INSERT INTO tbl VALUES ( 2 )
/*!*/;
SET TIMESTAMP=1240066018/*!*/;
INSERT INTO tbl VALUES ( 4 )
/*!*/;
# End of log file
ROLLBACK /* added by mysqlbinlog */;

As the two binlogs are independent and can't contain collisions we can just merge the two binlogs based on the timestamp:

$ mysqlbinlog binlog-merged.log -s
SET TIMESTAMP=1240066015/*!*/;
INSERT INTO tbl VALUES ( 1 )
/*!*/;
SET TIMESTAMP=1240066016/*!*/;
INSERT INTO tbl VALUES ( 2 )
/*!*/;
SET TIMESTAMP=1240066017/*!*/;
INSERT INTO tbl VALUES ( 3 )
/*!*/;
SET TIMESTAMP=1240066018/*!*/;
INSERT INTO tbl VALUES ( 4 )
/*!*/;
# End of log file
ROLLBACK /* added by mysqlbinlog */;

In my session I'll show you what is going in the background to merge any number of binlogs into one binlog. It is less than 50 lines of lua-code.

Just add a network-interface to it and another problem to make MySQL "sharding-complete" is solved.

MySQL Proxy: meets binlogs - sneak peek I

2009-04-15T10:33:00Z

... is the title of my session at the MySQL Conf 2009

I'll unveal the full thing at the session at next week Tuesday, but for now let me show you this:

local f = binlog.open("my-binlog", "w")
f:append({       
    server_id = 1,
    type = "INCIDENT_EVENT",
    incident = {
        incident = 42,
        message = "The answer"
    }       
})
f:close()

... gives me:

$ mysqlbinlog my-binlog
...
#090415 12:02:32 server id 1  end_log_pos 350 
# Incident: UåìEuø}ü}
                        DED$F$ègö# at 350
...

What ever I did: I can break mysqlbinlog :)

Hint: the above is lua code. A working encoder/decoder for binlogs with a c-library and a lua-wrapper on top.

If you have some ideas what this could be useful for add it as comment to this article and let's see what I can show at the MySQL Conf.