Blue Topaz Games
A developers blog.

Javascript Libraries and Tools.

Here are some javascript libraries and tools that I find useful:

http://jquery.com/

http://www.bitrepository.com/ajax-login-modal-box.html

http://tablesorter.com/docs/

Comment on Javascript Libraries and Tools.

MySQL Insert

Ok, this is a simple post on something that really is annoying about MySQL and Navicat 8.

Basically anytime you execute a single INSERT statement it’s slow as heck on MySQL.
Very very very very slow.
I honestly think there is a bug / very poor implementation issue that should be fixed.

For instance I was inserting about 22,000 rows using an SQL file produced by a tool called Navicat 8.
It took about 5 minutes to do that. That’s really not an acceptable amount of time when you have users like me waiting on that data being inserted.

So in light of this after a bit of googling I found an actual page on the mysql site highlighting
Speed of Insert statements.

Specifically it mentions that you can insert more then 1 row per an insert statement.
OK? So lets try that out:

BOOM 22,000 records inserted in less then one SECOND.

Now that’s what I’m talking about. I’ve taken 22,000 inserts and made them take about the same amount of time as 1 single insert. It’s incredible.

Here’s an actual example of multiply row inserts per a single insert statement.

LOCK TABLES a WRITE;  // makes the inserts even faster
INSERT INTO table_a VALUES (1,23),(2,34),(4,33); // insert 3 rows into table_a
INSERT INTO table_b VALUES (8,26),(6,29); // insert 2 rows into table_b
...
UNLOCK TABLES;

I’m not sure who to blame more here MySql 5 or Navicat 8.

For MySql 5 there shouldn’t be such a big difference in speed between a single insert and an insert /w many rows at once.

Navicat 8 has a poorly implemented Export Wizard for SQL files. The alernative to Navicat 8 export is mysqldump –user username –password database table which is what most people use when their GUI tool breaks down like this.

Originally I was going to blame PHP for my problems but after writing and rewriting a function to load the SQL files from DISK I found that the speed issues were really MySql’s problem worsened by the SQL queries produced by Navicat 8.

Here’s a simple function I wrote for reading a SQL file from disk and installing it to whatever database you specify in the settings.

	/**
	 * Simple function for installing sql files.
	 * http://dev.mysql.com/doc/refman/5.0/en/insert.html
	 *
	 * **WARNING**
	 * A single insert with many values is about 1000000x faster
	 * then multiple inserts due to way mysql is coded.
	 * Use text search and replace to fix multiply inserts into a single insert.
	 * **WARNING**
	 *
	 * @param unknown_type $filename An SQL file to read in.
	 */
	function fast_install( $filename )
	{
		$mysqli = new mysqli();

		$mysqli->connect( DB_SERVER, DB_USERNAME, DB_PASSWORD, DATABASE );

		$content = file_get_contents( $filename );

		// set a 16 meg limit on the query
		$result = $mysqli->multi_query( "SET GLOBAL max_allowed_packet=16*1024*1024;" ) or die( mysqli_error( $mysqli ) );

		$result = $mysqli->multi_query( $content ) or die( mysqli_error( $mysqli ) );

		$mysqli->close();
	}

Hadoop: Buggy on Windows.

So yeah running hadoop on a windows box has a few bugs / issues that will definitely frustrate you as a developer.

If you suddenly start getting strange errors from hadoop and can’t seem to fix them by either restarting hadoop with stop-all.sh/start-all.sh the next best thing is just to clean that sucker up.

How do you clean up the hadoop distributed filesystem?

1. bin/stop-all.sh
2. delete the temporary files/directories c:\tmp
3. bin/hadoop namenode -format
4. bin/start-all.sh
5. bin/hadoop dfs -put local_dir dfs_dir

Of course that means you just formated/removed all of the data you put into the DFS but if all your doing is testing and development it shouldn’t be a big deal to put that data right back.
Here’s an example of an error that’s nearly unfixable and will just pop up and ruin your day for no reason:

09/04/13 14:48:35 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: jav
a.io.IOException: File /user/Adminestrator/input/better.PNG could only be replicated to 0 nodes, instead of 1

Bummer.