Uncategorized
S3CMD continued
Ok, I’ve been using the s3cmd more and more lately.
s3cmd has a sync and delete command that can actually speed up deletes significantly. If you want to delete a few thousand files or maybe a million instead of taking days with other tools you can blast through deleting s3 objects by running several processes of s3cmd at once.
There are 2 ways to delete files.
s3cmd del --recursive --force s3://bucketname
s3cmd sync --delete-removed /tmp/empty s3://bucketname
/tmp/empty is an empty directory.
Get 5-20 processes with these commands running and you’ll be motoring along. I’ve read horror stories about people trying to delete a billion records it costing them lots of money because it takes a long time.
Doing 5 processes I’m deleting probably 5x as fast as 1 process right now so I’m happy with the delete recursive command.
SQL finding duplicate records.
When adding unique indexs to tables to prevent data duplication you discover there are already duplicates in the table you wish to fix/improve.
Here’s a quick way to list records with duplicates.
select count( id ) as cnt, id from table
group by id
having cnt > 1
You still have to go through and figure out what data is valid and such but it definitely saves time to run that query first. In MySQL you can run the following query to view all of the records in a table that are duplicate (make’s it easier to know which ones to delete then).
select * from table where id in (
select id from (
select count( id ) as cnt, id from table group by id
having cnt > 1 ) as table_alias )
order by id
This will list of the rows that have duplicates in order so you can see which records to delete in your favorite mysql client.
If you want something simpler and are not necessarily interested in what record might have better data.
Follow these instructions here on creating a new table to get rid of duplicate rows
Web based chat and shout boxes.
Here are some of the solutions I’ve found or have dealt with lately when trying to find the perfect solution to website(s) I’m working on.
Web based chat and shout boxes. - continue reading …PHP Performance.
I’ve noticed the performance of my PHP is dog slow. I’ve since put in some debugging code and performance measuring code. My code might not be as modular or pretty looking but at least it will work in a reasonable amount of time. Object oriented PHP is causing me many performance headaches. It was disgustingly slow.
Here’s a link to probably the best guide/page on the net on PHP optimisation.
http://www.wardontheweb.com/10-performance-tips-to-speed-up-php/
Another decent site/page with citations/references:
http://www.hm2k.com/posts/50-php-optimisation-tips-revisited
Problem Solved
It turns out much of my class headaches came from an extremely slow classloader function.
So if you are having issues with PHP class loading look into what your class loader is doing.
Mine was searching everywhere for the class and this was wasting gobs of time.
Meanwhiles all classes were in 2 directories.
Yum and Git on Centros 4/4.5
Trying to install software on linux sometimes can be a real pain.
Sure yum is great but some hosts such as MediaTemple don’t have that installed.
So here’s a list of the commands I had to run in order to get Yum installed
on a Centros 4.* box in an attempt to install another program:
Everything for centros 4.* is located here now => http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/rpm-libs-4.3.3-32_nonptl.i386.rpm http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/rpm-4.3.3-32_nonptl.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/popt-1.9.1-32_nonptl.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/rpm-python-4.3.3-32_nonptl.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/libxml2-2.6.16-12.6.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/libxml2-python-2.6.16-12.6.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/python-elementtree-1.2.6-5.el4.centos.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/sqlite-3.3.6-2.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/python-sqlite-1.1.7-1.2.1.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/rpm-python-4.3.3-32_nonptl.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/python-urlgrabber-2.9.8-2.noarch.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/yum-metadata-parser-1.0-8.el4.centos.i386.rpm
rpm -Uvh http://mirror.centos.org/centos/4/os/i386/CentOS/RPMS/yum-2.4.3-4.el4.centos.noarch.rpm
After installing that amazing number of packages for yum. It turns out the program I wanted wasn’t even available from yum. So I looked elsewhere.
To install “Git” you can do the following:
rpm -Uvh http://download.fedora.redhat.com/pub/epel/4/i386/git-core-1.5.4.7-3.el4.i386.rpm http://download.fedora.redhat.com/pub/epel/4/i386/perl-Git-1.5.4.7-3.el4.i386.rpm
Everything is downloaded from Fedora Redhat.
It runs ~/git woo. Now to see if I can use git to fetch my source/respository.
Perforce/VSS/CSV and Subversion: Mistakes Made
A Bug/Version control software story:
I figured I would share one of my most embarassing development stories with the whole wide internet. This happened about 10 years ago and never happened again since so I think I’ve learned from the mistake.
I had been working on a particularly crappy piece of code that no one else in the company wanted to do. I call it crappy because no one in their right mind would go about trying to modify MFC code to do what I was doing.
I was Hacking MFC to make tab panes for dialogs that had a particular behaviour. At the time I had barely an inkling of win32 coding at the time and modifying MFC is was a lesson in pain I wouldn’t want to inflict on my worst enemy.
After spending nearly a week or more I had just finished the tabbed dialogs and went to check in my completed work.
I had not checked in my work for nearly a week but since everything seemed to work AOK I went to check it in.
However, instead of checking in my code I mistakenly got the lastest code I checked in. It overwrote all of my work and changes up until that point.
Lets just say I had a panic attack. I’ve had 1 case before that where someone else had overwritten my code but up until that point I’ve never had that myself. In the previous case I was able to retrieve the code because it was on a FAT disk system. For whatever reason this time around I wasn’t able to get my code back. (NTFS ARGGG)
All I know is I didn’t retrieve files this time around and within a couple weeks I was out of a job.
What did I learn from that mistake?
1. Check your code in daily. If the class compiles. Check it in.
2. Use an editor that has a local history. (eclipse, etc)
That way if you do “get latest” over some file you can restore it.
Occationally I do “get lastest” over code I had not intended. However with the tools and experience
I’ve found the mistakes can be minimized.
Javascript prompt() replacements
javascript prompt() replacement.
I wanted a simple to use function to ask the user for some information.
The javascript prompt() function seemed like a good way of doing this until
you notice the security issues with IE.
So I visited google and started looking for replacements.
http://www.anyexample.com/webdev/javascript/ie7_javascript_prompt()_alternative.xml
Tried it out. Works. Didn’t like the design.
http://www.skybound.nl/products/javascript/Prompt/
Tried it. Works. Not polished.
http://meteora.astrata.com.mx/
Tried it. Works. Works and looks nice.
http://abeautifulsite.net/2008/12/jquery-alert-dialogs/
Tried it. Using it based on suggestion from another developer.
Finding and Removing duplicate data from SQL Tables
Here’s an decent article with various methods of finding duplicate data in databases with SQL.
http://www.delphifaq.com/faq/delphi/database/f20.shtml
Removing duplicate data can be a troublesome task as well. One of the fastest ways to do so involves creating a new table with a unique index to the column you wish to purge of duplicate data.
Then you copy data over form the original table to the new using a insert ignore into table ( columns ) select columns from table type of query. Don’t forget the “ignore” otherwise the batch insert will fail at the first case of a duplicate piece of data.
Comment on Finding and Removing duplicate data from SQL Tables
Determine if mod_rewrite is enabled using php/environment vars
A useful way to determine if mod_rewrite is enabled using php/environment vars. I googled this post and thought it was useful. This is useful for anyone switching between windows development + a linux deployment /w mod rewrite. It was on the 2nd or 3rd page of my google results so I’m putting it here 
http://christian.roy.name/blog/detecting-modrewrite-using-php
Comment on Determine if mod_rewrite is enabled using php/environment vars
S3 Amazon Simple Storage Service
So yeah I’ve been looking around for tools I can use for Amazon S3.
I originally used S3fox. It’s a useful browser plugin for Firefox. Works great for small jobs and copy single
files and small directories of files. However if you want something a little more robust the best tool
out there is s3cmd.
I’ve only been using it for a bit but it seems to do everything this programmer could want.
It’s a command line tool for linux. It’s pretty good so far I’ve copied 4900 images with it in a single bound.
It died when I tried copying over 100000 files but yeah that was probably the operating system
wimpering and not s3cmd. I switched to s3cmd put a* s3://bucket/directory/ instead to send
all of the files that start with the letter a first to split things up. That worked fine.
Installation was a breeze on centros linux.
# yum install s3cmd
# s3cmd –configure
Then once that’s all done you can start running these commands.
s3cmd ls
s3cmd put * s3://bucketname/directory/
s3cmd get s3://bucketname/directory/file
I was a little disappointed that the s3cmd –help didn’t list the commands. You have to sort of look around on the website. I’ve listed a few common commands here. There are others listed on the website. I use s3fox for making directories and buckets but here are the commands for s3cmd.
s3cmd mb s3://bucketname
// makes a bucket
s3cmd rb s3://buckettoremove
// removes a bucket
Check out here for more information and documentation..
The man pages for ubuntu actually have better documentation then the original website
Finally I was having issues running s3cmd from php shell_exec on certain boxes/configurations.
