Clker.com - weblog http://www.clker.com/blog Just another WordPress weblog Mon, 12 May 2008 15:55:57 +0000 http://wordpress.org/?v=2.5.1 en Caching SQL results with PHP http://www.clker.com/blog/2008/05/12/caching-sql-results-with-php/ http://www.clker.com/blog/2008/05/12/caching-sql-results-with-php/#comments Mon, 12 May 2008 15:54:36 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/05/12/caching-sql-results-with-php/ I’ve been looking around lately on the best way to cache SQL results in PHP. I found some interesting articles posted in lots of places, but I didn’t find any that exactly matches my needs. The problem I have on hand is basically the same every growing website faces: decreasing mean resource usage per page request.

Now - this is my plan A to keep up with the website’s growth without a lot of hardware upgrades. There is a plan B, but I will keep that to a later post.

Usually, the time consumed by executing the SQL statements is more than 80% of the whole script time. Reducing this time will significantly reduce the resource usage. CPU is not the bottle neck, usually SQL queries result in a significant disk access especially if there’s a lot of inserts/updates/deletes.

The easiest and best way I found is to use PHP’s serialize to store the SQL results. This requires that we fetch all the rows we’re dealing with. So the sequence of operations is to check the cache file, if expired or does not exist then fetch all the rows from the result set, serialize in one variable, write it to the cache file.

  1. <?
  2. require_once "utils.php";
  3.  
  4. $cache_prefix=‘/tmp/clkercache/’;
  5.  
  6. function cache_fetch($filename, $expiration=3600)
  7. {
  8.   global $cache_prefix;
  9.   $filename=$cache_prefix.$filename;
  10.  
  11.   if (!file_exists($filename)) return false;
  12.   if (time(0)-filemtime($filename)>$expiration) return false;
  13.  
  14.   $data=unserialize(base64_decode(file_get_contents($filename)));
  15.  
  16.   return $data;
  17. }
  18.  
  19. function cache_store($filename,$data)
  20. {
  21.   global $cache_prefix;
  22.   $filename=$cache_prefix.$filename;
  23.  
  24.   @mk_dir($filename);
  25.   return file_put_contents($filename,base64_encode(serialize($data)));
  26. }
  27. ?>
  28.  

So the way to use the caching functions would be something similar to this:

  1. <?
  2. if ( !($pictures=cache_fetch("mypictures.cache")) ){
  3.   $res=query("select pictureid from pictures order by rating15days desc limit 100");
  4.   while ($row=fetch_array($res))
  5.     $pictures[]=$row;
  6.  
  7.   cache_store("mypictures.cache",$pictures);
  8. }
  9. ?>

So how to do that if you have multiple queries? The way I store my cache files is by giving them suffix or prefix names depending on what is being stores. For example a file that store the pictures that are related to a picture whose id is 1500 being displayed, will be called related-1500.cache. For more complicated cases like searches, I would use the md5sum of the request URI.

The results were good. Using apache bench I tested the homepage. Before any caching it used to support 3 pages per second. After caching the result was 35. I think in my current situation with my current traffic, this is a very good result.

One might think so what does it take to generate pages at a much higher rate - say 300, 400 or even a 1000 or more? Well, that’s my plan B, and I’ll discuss that in more detail in another post. This plan B will generate that much pages per second.

Technorati Tags: , , , , , ,

]]> http://www.clker.com/blog/2008/05/12/caching-sql-results-with-php/feed/ Gatner “Windows is collapsing” http://www.clker.com/blog/2008/04/11/gatner-windows-is-collapsing/ http://www.clker.com/blog/2008/04/11/gatner-windows-is-collapsing/#comments Fri, 11 Apr 2008 15:28:17 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/04/11/gatner-windows-is-collapsing/ So Gartner, one of the largest analysis companies said in a conference that windows is collapsing. Lots of reasons were given including a large code for windows that became un-maintainable and lead Microsoft to start from a more stable release of Windows 2003, and improve on for Vista. This in turn lead to Vista not really delivering meaningful enhancements from the point of view of the end consumer. Also, the fact that applications now are moving on the web and becoming OS agnostic where you can run many available software almost on all operating systems (openoffice for example, inkscape and others based on GNU’s compilers or java).

Other reasons included a hardware demanding OS, which can’t fit on small PDA’s and thus giving the chance for Linux and OS X. I add to this list, the complete prevalence of Linux in embedded OSs in appliances like Tivos, with zero competition from Microsoft.

A big major contribution from my point of view is the reason of making a new release. When a company decides to allocate resources for the development of a software, they need goals. The only goal that is obvious from the release of Vista was to duress people into DRM, using windows position as a must be OS on every PC or  laptop. A very important lesson that you can’t force the whole world into only your own vision, but you have to embrace what people want, make it your vision and may be add more on the side. After all people are the consumer. Unfortunately, that was the second time Microsoft did that, after they already suffered losses on the server side and hosting.

However, I do disagree with Gartner on the result that Microsoft or windows will crash. I believe that Microsoft is trying to embrace the standards and work with others, it’s just they’re not doing enough and lots of people doubt their intentions. My view is Microsoft becoming more like SUN or IBM where they will have to work with people and provide solutions for the problems that people see using ways that the people want.

Technorati Tags: , , , , , , , , , , ,

]]> http://www.clker.com/blog/2008/04/11/gatner-windows-is-collapsing/feed/ Multiple SVG uploads http://www.clker.com/blog/2008/04/07/multiple-svg-uploads/ http://www.clker.com/blog/2008/04/07/multiple-svg-uploads/#comments Mon, 07 Apr 2008 16:14:18 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/04/07/multiple-svg-uploads/ Now you can upload multiple files at ones. Zip all your SVG images in one zip file, and upload the zip file. The upload page will then loop through all SVG files stores inside the zip, and will try to add them.  Upload limit is still 2MBytes.

Technorati Tags: , , ,

]]> http://www.clker.com/blog/2008/04/07/multiple-svg-uploads/feed/ Creating a tar gz on the fly using PHP http://www.clker.com/blog/2008/03/27/creating-a-tar-gz-on-the-fly-using-php/ http://www.clker.com/blog/2008/03/27/creating-a-tar-gz-on-the-fly-using-php/#comments Thu, 27 Mar 2008 17:41:53 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/03/27/creating-a-tar-gz-on-the-fly-using-php/ A while ago, I thought about creating a tar.gz file for every download, so that if someone runs a search, he/she then can download all the images in the results. After a little bit of research, I found that PHP has a function for gzip. I also knew that the tar format just sticks files after one another, so if I can implement the tar format in PHP then I can gzip all images in the results.

I found this LGPL code that implemented the tar format. I used it (and modified it a little bit) to produce the online tar.gz functions:

  1. // Computes the unsigned Checksum of a file’s header
  2. // to try to ensure valid file
  3. // PRIVATE ACCESS FUNCTION
  4. function __computeUnsignedChecksum($bytestring)
  5. {
  6.   for($i=0; $i<512; $i++)
  7.     $unsigned_chksum += ord($bytestring[$i]);
  8.   for($i=0; $i<8; $i++)
  9.     $unsigned_chksum -= ord($bytestring[148 + $i]);
  10.   $unsigned_chksum += ord(" ") * 8;
  11.  
  12.   return $unsigned_chksum;
  13. }
  14.  
  15. // Generates a TAR file from the processed data
  16. // PRIVATE ACCESS FUNCTION
  17. function tarSection($Name, $Data, $information=NULL)
  18. {
  19.   // Generate the TAR header for this file
  20.  
  21.   $header .= str_pad($Name,100,chr(0));
  22.   $header .= str_pad("777",7,"0",STR_PAD_LEFT) . chr(0);
  23.   $header .= str_pad(decoct($information["user_id"]),7,"0",STR_PAD_LEFT) . chr(0);
  24.   $header .= str_pad(decoct($information["group_id"]),7,"0",STR_PAD_LEFT) . chr(0);
  25.   $header .= str_pad(decoct(strlen($Data)),11,"0",STR_PAD_LEFT) . chr(0);
  26.   $header .= str_pad(decoct(time(0)),11,"0",STR_PAD_LEFT) . chr(0);
  27.   $header .= str_repeat(" ",8);
  28.   $header .= "0";
  29.   $header .= str_repeat(chr(0),100);
  30.   $header .= str_pad("ustar",6,chr(32));
  31.   $header .= chr(32) . chr(0);
  32.   $header .= str_pad($information["user_name"],32,chr(0));
  33.   $header .= str_pad($information["group_name"],32,chr(0));
  34.   $header .= str_repeat(chr(0),8);
  35.   $header .= str_repeat(chr(0),8);
  36.   $header .= str_repeat(chr(0),155);
  37.   $header .= str_repeat(chr(0),12);
  38.  
  39.   // Compute header checksum
  40.   $checksum = str_pad(decoct(__computeUnsignedChecksum($header)),6,"0",STR_PAD_LEFT);
  41.   for($i=0; $i<6; $i++) {
  42.     $header[(148 + $i)] = substr($checksum,$i,1);
  43.   }
  44.   $header[154] = chr(0);
  45.   $header[155] = chr(32);
  46.  
  47.   // Pad file contents to byte count divisible by 512
  48.   $file_contents = str_pad($Data,(ceil(strlen($Data) / 512) * 512),chr(0));
  49.  
  50.   // Add new tar formatted data to tar file contents
  51.   $tar_file = $header . $file_contents;
  52.  
  53.   return $tar_file;
  54. }
  55.  
  56. function targz($Name, $Data)
  57. {
  58.   return gzencode(tarSection($Name,$Data),9);
  59. }
  60.  

To use those functions all you have to do is send a header with the mime type for the tar gz ( application/x-gzip ) using the php header function. To add a tar/gz section for a file, read the file in an array using filegetcontents and pass the filename and data to the targz function. Echo what is returned. That’s it!

So why is it not active on clker.com website? I actually tried it and found that compression consumes a lot of CPU. In the first 20 minute I had more than one hundred connections for different users downloading their results and the CPU was saturated. This basically left no CPU for searching. So use it carefully, and only if you really need that functionality.

Technorati Tags: , , , ,

]]> http://www.clker.com/blog/2008/03/27/creating-a-tar-gz-on-the-fly-using-php/feed/ Website updates http://www.clker.com/blog/2008/03/05/website-updates/ http://www.clker.com/blog/2008/03/05/website-updates/#comments Wed, 05 Mar 2008 17:15:15 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/03/05/website-updates/ I’ve been recently working on the website theme, colors both on the weblog side and the main website’s side. I believe it looks much nicer and less crowded this way. I will be synchronizing the style sheet of Wordpress and the website’s main part so that visitors and users won’t feel they’ve moved into a different place.

I also added a privacy policy page, still have to link that to the footer of the blog. Sometimes, after commits and updates some functionalities are broken. I will be checking and testing around the website to make sure everything still works. You can leave a comment here if you face.

I’ve been experimenting spatial image querying lately. Hopefully, below every image spatially similar ones will be showing soon regardless whether they were tagged or no. There’s a lot of work involved here, and hopefully I will roll out soon at least some experimental version to see how users react to it.

Technorati Tags: , , , ,

]]> http://www.clker.com/blog/2008/03/05/website-updates/feed/ Storing images in your sql database versus filesystem http://www.clker.com/blog/2008/03/03/storing-images-in-your-sql-database-versus-filesystem/ http://www.clker.com/blog/2008/03/03/storing-images-in-your-sql-database-versus-filesystem/#comments Tue, 04 Mar 2008 04:56:10 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/03/03/storing-images-in-your-sql-database-versus-filesystem/ Many websites today rely on media be it images or videos. One reason is humans are visual creatures, they like seeing things versus reading long articles.

Images on websites can be divided into two different types: Images used in the website theme including logos, rounded corners, backgrounds …etc. and images used as content. Obviously, the ones discussed here are the content images. Images used in the website theme will be accessed frequently and almost with every page view, and usually they are very few and better managed by storing them in a directory.

However, when dealing with images that build the content of the websites developers will tend sometimes to store them in their sql databases rather than just storing references to them. There are pros and cons for each approach:

Storing the images outside the database:

For a very long time, this approach has been considered the most appropriate approach. The binaries itself gets stores outside the database, while references are stored inside the database. This type of implementation is driven by the ideas that:

  1. It will be faster to access images off the hard drives, than through the database.
  2. It will take less space, since the database servers will usually add more records for locking and several other extra functionalities.

While actually true, problems start once the number of images grow. Although one can maintain referential integrity inside the database, there is nothing that will stop an image from being deleted from the directory structure. Also, nothing will stop and image from being deleted while its record still exists in the database resulting in dangling pointers.

A third and more serious problem is enforcing access rules. One of the biggest implementers of this approach is the famous image sharing website flickr.com . Flickr uses mysql to store image pointers, while the images itself are stored on disk and can be accessed through the lighthttpd server. While flickr enforces several rules to filter the pictures returned in its search results using geographic and login usernames/password any image can be viewed without logging in using its URL regardless of the fact that it needs to be filtered.

Storing images inside the sql database server:

That approach solves the previous problems. Since the image is stored inside the database:

  1. Authentication and access rights can be verified before the image is returned.
  2. On the fly thumbnails can be created thus more space saved.
  3. Referential integrity can be maintained.
  4. Cascaded deletions can be implemented. So when a user deletes an album, all the images inside the album will also be deleted.
  5. The whole application will be less complex.

Database engines use several types to store binary data. Postgresql uses byta, mysql uses blob types. Both achieve the same objective of storing binary object. This approach has its own problems as well. The database will now occupy a larger portion of the hard drive, larger than the size of the binary images itself. Retrieving the images will be slightly slower. My experiments in my other website (mibrahim.net) showed that retrieving images from postgresql was actually not bad, and for users accessing the website they won’t even feel the difference. One major disadvantage is database backup/restore. Since the size of the database is larger, dumping the database for backup can sometimes take days. If you are using WALs then most likely your WALs will be bloated and big from binary inserts. Bottom line, you’ll gain a lot but you’ll have to pay more money and buy larger drives to store the larger data.

There is a nice article written by Oracle having more details about this issue.

So in terms of clker.com, this is a planned functionality - to move all images and pictures inside the postgresql database. I believe that my server has the cpu and HD cycles to carry this load, and I know from my previous experience it will definitely help in decreasing the complications I have now.

Technorati Tags: , , , , , , ,

]]> http://www.clker.com/blog/2008/03/03/storing-images-in-your-sql-database-versus-filesystem/feed/ Running your server is easy, fun but involved http://www.clker.com/blog/2008/03/01/running-your-server-is-easy-fun-but-involved/ http://www.clker.com/blog/2008/03/01/running-your-server-is-easy-fun-but-involved/#comments Sun, 02 Mar 2008 01:09:09 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/03/01/running-your-server-is-easy-fun-but-involved/ I love using Linux to do my work. My best usage of Linux is my web server, although I recall I read once that Linus never intended for the kernel to be used as a server. He was more focused on using the kernel in desktops. I’ve been running my own web server for almost a year now, which runs two websites mibrahim.net my real estate website, and clker.com a to be online clipart website - we’re halfway there.

The fun part is simply everything just works. You’ll have all the tools you need starting from database engines like postgres, mysql to scripting languages like php, ruby with different types of webservers apache, lighthttpd and others. All the tools you might think of are there and under your own hand. Building your own server is not expensive - around $100 will do it. You don’t need a super quad core machine to produce extremely fast websites, unless you are already getting more than 50 page requests per second and at that point you will need something faster.

The performance bottle neck is never the CPU, it’s the hard drives read or write speeds. You can improve on that using fake RAIDs. Almost all Linux distros offer fake RAIDs and that is the cheapest way to improve the read speed.

Setting up your server is not a hard process. The best distributions that I recommend are Debian and Ubuntu. The reason is the very large library of software that comes with each. I believe that now the full distribution has grown more than 11 CDs. I used to run Debian and switched to Ubuntu a year ago and the reason behind the switch is the faster updates I get from Ubuntu, which enables me to use more recent and updated versions of PHP and the database engines.

The easiest setup is using the Ubuntu server CD, which is not any different from the desktop CD in terms of binaries. The only difference is that it won’t install the X11-server (GUI) and the window managers (gnome or kde) and the install program itself runs over the console and not VGA graphics. I use the server installation, and connect to my server using ssh. I have another old machine that runs Ubutu as well, and is used to run freenx. By that way I keep the server’s memory for the services running, and I can add all the GUI programs I want on this old machine.

Since I greatly benefited from running my own web server, I will share my experiences every now and then when I’ve got time to write.

Technorati Tags: , , , , , , ,

]]> http://www.clker.com/blog/2008/03/01/running-your-server-is-easy-fun-but-involved/feed/ Hardware problems - the mystery solved http://www.clker.com/blog/2007/12/11/hardware-problems-the-mystery-solved/ http://www.clker.com/blog/2007/12/11/hardware-problems-the-mystery-solved/#comments Tue, 11 Dec 2007 18:34:54 +0000 Mohamed Ibrahim http://www.clker.com/blog/2007/12/11/hardware-problems-the-mystery-solved/ I previously wrote about hardware problems that were puzzling me, and I finally found the answer. I installed sensors on ubuntu and ran ksensors and just watched what’s going on. The old server that was shutting down suddenly had an opteron dual core processor, which ran at 50 degrees Celsius at no load. I had three websites running there, the most recent addition is clker.com . Due to the large number of people that were trying to access the website, the CPU was loaded and overheated, which inturn lead to it shutting down.

Of course the solution is to put the server in a very cold room, which I don’t have. I resorted to buying a new board and CPU, which does not heat as much as the old ones and transfered the database and pictures to it. Currently the no load temperature is 11 degrees, which I think is great. This new board is a special sale from microcenter, both the board and CPU were for $94 with $15 rebate Sempron 64! I added 1GB of RAM. It really worked much better than my expectations, as I had very very low performance expectations from that CPU

Currently this server is running clker.com only, and will run mibrahim.net today. The CPU load as shown in top does not pass 2% except in spikes. People are still digging as well as search engine robots. So that’s a good value if someone wants to run his/her own starter website.

Technorati Tags: , , , , , , , , , ,

]]> http://www.clker.com/blog/2007/12/11/hardware-problems-the-mystery-solved/feed/ Hardware problems http://www.clker.com/blog/2007/12/08/hardware-problems/ http://www.clker.com/blog/2007/12/08/hardware-problems/#comments Sat, 08 Dec 2007 20:37:37 +0000 Mohamed Ibrahim http://www.clker.com/blog/2007/12/08/hardware-problems/ The web server has been shutting down by itself for the past week. I tried looking at the logs to figure out why I come to my office and I find it turned off. It looks like there’s a hardware problem causing the server to shutdown suddenly.

Luckily, I didn’t get any database corruptions - not until now. I changed the power supply and testing to see the results. In my experience almost 90% of the time it is a power supply problem especially that I’m loading the server with an extra useless PATA hard drive just to make GRUB happy.

Technorati Tags: , , , , ,

]]> http://www.clker.com/blog/2007/12/08/hardware-problems/feed/ v0.1 Beta release of openoffice.org addon http://www.clker.com/blog/2007/11/30/beta-release-of-openofficeorg-addon/ http://www.clker.com/blog/2007/11/30/beta-release-of-openofficeorg-addon/#comments Fri, 30 Nov 2007 09:35:43 +0000 Mohamed Ibrahim http://www.clker.com/blog/2007/11/30/beta-release-of-openofficeorg-addon/ Finally, there’s a light at the end of the tunnel. It’s 4:21 AM and I’m very happy to reach this stage with the openoffice addon.

You can download it Openoffice.org addon for clker.com integration . Installation is very easy, from the tools menu in openoffice, select Extension Manager. Click add, and point to the file. You will need to restart openoffice for it to work.

openoffice.org addon screenshot

This extension will show you the contents of your basket stored on clker, and will allow you to place any clipart image in the clipboard, then you can paste it in the correct place in your document.

Still there are some minor issues:

  1. The addon parent is currently null, so an extra window will appear in the task bar
  2. The way we place the picture in the clipboard is by loading it in draw, select all and copy then closing draw. There are better ways for doing this. You will notice that the screen will flicker especially if your using the ODG or the SVG formats since draw takes some time to render the vectors.

In case you wanted to play with the source, the addon source is here.

Technorati Tags: , , , , , ,

]]> http://www.clker.com/blog/2007/11/30/beta-release-of-openofficeorg-addon/feed/