Clker.com - weblog » php http://www.clker.com/blog Just another WordPress weblog Sat, 21 Nov 2009 03:49:42 +0000 http://wordpress.org/?v=2.8.6 en hourly 1 Caching SQL results with PHP http://www.clker.com/blog/2008/05/12/caching-sql-results-with-php/ http://www.clker.com/blog/2008/05/12/caching-sql-results-with-php/#comments Mon, 12 May 2008 15:54:36 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/05/12/caching-sql-results-with-php/ I’ve been looking around lately on the best way to cache SQL results in PHP. I found some interesting articles posted in lots of places, but I didn’t find any that exactly matches my needs. The problem I have on hand is basically the same every growing website faces: decreasing mean resource usage per page request.

Now – this is my plan A to keep up with the website’s growth without a lot of hardware upgrades. There is a plan B, but I will keep that to a later post.

Usually, the time consumed by executing the SQL statements is more than 80% of the whole script time. Reducing this time will significantly reduce the resource usage. CPU is not the bottle neck, usually SQL queries result in a significant disk access especially if there’s a lot of inserts/updates/deletes.

The easiest and best way I found is to use PHP’s serialize to store the SQL results. This requires that we fetch all the rows we’re dealing with. So the sequence of operations is to check the cache file, if expired or does not exist then fetch all the rows from the result set, serialize in one variable, write it to the cache file.

  1. <?
  2. require_once "utils.php";
  3.  
  4. $cache_prefix=‘/tmp/clkercache/’;
  5.  
  6. function cache_fetch($filename, $expiration=3600)
  7. {
  8.   global $cache_prefix;
  9.   $filename=$cache_prefix.$filename;
  10.  
  11.   if (!file_exists($filename)) return false;
  12.   if (time(0)-filemtime($filename)>$expiration) return false;
  13.  
  14.   $data=unserialize(base64_decode(file_get_contents($filename)));
  15.  
  16.   return $data;
  17. }
  18.  
  19. function cache_store($filename,$data)
  20. {
  21.   global $cache_prefix;
  22.   $filename=$cache_prefix.$filename;
  23.  
  24.   @mk_dir($filename);
  25.   return file_put_contents($filename,base64_encode(serialize($data)));
  26. }
  27. ?>
  28.  

So the way to use the caching functions would be something similar to this:

  1. <?
  2. if ( !($pictures=cache_fetch("mypictures.cache")) ){
  3.   $res=query("select pictureid from pictures order by rating15days desc limit 100");
  4.   while ($row=fetch_array($res))
  5.     $pictures[]=$row;
  6.  
  7.   cache_store("mypictures.cache",$pictures);
  8. }
  9. ?>

So how to do that if you have multiple queries? The way I store my cache files is by giving them suffix or prefix names depending on what is being stores. For example a file that store the pictures that are related to a picture whose id is 1500 being displayed, will be called related-1500.cache. For more complicated cases like searches, I would use the md5sum of the request URI.

The results were good. Using apache bench I tested the homepage. Before any caching it used to support 3 pages per second. After caching the result was 35. I think in my current situation with my current traffic, this is a very good result.

One might think so what does it take to generate pages at a much higher rate – say 300, 400 or even a 1000 or more? Well, that’s my plan B, and I’ll discuss that in more detail in another post. This plan B will generate that much pages per second.

Technorati Tags: , , , , , ,

]]> http://www.clker.com/blog/2008/05/12/caching-sql-results-with-php/feed/ 1 Creating a tar gz on the fly using PHP http://www.clker.com/blog/2008/03/27/creating-a-tar-gz-on-the-fly-using-php/ http://www.clker.com/blog/2008/03/27/creating-a-tar-gz-on-the-fly-using-php/#comments Thu, 27 Mar 2008 17:41:53 +0000 Mohamed Ibrahim http://www.clker.com/blog/2008/03/27/creating-a-tar-gz-on-the-fly-using-php/ A while ago, I thought about creating a tar.gz file for every download, so that if someone runs a search, he/she then can download all the images in the results. After a little bit of research, I found that PHP has a function for gzip. I also knew that the tar format just sticks files after one another, so if I can implement the tar format in PHP then I can gzip all images in the results.

I found this LGPL code that implemented the tar format. I used it (and modified it a little bit) to produce the online tar.gz functions:

  1. // Computes the unsigned Checksum of a file’s header
  2. // to try to ensure valid file
  3. // PRIVATE ACCESS FUNCTION
  4. function __computeUnsignedChecksum($bytestring)
  5. {
  6.   for($i=0; $i<512; $i++)
  7.     $unsigned_chksum += ord($bytestring[$i]);
  8.   for($i=0; $i<8; $i++)
  9.     $unsigned_chksum -= ord($bytestring[148 + $i]);
  10.   $unsigned_chksum += ord(" ") * 8;
  11.  
  12.   return $unsigned_chksum;
  13. }
  14.  
  15. // Generates a TAR file from the processed data
  16. // PRIVATE ACCESS FUNCTION
  17. function tarSection($Name, $Data, $information=NULL)
  18. {
  19.   // Generate the TAR header for this file
  20.  
  21.   $header .= str_pad($Name,100,chr(0));
  22.   $header .= str_pad("777",7,"0",STR_PAD_LEFT) . chr(0);
  23.   $header .= str_pad(decoct($information["user_id"]),7,"0",STR_PAD_LEFT) . chr(0);
  24.   $header .= str_pad(decoct($information["group_id"]),7,"0",STR_PAD_LEFT) . chr(0);
  25.   $header .= str_pad(decoct(strlen($Data)),11,"0",STR_PAD_LEFT) . chr(0);
  26.   $header .= str_pad(decoct(time(0)),11,"0",STR_PAD_LEFT) . chr(0);
  27.   $header .= str_repeat(" ",8);
  28.   $header .= "0";
  29.   $header .= str_repeat(chr(0),100);
  30.   $header .= str_pad("ustar",6,chr(32));
  31.   $header .= chr(32) . chr(0);
  32.   $header .= str_pad($information["user_name"],32,chr(0));
  33.   $header .= str_pad($information["group_name"],32,chr(0));
  34.   $header .= str_repeat(chr(0),8);
  35.   $header .= str_repeat(chr(0),8);
  36.   $header .= str_repeat(chr(0),155);
  37.   $header .= str_repeat(chr(0),12);
  38.  
  39.   // Compute header checksum
  40.   $checksum = str_pad(decoct(__computeUnsignedChecksum($header)),6,"0",STR_PAD_LEFT);
  41.   for($i=0; $i<6; $i++) {
  42.     $header[(148 + $i)] = substr($checksum,$i,1);
  43.   }
  44.   $header[154] = chr(0);
  45.   $header[155] = chr(32);
  46.  
  47.   // Pad file contents to byte count divisible by 512
  48.   $file_contents = str_pad($Data,(ceil(strlen($Data) / 512) * 512),chr(0));
  49.  
  50.   // Add new tar formatted data to tar file contents
  51.   $tar_file = $header . $file_contents;
  52.  
  53.   return $tar_file;
  54. }
  55.  
  56. function targz($Name, $Data)
  57. {
  58.   return gzencode(tarSection($Name,$Data),9);
  59. }
  60.  

To use those functions all you have to do is send a header with the mime type for the tar gz ( application/x-gzip ) using the php header function. To add a tar/gz section for a file, read the file in an array using filegetcontents and pass the filename and data to the targz function. Echo what is returned. That’s it!

So why is it not active on clker.com website? I actually tried it and found that compression consumes a lot of CPU. In the first 20 minute I had more than one hundred connections for different users downloading their results and the CPU was saturated. This basically left no CPU for searching. So use it carefully, and only if you really need that functionality.

Technorati Tags: , , , ,

]]> http://www.clker.com/blog/2008/03/27/creating-a-tar-gz-on-the-fly-using-php/feed/ 0 Hardware problems – the mystery solved http://www.clker.com/blog/2007/12/11/hardware-problems-the-mystery-solved/ http://www.clker.com/blog/2007/12/11/hardware-problems-the-mystery-solved/#comments Tue, 11 Dec 2007 18:34:54 +0000 Mohamed Ibrahim http://www.clker.com/blog/2007/12/11/hardware-problems-the-mystery-solved/ I previously wrote about hardware problems that were puzzling me, and I finally found the answer. I installed sensors on ubuntu and ran ksensors and just watched what’s going on. The old server that was shutting down suddenly had an opteron dual core processor, which ran at 50 degrees Celsius at no load. I had three websites running there, the most recent addition is clker.com . Due to the large number of people that were trying to access the website, the CPU was loaded and overheated, which inturn lead to it shutting down.

Of course the solution is to put the server in a very cold room, which I don’t have. I resorted to buying a new board and CPU, which does not heat as much as the old ones and transfered the database and pictures to it. Currently the no load temperature is 11 degrees, which I think is great. This new board is a special sale from microcenter, both the board and CPU were for $94 with $15 rebate Sempron 64! I added 1GB of RAM. It really worked much better than my expectations, as I had very very low performance expectations from that CPU

Currently this server is running clker.com only, and will run mibrahim.net today. The CPU load as shown in top does not pass 2% except in spikes. People are still digging as well as search engine robots. So that’s a good value if someone wants to run his/her own starter website.

Technorati Tags: , , , , , , , , , ,

]]> http://www.clker.com/blog/2007/12/11/hardware-problems-the-mystery-solved/feed/ 1