You are here

How popular is this blog part II

Primary tabs

we started this blog on 20th of march 2004, since then we moved it from server to server 7 times, the resources required for the aggregator and hosting platform are not trivial and maintaining the whole thing has been a tough, time consuming and expensive job.

now we are running on a dedicated server (an old one true, but still we can't go higher than that), which means we are staying here for a year at least.

and finally after a whole year of tweaking I think I'm approaching a stable setup, the new aggregator code performs much better and things are stable now.

in the past few months the load was so high we could never serve everyone who wanted to visit the website, it was always down or too slow on rush hours, now we sometimes see 600 visitors on a single hour.

what this means we won't be able to measure the true popularity of this website until the end of december when a month has passed with the new setup.

but it is still interesting to look at server logs from time to time, specially after we won that BOB.

last time I analyzed server logs was on June 2005, here is a quick analysis of October 2005 logs.

in the last move I misconfigured apache, the logs do not contain referral or user agent info which means I can't filter out bots and spam accurately this time, so I'm just extrapolating here, as with any log analysis take this with a grain of slat.

there are some surprises when one compares June and October.

Quick Analysis of October 2005 traffic


Successful GET hits
Successful POST hits
Other successful hits

we got 1.5 times the number of hits we got on June not bad.


Administration overhead hits
we got 4359 on robots.txt, double the amount in June, extrapolating this probably means something close to 30000 hits by BOTS but could be anything
Visits to Atom2RSS gateway by aggregator
Site Logo

as usual I'm counting overlaps here so overhead is not really that big.

interestingly bots, css and logo hits almost doubled since June, but favicon is still the same, what does it mean? some of the extra traffic is by frequent visitors visiting more frequently maybe?

Atom2RSS gateway overhead doubled because we added more blogs, only normal, this is the last time we'll see this overhead, new aggregator code can parse atom without the need for a gateway.

The Aggregator

the aggregator had 23354 visits last month, thats actually less than in June, it either means the aggregator is loosing popularity, or people have no time to browse it's pages and just use the sidebars, or this is due to the downtime caused by the aggregator. (I did take it down several times on september and october after all, we'll find out for sure next month).

Main Page

front page hits
site rss feed hits
events calendar
image galleries
bloggers forum
hotspots reviews
new posts and comments trakcer

note that I only calculated front page and rss feeds visits in June, front page hits almost doubled. RSS feed is much more popular now.

the forums, and event calendars are new features, they seem to be popular enough I just wish people would contribute to them more.

not when I say forum, event calendar, hotspot reviews and image galleries I mean people who browse these features, not just land on page related to them. ya3ni many more people look at photos through links in articles than through browsing galleries.



I no longer serve all multimedia files from the files/ directory, I run a separate small footprint http server for large files in order to conserve RAM so this does not show videos and audio files.

no increase here though, our multimedia files have fixed popularity (note though that we aren't coming up with as many interesting photos as we used to).



I was too lazy to figure out what would have been the key political articles in October and check their trends, I just looked at the famous never forget document, old material doesn't live long on the blogosphere and thats why we need to thing of hybrid formats, the document is obviously not as popular as it was, but then again maybe everyone already got a copy, or maybe it is hosted in hundreds of different places, who knows.


we got 58096 visits on full article pages last month, note this means people who probably read the whole article and it's comments, sme of them came through the rss feed, the front page, the aggregator, google, or a link referring to one of the blog posts.

again almost double the rate in June.

if I can make a conclusion it would be

this blog's popularity increased but the aggregator did no

I'm not sure this is accurate, remember the aggregator was not performing well and I can't be sure about BOT activity.

but on the other hand it could be true, drupal is reporting more page hits and we got more comments than we used to get.


I never used aggregator I use sidebar all the time.

I guess this means you can begin thinking of yourselves as bloggers, more than bucketers.

Congratulations on the Deutsche Welle Award - and thank you for your great blog.

Best wishes

what is the aggregator?

but I'm just too lazzy

the Aggregator refers to the Egyptian Blogs Aggregator (which is what populates the blocks on the right sidebar and what you get when you click in the link that says Egyptian Blogs in the top navigation menu).

just in case you don't know.

An aggregator is a piece of software that checks RSS and ATOM feeds regularly to see if any blogs (out of a given set) recieved updates since the last program run. then it shows an excerpt of the new posts with links to the original articles, aggregators are a great way of keeping up with many blogs.

The Egyptian blogs aggregator has a list of all known Egyptian blogs, every hour it automatically grabs new posts and publishes exceprts and links here in

the aggregator attracts thousands of readers and accounts for a large portion of the traffic in

so is it a bot or an independent web service component?

huh? what on earth does an independant web service component mean?

what are you, a java system analyst?

it's a bloody RSS and Atom aggregator, search freshmeat there are thousands of them, this particular aggregator happens to come in the form of a drupal module.

the aggregator loops over a set of urls stored in a database, grabs the xml feeds via http, parses the xml and injects data from new posts in the database, a cron job ensures the whole process ggets performed every hour.

thats all there is to it, a rather simple piece of software if you ask me.