Recursively upgrade package dependencies with pkgng

I’ve been hosting my own pkgng repository for a few months and loving it. One thing I’ve had problems with is upgrading package dependencies. For example, I would find myself upgrading Salt without upgrade ZMQ which led to lots of issues.

To upgrade a package and all its dependencies, run the following command

What this does is get a list of package dependencies from pkgng, run them through awk to remove version numbers, then xargs runs the pkg upgrade command on all packages

Simulating cron runs on the command line

Does this sound familiar?

You write a new script, start it from the command line, everything works properly, you add that script to your crontab then go home, the next day, nothing. So you end up setting up a schedule to run this script the next minute, the script fails, you change something, update the schedule, rinse and repeat. Read more Simulating cron runs on the command line

ADSL in Jordan

I’ve heard various complaints from a few friends of mine about the status of Internet connectivity in Jordan, especially ADSL. So I’m writing this article to explain what is going on and why ADSL the way it is, but first, some history.

Back in pre-broadband times, people used dial-up modems to connect to the Internet. The name “modem” came from the fact that data was being modulated (Turned from digital to analog form) to be sent over the phone lines, and demodulated (Turned from analog to digital form) at the other end. This utilized the same frequencies used for regular phone conversations, this is why you could hear the weird sounds if you picked the phone up while being connected to the Internet. Given the relatively narrow frequency range used by phones this technology was limited to about 56Kbps, the best speed I got was back in 2003 which was about 5KB/s for my downloads, and that was considered fast at the time.

Then someone figured out they could fit a much bigger frequency range in existing phone lines. The ingenious thing about this idea was the ability to leverage existing infrastructure and allow phone companies and ISPs (Internet Service Providers) to sell much faster speeds with insignificant investments compared to having to switch the entire infrastructure to fiber optics, also allowing people to use the phone even while being on line.

ADSL stands for “Asymmetric Digital Subscriber Line”, which means the technology allowing digital transmission of data over the existing wires of the local telephone network. The “Asymmetric” part means the upload and the download streams are not of the same speed. Home users almost always care more about download speeds, so it only made sense that the technology utilized more frequencies for download. This meant we could now get download speeds of up to 2Mbps, depending on how far you are away from the exchange.

Now for some technical details. Back in the 70s landlines were the only means of telecommunication. Governments laid down huge telephone networks. This is how a telephone network typically looks like. Please note that this is a very simplified diagram.

ADSL is a “Last Mile” solution, meaning it only cares about connecting subscribers to the exchange, and not how the whole network operates, so this diagram is enough for our needs. The interconnections between the exchanges have no effect on ADSL so we’re leaving them out.

As you can see in the diagram, telephone companies run cable bundles from each exchange to street cabinets, then from those cabinets to each house or office. This method is used to organize how the cables are laid down. Each cable bundle can reach up to 5Km in length. Cable bundles (1,200 cable pairs) and cabinets look something like this



As with every electrical signal traveling through a conductor, ADSL signals suffer from 2 electrical phenomena; Attenuation and crosstalk.

Attenuation is the gradual loss of signal intensity as it passes through a medium. Of course, the longer the distance the greater the attenuation. The greater the attenuation, the slower the final connection speed can be. Here is a graph detailing the relationship between cable length, attenuation, and final connection speed. Note that Orange uses both ADSL (Up to 8Mbps) and ADSL2+

Crosstalk is the phenomenon by which a signal traveling in one circuit creates an undesired effect on another circuit. It is usually caused by undesired capacitive, inductive, or conductive coupling of one circuit to another. What this means is when electrical signals pass through copper cables, they create interference for any adjacent cables. As you can see in the following diagram, all the cables around the victim cable are generating crosstalk which is affecting signal quality. Consequently, the more cables the more powerful crosstalk is.

Crosstalk generates noise, and attenuation lowers the original signal strength and makes it closer to that noise. The higher the speed, the more destructive the noise becomes. If a line is too noisy, the ADSL connection will become too slow or even drop.

To solve this issue, modems perform a sync operation with the exchange, they try to connect to the exchange at the maximum speed possible, then lower that speed to achieve an acceptable noise level, they also leave a noise margin to allow for fluctuations in noise throughout the day. This explains why most homes in Jordan cannot get anything faster than 2Mbps, with some of them getting as low as 128Kbps. This also explains why speeds go down during un-metered (A.K.A. free) download hours in some areas.

Times change, technology matures, content become more rich, and more importantly, competitors arrive. With the clear change of the type of content showing on the Internet and with the rise of competing technologies, i.e. WiMAX, Orange had to start upgrading their network. People were no longer content with the speeds provided by the ancient network built by the government during the 80s, and some of them were bound to leave ADSL and switch to WiMAX, especially since WiMAX didn’t have the phone line overhead and users were no longer confined to the locations they bought the subscriptions for.

Given the number and distribution of users in Jordan, delivering fiber optic connections to each house was out of the question. Contractors charge 22-30JDs per meter for excavation, so connecting a house 300m away would cost 6000JDs in digging costs alone, which is highly unfeasible for a 50JD per month return. In countries like South Korea, population is so dense, each tower has at least 100 users, if you consider a 6000JD cost of laying down the cables compared to a 2000$ monthly return on investment (Assuming 20$ per month for each connection), it’s a very easy decision.

Orange chose a cheaper approach and went with Fiber To The Cabinet (FTTC). FTTC allows Orange to connect a single fiber cable to a digital cabinet close to the user premises, usually next to one of the older copper cabinets. This cabinet acts like a small exchange and contains the equipment to which modems connect. They then connect short cables from the new cabinet to the old one, which is connected to the cable reaching the customer premises. So now, the telephone network looks something like this

I showed a single FTTC cabinet in the figure because they are still not as common as their copper counterparts. FTTC cabinets look like this

How is this useful you might ask. Well, fiber optic cables have very little crosstalk, if any, and attenuation is an order of magnitude smaller than copper wires, especially for long distance. This effectively lowers the distance between the exchange and the modem from 5-6Km to only a few hundred meters, 1Km the most. It also reduces crosstalk between copper wires because those branch out from the cabinet and are not bundled together.

This allowed Orange to reach their current speeds of 24Mbps for some areas. Please note that when I say 24Mbps, I mean for those who live within 100m of a fiber cabinet, any further and the speed will drop. I currently live 500m away from the FTTC cabinet and my line speed is 21Mbps, further distances will get even less speeds.

I’m disappointed with the fact that these FTTC cabinets are not more common and densely distributed. Most areas in Jordan are still covered with copper cabinets, giving most users a best case speed of 4Mbps. Given the amount of money Orange makes, they should be upgrading their network much more rapidly. This is one of the reasons why a lot of areas are leaving ADSL in favor to wireless broadband, especially after the launch of Zain’s HSPA+ service.

I hope this erases some of the misconceptions about ADSL, and I do hope Orange executives become more diligent about their upgrades and have a maximum distance of 500m from any FTTC cabinet.

First Thoughts of @ZainJo's HSPA+ Service

Lately ISPs have been bombarding us with all sorts of Internet related ads, promising “high speeds” or “unlimited downloads”. Osama Hajjaj said it best with this comic:

I want to get a few things straight before we move on

  • Since what you’re paying for is essentially shared bandwidth, it can never be “unlimited”. ISPs need to make money as well, so they can’t let you hog their bandwidth, but in the case of ADSL, unlimited downloads during the night is possible because most users are asleep, and those few heavy downloaders won’t affect the mostly un-utilized ISP uplink
  • For the same reason as above, what ISPs are advertising is a best case scenario, so there are times when you will be facing slowdowns. Reasons for slowdowns differ for each technology used. With wired being superior to wireless.
  • A respectable ISP will minimize those slowdowns as much as possible. Zain failed in this respect with WiMAX, let’s hope the same doesn’t happen to their HSPA+ service
  • ISPs try to sell a lot of corporate bandwidth, which is very profitable, then try to use that bandwidth during the night for residential users to minimize costs.

A question raises itself, do we really need all this speed? The FCC defines “broadband” as having 4Mbps or more, looking at the Jordanian market, most people are below 1Mbps, so maybe 80% do not have broadband. Another question raises itself, does it really matter? No, for most people anyways.

Let’s look at how Jordanians use the Internet, most people only use it for browsing, email, the occasional photo sharing, and in rare cases, video streaming.

Even for the pickiest of users, the difference in speed is only noticeable up to the 2Mbps mark, the above uses will not change the experience for anybody. Most websites are not properly built so they’re slow, and video streaming (for standard definition) will buffer faster than playback speed, so it won’t be of any benefit to the user. Even for those who use the lower speeds, they don’t really care.

Why then, you ask, do we need “proper” high speed Internet? The answer is simple, downloads. If you’ve ever waited on a file you need for work to finish downloading, you’ll realize why high speed is nice to have. I personally download a lot of ISO images to test numerous system configurations, I also download the actual programs to perform the required testing. Having to wait 15 minutes for a 70MB file is never good. I had to switch from FreeBSD to Ubuntu Linux for my prototyping needs because Ubuntu requires much less downloading. If I had 8Mbps (Been 3 weeks since I paid Orange for an 8Mbps account, which I’m still waiting for) instead of my current 1Mbps the download would take about 2 minutes, which is OK by all standards. If you download TV shows or movies, the problem becomes even worse; A 20 minute HD video is about 650MB, that’s a 3 hour download on a 1Mbps connection, with 8Mbps, that’s 20 minutes, a BIG difference.

Up until recently, there were no cheap Internet connectivity options. Or let me rephrase, no cheap “decent” connectivity options. 13JDs for a 128Kbps is not considered cheap, for an option to be considered cheap in Jordan, it’ll have to cost around 5-10JDs a month and it definitely has to be faster than 128Kbps to allow the occasional Youtube streaming without having to buffer for 15 minutes. For those who only used 2GB a month (yes, those people exist), they had to pay 30+JDs for 10-15GB which they never used.

After the introduction of WiMAX back in 2007, prices went down, but not much. Orange still had dominance over the market for 2 reasons; They were the only company that bought bandwidth from international ISPs, and their networks were far superior to WiMAX given the fact it’s wired, so the others were unable to match Orange on terms of speed. Not to mention that Orange’s network has been here for ages, which allowed them to upgrade it while still making money, which cannot be said about the others.

Now comes the primary reason for this post, Zain’s HSPA+. Now that Zain has come with a somewhat reasonable alternative, which had various options depending on your usage levels. Starting from 6JD for 1GB up to 49JDs for 30GB. Those should suffice the needs of about 97% of Jordanians.

I was given a 1 month subscription along with a 21Mbps dongle, the Huawei E367, which was surprisingly Linux compatible. Zain gets extra points for this particular selection.

The frequency used by HSPA+ is higher than GSM, which means GSM has better signal penetration. In my house, I can barely make voice calls. So when my phone barely picked up the signal I wasn’t surprised. I thought the dongle would have better reception than the phone, sadly that wasn’t the case. But if you look at any wireless technology, those don’t really work with the buildings we have in Jordan. If you have heat insulation you will most probably not get any good signal indoors.

When Zain first released the service I faced a lot of dropped calls and broken messages, but I haven’t faced them as much lately, I guess this comes with being an early adapter. I do however keep my phone on GSM most of the time to make the battery last longer, and to make use of GSM’s better coverage.

To do some testing, I took the laptop to the roof and tried to connect, sadly I only got 2 bars, very disappointing, but that gave me about 4Mbps download (no screenshot for this one folks, sorry).

Then I went to a coffee shop in Swiefeyyeh to try it out, 2 bars as well. I also assume this is because the network isn’t a 100% ready, so hopefully it’ll be better in a month or 2., but this time I gotGood, but still not as good as the advertised 21Mbps, I assume this is because Zain’s network is still new, and you’ll need to have a full signal to at least have a chance of getting the full speed. Downloading from a dedicated server gave me about 900-950KB/s, very good.

Then on the way home I bought a USB extension cable, upon arriving to my desk, I connected the dongle to my laptop using the cable, and thew the dongle out of the window, this time I got 2 bars, and the following speed:

This was OK. Download speed from the dedicated server got to about 300KB/s, good.

Now for the most important usage of this connection, browsing. To be honest, I was very disappointed at first, because page load times were not that of connections that get the speeds above. After a few minutes I changed the DNS settings on my connection to Google’s servers and browsing was much better afterwards.

I’ve noticed that Zains’ DNS servers have been slow ever since WiMAX, they use BIND which is one of the most popular DNS servers on the Internet, I assume they’re running it on some Linux distribution. I’m not a big fan of that particular setup. If someone at Zain reads this, please try using NSD for authoritative servers and Unbound for your caching servers, also try installing them on FreeBSD, which has a much better TCP/IP stack implementation.

Youtube was slower than I expected, but still fast enough, although I didn’t try watching HD video, so your mileage may vary. I also noticed the dongle becomes warm after some time, wasn’t very impressed with that.

OK, to the conclusion, is this worth buying? For me, it was a bit underwhelming, but for a lot of people, they’ll find it does the job. I would stay away from it if there a is commitment involved, but if you can cancel your contract if Zain starts messing up then by all means try it. No commitment means Zain has to keep the service working well to keep the customers.

For people like me who download a lot, I say stick with DSL, it’s still a better option. Given the fact it’s not very good indoors, at least in my house, I wouldn’t bother with it for anything other than a secondary connection when at clients, if it actually works.

My personal experience with Zain is that the service will be great at the beginning, but then the network will get congested and they won’t bother upgrading. Let’s hope the same doesn’t happen with HSPA+. Note however that their network is very young, and they have built it in record time. So facing these issues at this time is normal, let’s hope they make it better, not worse.

Oh, and if you get it, make sure you don’t deplete your download quota, and check it frequently. If you end up using a lot it will be VERY expensive.

Using the @ZainJo HSPA Dongle With Ubuntu Linux

After my experience with Zain’s WiMAX, I didn’t think I’d be using their service again. But Zain gave me a 21Mbps dongle (The Huawei E367) for free as to see how much better the new service is and I thought I might as well try it. I’ll be writing on my experience with the service in a separate post inshalla.

I’m a big Ubuntu fan and been using it as my primary desktop since 2007, and Linux compatibility has always been absent in the mobile Internet world. I wasn’t gonna switch my primary OS just to use a USB dongle.

After plugging it in nothing happened, didn’t see a notice on the screen, no pop-pups or anything to show that something happened. After a quick lsusb I found this:

Bus 002 Device 005: ID 12d1:14ac Huawei Technologies Co., Ltd.

Kudos Huawei on showing us Linux users some love. Zain gets extra points for this choice.

Ok let’s get this bad boy configured. I got the settings from Zain’s Windows application. The following was done on Ubuntu 10.10, other distributions might vary.

1) If the dongle was unplugged when you booted your computer, restart Network Manager (sudo service network-manager restart), wait a few seconds, then right click the Network Manager icon and choose “Edit Connections…”

2) Go to the “Mobile Broadband” tab and click “Add” (You will not see “Zain Jordan” on your screen)
3) The following screen will show up, if you didn’t restart Network Manager you will not see the Huawei modem, click “Forward”
4) Choose “United States” then click “Forward”, don’t bother looking for Jordan, you won’t find it
5) Choose “I can’t find my provider and I wish to enter it manually”, then enter any name you desire, I chose “Zain Jordan”
6) Choose “My plan is not listed…” and for “Selected plan APN (Access Point Name)” enter “zain”, all lower case letters, without the quotes, then click “Forward”
7) Click “Apply”
8 ) The following window appears, don’t change anything except “Username” and “Password”, set them to “Zain”, all lower case except for the Z, then click on “Apply”
9) You’re all set, to connect, left click on your Network Manager icon and choose your connection name
If you face any issues, just remove the dongle, reconnect, then restart Network Manager.

Tales of a disgruntled @ZainJo customer

I’ve been meaning to write this for a few weeks but was very busy, I finally got the time to do so.

If you know me, you’ll probably know I’ve been struggling with my Internet connectivity since 2005. I’ve been begging Orange (Jordan Telecom Group at the time) for anything faster than 512Kbps, but I was always told that my area was not covered. I live in an area filled with students who don’t have phone lines, and the very few people who use DSL here are content with the speeds Orange was offering. So Orange did not bother upgrading my area because it wasn’t worth it.

I’m not a big fan of wireless technology, so even when Umniah and WiTribe released WiMAX in 2007 I wasn’t keen on switching even though my DSL was disconnecting on weekly basis because of the horrible cables running from the exchange to my house.

I’ve had a Zain mobile line since 2004 and I absolutely love it, unless something major goes wrong, I won’t be switching any time soon, so when I heard they were launching a WiMAX service I decided to pay the premium and go with them (About 38JDs for the 2.4Mbps account per month), assuming their WiMAX service will at least be as good as their GSM service, and assuming they will give me the same level of support they gave me as a GSM customer.

So on April 6th 2009, I went and signed a contract, with a sales lady named “R”, who was very happy to answer my calls and help we with every question I had. But things changed when I had paid. After taking the CPE home I realized that the insulation in my house was too strong that WiMAX signals and the CPE was not working unless I had the “Abajoor” open. I tried calling R but she stopped answering her phone, so I tried calling support.

After some tuning I managed to get everything to work properly, the download speed was never slower than 280KB/s, it was like a dream come true. I finally had a “proper” Internet connection.

For those who don’t know, the WiMAX network Zain was using is owned by Mada, which is partly owned by Zain. Mada was supposed to concentrate on the network and corporate sales while Zain was supposed to handle retail. Mada’s relationship with Zain provided the former with a good amount of towers to install their base stations. The result was a truly stable network with excellent coverage. Even in terrible weather.

Zain’s policy of selling the CPEs kept the number of users minimal, if memory served me right, the USB dongle was about 100JDs and the CPE was about 150JDs, which not everybody could afford. Also, Zain made customers commit to yearly contracts, so you couldn’t skip a month or 2.

7 months later, things changed. Mada and Zain got into a disagreement and Mada opened its own retail shops. They offered the CPE for free if with yearly accounts and the USB dongle for free with quarterly accounts. They also had prepaid cards which allowed people to use the service whenever they wanted. That was very appealing to the students in my area. So suddenly the towers in my area became congested.

The download speed dropped from 280KB/s to 15KB/s, the equivalent of a 128Kbps connection, which cost 13JDs a month at the time. I tried calling support to get a solution, they always said that my account was not 2.4Mbps, it was “up to” 2.4Mbps.

As someone who has worked in 2 ISPs I know the reason for this “up to” thing; Since home users can’t pay the actual price for bandwidth (About 200JDs for 1Mbps), this bandwidth has to be shared between multiple people, this means speed is not guaranteed and during peek hours you will not get the full speed advertised. It is only a best case scenario.

2.4Mbps amounts to about 302KB/s, and accounting for protocol overhead you will lose anything between 13-16%, so you get 260-270KB/s as a maximum. It is totally acceptable to get speeds of 170 to 200 during peek hours.

But getting 15KB/s ALL the time was ridiculous. What made matters worse is the fact pinging the WiMAX gateway which is inside Mada’s network took anywhere up to 6 seconds. To put matters into perspective, the normal ping time is about 60 milliseconds which is 6 seconds divided by 100. This meant that even browsing was way too slow, as every request sent had to wait 6 seconds.

The solution was simple, Zain and Mada could have installed new towers or upgraded the existing ones. But nooooooooooooo, that would have cost them money. Whenever a customer care employee didn’t answer with the “up to” response I got “we’re working on it”, or, “It’s your PC”, or “It’s the CPE”. Testing the CPE at 4AM (The time with least Internet traffic) gave me consistent 270KB/s, so it was definitely neither my CPE nor my PC. But it was (According to Zain’s support) never Zain’s problem, another common response was “Nobody else complained, it’s just you”. I knew a few people in my area who complained, but it seems they don’t exist in Zain’s fantasy world.

My brother and I are very heavy users, the 25GB quota I had on the WiMAX connection wasn’t enough, so we also have a 1Mbps DSL line, which was working at about 600Kbps, a little better than getting a 512Kbps package.

My cousin, who is my next door neighbor, got fed up with Orange and threatened to complain to the TRC, Orange came the next day and connected him to a digital cabinet 200m away from our house (Orange didn’t agree to connect me to that particular cabinet saying it doesn’t cover my street) and now he was able to get 8Mbps.

This made me furious, at first Orange ignored my requests to be hooked to that cabinet, and I was stuck with this horrible WiMAX service that Zain didn’t bother to fix (Why would they? I’m but a mere customer, the least important aspect in any business). I decided to call the TRC, 3 days later Orange connected me to the same cabinet and the 1Mbps became much faster than before.

Mysteriously Mada somehow fixed something so I was then getting better speeds, but by no means consistent, speed was anything between 70 to 150KB/s. And even on a good day, browsing on the 1Mbps DSL was faster for light websites.

Seeing that I finally had a proper DSL line, it was time to disconnect the horrible WiMAX that hasn’t been working properly (Except during university vacations). But of course Zain had me on a contract, and said I can’t terminate without paying a penalty, which is the subscription till the end of my contract. So I decided to wait a little on disconnecting the WiMAX connection since I’ll be paying for it anyways, so I might as well use it.

In February things went down the toilet, download speeds were as low as 15KB/s again and I had my CPE off for about 3 weeks as the connection was utterly useless. When I went to disconnect the line Zain had the audacity to say “Even though the speed is horrible, that doesn’t mean you can break the contract”. What the Hell? What kind of respectable company says that to a customer?

After I voiced my intentions of not renewing my contract with Zain, I got 4 calls from them in 3 days. I wonder where they were when I was having problems :/

Both people who talked to me tried to convince me to switch to HSPA+, saying they can offer “up to” 21Mbps. Of course my response was “You couldn’t deliver 2Mbps, how can you expect me to believe you can deliver 21? I do not trust Zain as a data provider, and whenever someone asks me if they should subscribe with your service, I advise against doing so, saying you will twist their arms with the commitment period, and then you won’t deliver what you promised. Don’t bother calling me again, I’ve already given the CPE to a friend of mine who is willing to use it till the commitment is over, and I signed a contract for an 8Mbps account in addition to my current 1Mbps line”

I’m not the only one complaining, just take a look at Zain’s Facebook page and you’ll see a lot of people complaining about the horrible service.

Orange isn’t much better, but at least with Orange, if you’re connected to their fiber network, it works, and it’s consistent.

Zain does indeed have the potential to do something really good. I only wrote this post to grab their attention and at least get enough support to force them to stand behind their service and make it a proper alternative to Orange, not just give a half assed service as they did with WiMAX.

To all my readers, I'm sorry

Hello everybody,

I know it’s been a while since I last wrote anything. I started a new job and it’s been crazy busy the past few months. And for some reason the blog stopped sending notification emails so I thought nobody read anything here.

I’ve upgraded the blog to the latest WordPress release, and hopefully I’ll be writing more stuff here.

Good night