FON fixes maps vulnerability, and why Martin should apologize

You probably remember the post I made regarding FON’s figures, and how much I thought they differed from reality. It got quite a lot of attention, particularly from detractors, and from Martin Varsavsky himself. Many comments were posted on my blog and some others, which pointed towards the fact that I am involved in a startup which supposedly is a clone of FON, and thus I was biased and in no position to comment on FON. To cut a long story short, Martin posted a rather vicious personal attack on his blog, which I answered, he counter-commented, to which I again answered, but he never conceded a bit.Fon Maps

During my investigations that led to the statistics post, I also discovered a serious flaw in the maps management system, which would allow anyone to re-position any FON hotspot and change its address without first logging into the user area.

All that was required was the node’s ID and the hotspot owner’s user ID, both easily obtainable from the public queries that maps.fon.com launches against the database where hotspot data is held, and which I used to gather the statistics. For a determined attacker, it would have been very easy to place every single FON hotspot right in the middle of 1600 Pennsylvania Avenue, Washington DC.

I could have very easily posted about this, but I refrained from doing so for a reason – while I do not work full-time in the IT security industry, I have done quite a bit of consultancy work in the past, related to IT security, particularly in the wireless field. This means that I am fully aware of the industry-approved vulnerability disclosure procedure, which can be explained simply as:

  • Document the vulnerability, and inform the company about the fact that you have found it.
  • Wait for an initial response, establish contact points, and work a schedule for fixing the issue.
  • Work with the company to help them solve the issue.
  • Once the issue has been fixed, make a public disclosure on both sides about the vulnerability, giving credit to the person or company that discovered it.

You can find more references to this policy at Microsoft’s Security Response Center, here and here. A PDF from oisafety.org also describes this process in detail. A perfect example on how not to do things is the recent disclosure of a code injection vulnerability, which allowed manipulation of FON’s routers without even having to open them – even though their points are valid, they should have given FON the chance to fix the problem before going public.

In this case, I contacted FON’s support email first September 27th, and received a response on the 29th. This was really generic, only wanting to know about the details, and not acknowledging the normal procedure as I have explained above. On October 2nd, I emailed them again, asking to confirm that they understood the procedure, and on the 3rd they replied that they agreed on following the procedure.

I started compiling the information I had into a working document, but after becoming so frustrated at the attacks received as a result on my post about the statistics, the decision was to simply let the issue go, forget about FON, and concentrate on my own project. A couple of days ago, browsing around for stuff to clean up on the laptop, I came across the half-written report, and decided to finish it and send it to FON support, with CC to Martin, just to close the case. I received a reply today that they have in fact fixed the vulnerability, with a short ‘thanks’ (actually, quoting his email in full: “thanks Mike, i understand its been fixed”) from Martin.

The public acknowledgement of the discovery posted by FON is found in this forum post. Only in the English forums, by a user created apparently for this particular purpose, as this is his first post ever, where it is not likely to draw much attention. This would be fine by me, had not there been the precedent of Martin’s fierce replies to my statistics post, followed by countless attacks by FON’s followers, including an unfortunate incident better left forgotten. What I really cannot understand is that, when I criticize FON, I get such a huge public lashing, whereas when I help them out, I get a three-line remark in a forum where it will go mostly unnoticed. The end result may well be that other vulnerabilities, and it is likely they exist, go unreported.

Whatever the case, this should show those who accused me of unfair, biased attacks on FON that I really just call the shots as I see them, when I smell bullshit, I will point to it, when I see a hole, I will help them fix it – again, IMHO, blogging is not about being or not biased, it is about being ethical and maintaining a set of standards. In my view, it should also prompt Martin to write an apology, but I am not holding my breath. Not that I care much either, what is most important is my work; this is my blog, where I spend part of my spare time, which is not actually that much.

The naked Fonera

After a few days of silence, digesting the hubbub created by my analysis of Fon’s status, I’ve put my head back into more useful things than answering hate mail and out-of-line comments (thanks to those who provided balanced views, either for or against!). So, I decided to open a Fonera and see what lives inside.

A full review is coming, but first impressions:

  • The plastic casing looks and feels very nice, the molds must have been expensive, as the different parts mate very well.
  • Inside lives a single PCB, with components on both sides. The top holds the bulkier components, such as power regulator, RAM and WiFi section, inside an aluminium RF shield.
  • The PCB looks professional and well laid out on first inspection.
  • Components used (I haven’t opened the aluminium chassis yet) are older SOIC and TSSOP, thus cheaper to handle and solder. Balled components require from special handling, such as baking in hydrogen for 24 hours to dry them before soldering, etc.

Here are some pics (click each photo for bigger views on Flickr) I have taken with a Nokia N93 (really nice phone btw, mini-review coming):

Fonera - underside of casing

The underside of the case, with screws off.

Fonera - perspective view

Perspective view of the top PCB.

Fonera - Bottom PCB

Bottom side of the PCB.

Fonera - firmware version

Sticker on the flash IC showing the firmware version.

The real FON statistics – lies, manipulation or fantasy

On September 14th, FON launched the new version of their online mapping service, after several months of complaints from users that the service wasn’t up to scratch, and announcements stating the development team was working on the problem.

On a first look, the maps look really nice – they use Google Maps, by default in the mixed view, where you see a satellite image and an overlay of roads and placemarks. I will not bore you with the details, as it is better that you check them out yourself and make up your mind.

This post is not intended as a review of the service itself, but rather, a revelation of the real figures behind FON’s network – peeking under the layer of PR and flamboyance. Martin Varsavsky is always boasting about FON being the largest WiFi community of the world – in my view, this is not accurate.
During months, FON has been claiming to be a “movement”, with a marked communist image behind (the marching workers, the spray-painted logos, etc.). This movement was supposed to kill mobile operators, who currently oppress people with their sky-high tariffs. We could go into a long debate just on this topic, but lets move on. During all this time, FON has suffered untold problems with staffing, PR mini-scandals, shipping broken routers or taking weeks and months to even send them out, not replying to repeated requests to support@fon.com, and blatantly ignoring the public forums, where the community behind the movement was expressing its increasing anger and frustration.

The blinding truth – less than 3.700 routers online worldwide

Digging a bit deeper into the workings behind the maps, I have found that there is a method to run a query to retrieve all the hotspots in FON’s database, not just two hundred, or those in a particular region. If you want to see an example, click here. This is a query that will return all hotspots on the planet that have been FONing home during the last hour. It can take a little while to load, so be patient. Until a couple of days ago, results were returned in XML format, which has been dropped in favor of the new plain, comma-delimited format.

I predict that FON will not like the above link, and thus will try to either change the format of the php call, or add artificial records to confuse the application I have written to process the data. First, I wrote a simple application using RealBasic (having been a long-time Visual Basic acolyte, it is a welcome change, allowing me to code under Mac and Windows transparently) – source code here. A screenshot of a full run is shown below.

Application screenshot

Just from the details shown after the run, a few enlightening facts surface:

  • The highest user ID found is 92.192, but the total amount of processed records is only 55.384. I have to investigate a bit further, but it appears that in some cases, a record is stored twice, once holding the user type (Linus, Alien or Bill), and again holding the router mode (online or unknown). This is the reason why some people see both the orange dot and the green halo on their locations at maps.fon.com, and also the reason why at this time I cannot confirm that the real number of Foneros is 43.896.
  • There are only 3.674 routers online on the entire planet. So much for the largest WiFi community in the world. The other 7.814 are registered routers, from which nothing has been heard during the last hour. These figures have been checked a few times during the last few days, and they stay more or less constant.
  • Out of the entire user base, only 1.317 have become Bills. So much for milking one’s WiFi.
  • The highest router ID found in the results was 19.889, so if we add offline and online routers (best case scenario), then around 8.401 routers have never been registered, representing 42.2% of sold routers. Extrapolating this to the 1 million routers Martin wants to sell would results in a loss of $10.55 million!.

Looking at the per-country statistics (per-city could be made, given some extra time and coding), some curious details also stand out:

  • There are two registered routers in Afghanistan – but neither is online. Not surprising, considering the amount of explosives that have been dropped on the place.
  • China and Taiwan have 9 routers registered, but none online. Martin was blogging about his expansion into Asia, which looks rather bleak right now. 165 Foneros are registered however.
  • Germany and Spain have around the same number of registered routers, although Germany almost doubles Spain in the number of online routers.
  • The United States ranks third in number of registered and online routers, however, it holds the highest number of Bills (408). The next is Germany, with 237.

Finally, we can derive a few figures from these numbers. These are highly interpreted, and must be taken as theoretical extremes.

  • If FON sold one $3 one-day pass every day of the year on each of the online routers, it would make a gross income of $4 million. This is before tax and the Bill’s share where applicable. You at the back, stop giggling!
  • Making a wild assumption that each router’s signal reaches 100 people, FON would only cover 0.11% of Germany’s population of 82 million.
  • Boingo gives you access to 45.000 hotspots. FON has about 8% of that figure, and with location quality debatable – it is a fact most FON hotspots will not be optimized for even street-level coverage.

I believe it is time for FON to stop boasting about having the largest WiFi community in the world, and start concentrating on its real problems. And if they still don’t know what these are, they have a nice summary at the online forums. Besides, for spending 500.000 Euros per month, this is a pretty poor show, in my humble opinion.

The clueless pitcher’s guide to TechCrunch parties, and mini-review of TechCrunch 7

Friday night I attended Michael Arrington’s TechCrunch 7 party, held at August Capital in Menlo Park, where VCs, startups, and where everyone who is anyone was supposed to be. I saw Robert Scoble, Shel Israel, Guy Kawasaki and his permasmile, Michael of course, and a bunch of other people. I also managed to get in touch with those who I had in mind, so all went well.

What I found curious was the large amount of atendees who were wondering around with a clueless face, obviously looking for a VC or famous person to pitch their startup. I talked to a guy that wants people to anonymously share their wage, so that others in the same industry can check whether they are being overpaid (yeah, right!), or driven as slaves. The revenue model? deep breath…ready?……advertising! The startup I’m involved with right now has ads right at the bottom of the food chain, as something nice to have. Here are a few tips for you guys, when you attend the next TechCrunch party:

  • Plan who you want to talk to beforehand, and go directly to get them. Arrive early, well before the start time, so you can at least catch Michael, if he is on your list, before he starts getting nervous from all the attention. You will find it quite hard to get him to listen to you at 11:30, with the amount of people he always has around. However, if the person you want to talk to is not such an attention-getter, but likes his spirits (in an alcoholic sense), maybe he will sign you a nice blank check, so it may be worth a longer wait.
  • Mount guard in front of the restroom. Everyone is bound to go there at some stage during the party, specially as the flow of beverages increases. Most people won’t follow Scoble to take a pee with him while they discuss the next video blogging development – but you are desperate, so go for it!
  • Have a great conversation startup line ready – going up to someone and babbling something about how much you follow their blog or what great things they have done will usually get you a yes-I-know-you-do-like-all-the-other-hundreds-of-thousands-like-you smile, a handshake if you are lucky, and a chance to take your photo with them, but that is it. For example, if you wanted to talk to Guy Kawasaki, you could say “hey Guy, I am looking for advice, so I will ask you for money” (Hint: read his book). Guy, if you read this, let me know if it would have worked – it was the best I could think of.
  • Wear a tee that says something about your startup. A colleague of mine wore one that always got the “what is xyz about?” question. Don’t overdoit, so you look like an walking ad, and don’t put your 10 PowerPoint pitch slides on the front and back. A simple, intriguing, eye-catching message will do. As a practical example, take a look at how many pics of that girl with the “hate your job?” tee are on Flickr.
  • Take your laptop, making sure the battery is fully charged, and also take a printed copy of your product presentation or pitch. Have it printed on quality paper and nicely bound – it can cost you $50, but will you regret it if it helps bring in $5 million? Don’t take them with you to the party, nothing looks worse than someone walking around with a PowerPoint presentation under his arm – leave them in the car, you can always quickly dash out to get them if someone gets so interested in you that they want to have a more official presentation on the spot.
  • Be prepared to take criticism, as you will meet people who are completely oblivious to what you are doing, and may just have some fun taking a big poke at your ideas. This is something that actually happened to me with a guy from this company that starts with Y and ends with hoo!.
  • Make yourself a decent badge – don’t just write “Mike” on it, write your name, surname, and the company you represent. I noticed almost everyone that crossed your path looked at your badge, so make it clear and easy to read.

As a whole, the party was quite good. It even had a streaker, who most likely earned only $100, as a proper streaker would have been completely naked (and maybe with a pitch written on his back). There was plenty to eat and drink, and many interesting people to talk with. Some random thoughts:

  • One comment to the Pandora guys: next time, invest some money on a decent audio system, the two tiny speakers and cheap microphone just didn’t cut it. Also, place a couple of speakers at the far corners from the stand, otherwise the people at the end don’t even realise someone is giving a speech. The Pandora service itself rocks!
  • It was surprising that when we arrived, the registration desk hadn’t received the Wiki list, thus entry was denied to non-sponsors for a while. The situation was quickly resolved, and all turned out well.
  • The icecream sandwitches were awsome, please have them around at the next party!
  • The lights could have been switched on early, there was a time when it became hard to read other people’s badges.
  • It was curious to watch Robert Scoble handle his high-def camera while recording Michael’s speech – he went through like a dozen menus to configure the low-light settings. Whether he had to go through so many options was because they were really needed and he completely masters the menus, or it was him being still a bit new to the camera is unknown. Here is a pic of him handling the thing:

Scoble and his camera

  • The Soonr demo was excellent, albeit Song was very excited and looked a tad over-revved. Thanks for the tee! I will be certainly be using your service, the Skype relay looks very promising.
  • The Plazes CEO was very dry when we asked him to confirm rumours about their relationship with FON (other than Martin Varsavsky investing in them), and as to them handling the geocoding of FON’s hotspots. There has been a deluge of criticisms towards FON’s Google maps, and the inaccuracy of the positioning of the hotspots, so maybe this was the reason he was uncomfortable.
  • I have a theory – Michael Arrington envisioned the riot of people trying to talk to him, and thus placed some convenient doubles around the crowd. This picture is an example:

Fake Mike Arrington

  • The venue was great, plenty of space, plenty of seating, and the weather turned out great – thanks go to Augusta Capital for providing the space, and to Michael for hosting the event.
  • Guy Kawasaki seemed to come to take pictures of everything, at least, that’s all I saw him do – when I wanted to find him later on for a quick talk, he was nowhere to be seen.
  • I can recommend the Menlo Park Inn if you want accomodation in a well-communicated place (at least by car), and particularly, with free WiFi. The rooms were spacious and tidy, and the breakfast simple but sufficient, and included in the room price.

In all, I had a really good time, got things done, and so I am looking forward to the next one!

My last day as a Fonero – bye FON, hello future

Today is my last day as a Fonero, which is the way people registered in FON’s network are called (IMHO a rather ugly name). Why this decision? There are a number of reasons, and I have chosen to simply make a list.

  • The most important reason is that I have taken a position at a company that makes it unethical for me to continue participating in FON. I will no longer post on their forums; however, I will continue to post my thoughts about FON on my blog, and replying to Martin Varsavsky in his blog when I see it appropriate.
  • FON has been a downhill experience from day one. I ordered my “social” router, and got charged by PayPal, but no confirmation from the company, no tracking number, nothing. I emailed their support address, no reply. It eventually arrived, admitedly faster than the month or two some people were reporting on the forums. After a few futile attempts at configuring the router to work with my DSL line, and a couple of completely ignored emails to FON support, I simply gave up. The router is now waiting for a PCB to turn it into a WaRThog.
  • Every time I see a new crazy idea in Martin’s blog I feel more depressed about the FON project – does he really think WiFi is the way for homeless people to make a living, reselling VoIP services over Bluetooth? (don’t ask!). Where would he send them the money? Then there are the times when he takes a product and claims it was designed by FON, sometimes in secret collaboration with his backers Skype or Google. The latest is the Skype-compatible WiFi phone made by an Accton subsidiary – this is a design that Accton started way before FON even saw it, and way before Martin could have his logos photoshopped onto the mockups. As a matter of fact, out of the box this phone will not work at FON hostspots, as it lacks the browser required to perform user login – so they will have to work some magic.
  • The english and spanish forums are another source of disappointment, with daily posts from people complaining about the extremely poor support that FON is providing them. Some have even taken to posting comments on Martin’s blog to air their issues, something blogtiquette considers a no-no. I posted a few days ago about this particular issue.
  • They have followed an ill-conceived path to gaining publicity through bloggers, resulting in serious backslash from the spanish blogosphere (see here and here). Martin seems to think that by surrounding himself with top bloggers in exchange for dubious stock options or a seat in the board will get him a free ticket to stardom.
  • I believe that FON serves two purposes – one is to give a personal vehicle of shininess to Martin’s ego. See this post by Glenn Fleishman on FON’s crazy deal announcements, later called off as a lie by Speakeasy – typical example of how he manipulates a phone conversation into front-line news. Om Malik also reported on this particular issue. Martin is someone who cannot be seen as co-founding anything, but as a leader and innovator.
    Secondly, FON serves as an experiment for Skype and Google, who somehow convinced Index and Sequoia to go along. I don’t believe the two VC firms are into experiments, but FON would certainly provide good feedback to S & G about socializing WiFi, hardware distribution, and the adoption of the Bill model as a viable way to extend a WiFi network. Other stuff such as amount of logins at each location/router, number of registered users, daily passes sold, etc. would make nice colored graphs in the resulting corporate presentation.
    But, the problem is that FON is a huge fiasco in terms of hardware distribution, firmware development, public relations, and costumer support. I thus question the validity of any figures that come out of this rather expensive experiment.
  • Their firmware development process seems to be a closely guarded secret – but not for the same reasons Apple safeguards its own developments. FON started working with Brainslayer, the creator of DD-WRT, a free Linux distribution for Linksys (and other) routers. Apparently, Brainslayer was not very well treated by FON, and he parted to work in the Sputnik project, amongst other developments.
  • Just as Mark Evans did, I have voiced my concerns about FON’s business model and strategy – now that they finally launched the Bills, it looks more ill-fated than ever.

I find it really amazing how FON, with the $21.7 million they got in funding, cannot manage to hire a competent team of support personnel, outsource their obviously ill router redistribution system, and get some muscle behind the community effort. Martin Varsavsky is known in Spain for starting companies, pumping them up, and selling at the best possible gain – then leaving them behind with serious problems. Just look at what people think of Jazztel, or what troubles the Ya.com portal went through.

For me, the FON adventure is over, and a new, better adventure is starting. We will start disclosing things around the end of August, so if you want to stay updated, you are welcome to subscribe to the RSS feed.

How can FON expect to win?

Today I decided to attempt a second round at configuring the router FON sent me a few days ago, since my first out-of-the-box experience hadn’t been that good. Emails to tech support unanswered, which seems to be an endemic problem, as can be seen on FON’s forums, I finally gave up.

After plugging in the WRT54GS router as briefly described in the brief manual supplied with it (a third of one side of an A4 sheet of paper), I connect to the FON_HotSpot SSID detected by the MacBook. Fire up Firefox, and I’m promptly greeted with a welcome page that states the router could not configure itself, and thus has no connection to the Internet. It shows a few scenarios that one can check for problems, also suggesting one should consult again the third-of-a-page-handbook, and, failing all this, to try manual configuration of the router.

After about an hour of changing IP addresses of the WAN and LAN interfaces (and where is the WiFi interface? or is it linked to the LAN or WAN?), I have finally given up again. I’m not a networking überguru, but I know a bit about routing and setting up IP interfaces, and this thing just managed to get on my nerves. You cannot find a clear manual with diagrams of network connectivity, setups and scenarios, a description of the theory of operation of the hotspot, and as it has been shown, sending emails to FON support is usually futile. The forums are more helpful, but not because there is a healthy bunch of FON staff there, but because a number of talented and skilled individuals have taken upon themselves the task of helping others through the ordeal.

I’m sure that a lot, if not most, users that plug in the FON router can simply connect to it, register and start surfing, but in cases like mine, where I simply have a DSL router to which I plug in the FON router and it’s supposed to work – but doesn’t – a blank void is all there is left to stare at.

Maybe a last attempt will be to flash the new release of the firmware, once they have fixed the problems in v.0.6.6

Bottom line is that FON cannot expect to create a WiFi planet with people roaming for free on the 1 million routers they are going to distribute, once they get their logistics right, based on complex hardware that requires from either skilled operators, or very good tech support and clear setup and troubleshooting guides. A couple of days ago, someone posted on the forum that FON was a beta company. How can a company class itself in beta? It can have a service in beta, but the company must be running, if not totally smooth, at least with agility and responsiveness, fixing its problems quickly and providing first-class customer service.

FON – a bliss for ISPs?

Today, I received a nice email from Martin Varsavsky of FON, telling me that they have activated the Bills mode. This means that, in theory, you can resell your excess bandwidth to other users via your WiFi router, sharing the generated revenue with FON. People passing by can open their laptop, connect to your signal, and purchase access packs for $5 $3 $2 or 2$ a day – yes, the revenue model has been changing that much over the last few weeks.

In theory I say, since many ISPs have very restrictive terms in the service contracts you sign with them, some in the US even going as far as classifying sharing your broadband with others as a federal crime. So, one of the first items in FON’s Ts&Cs is that you must have your ISP’s permission to resell your broadband. So far, very few ISPs have signed up for this sharing model, one being Jazztel, a company started by Martin himself.

The email also states, this time in size 8 font, lest anyone notice it, the following:

“We would like to inform you that from now on it will be FON WIRELESS, LTD., the entity providing the Service, instead of FON TECHNOLOGY, S.L., and that we have made some minor changes to the Terms and Conditions”

The new Terms and Conditions [PDF] are a further developed version, no doubt washed down with plenty of lawyer juice. The point that most interests me is section 6.6, where the contract states that “As a Linus or Bill, you agree to keep the FON Hotspot active during 24 hours 7 days a week in order for the other Linuses and Aliens to be able to connect thereto.” Additionally, FON USA’s shopping conditions state that you have to keep the router active for one year.

So, apart from FON, who are the big winners? The ISPs of course! Martin was right in saying that FON is good for ISPs, as they are forcing FON members to keep their routers online all year round. Nowhere in the contract one can find what the penalty for not complying is, but some posts suggest they will charge you around $50 to cover the cost of the router they sent for $5.

The only way that you can prove to Fon that your hotspot is active is by allowing the router to phone home on a regular basis, which means that you must have a broadband connection with an ISP for this to happen. If you cancel your broadband, you will automatically violate FON’s terms, and thus be liable to whatever charge they want to levy. Fair?