Me 2.0

Today’s cloud service providers come from two ends of a spectrum: On one side are IaaS providers like Amazon AWS, Rackspace and Terramark. They sell CPU cycles and Hard disk space.

The other camp is the SaaS ones: Salesforce, Webex, Google Apps and the like. They sell a fully developed application hosted somewhere out there.

However, the gap between these two is getting narrower by the day: Microsoft is working hard to add more added-value services to Azure. Amazon is opening up its CloudWatch service to become a business intelligent dashboard tool. Salesforce is building Database.com and buys Heroku. Werner Vogels, CTO of Amazon said the same thing in the Structure conference.

It seems that Cloud computing is maturing up and moving to become more seamless. This is a reaction to the push from the users. Until now, most of hosted-on-the-premise applications were simply put on those cloud boxes to satisfy the “cloud initiatives” of their companies. On their way, they would gain some availability and reliability but there was still a lot that could be achieved by designing a system truly for the cloud.

PaaS providers like dotCloud are providing some added value on top of the basic CPU cycles and VMWare is trying to sell its vSphere 5 as the OS for the cloud. However, the main move is going to be to a Component As A Service (CaaS) ecosystem where applications will be reduced more and more to their core functionalities and leave the rest to the cloud providers.

These will be the systems that help programs become systems in a short period of time. They will make developers stars by letting them focusing on supporting the business, handling its growth while reducing the time to market short.

The most common in-place-editor in Rails is In Place Editing. However it doesn’t not work very well with nested resources or non-default routes. I used Rest In Place by Jan Varwig in Rails 3 and I am very happy with it. Here is how:

First, install the plugin. In Rails 3 it is:
rails install plugin git://github.com/janv/rest_in_place.git
Then include the jquery.rest_in_place.js to the scripts loaded on the page you’d like to have the in place editing on.

Once that’s done, make sure the jQuery loader, starts the scripts:

jQuery(".rest_in_place").rest_in_place(); After this, you can add rest_in_place as the class name of the wrapper element around your editable item. Something like this:
<span data-url="/users/1" data-object="user" data-attribute="name">
<%= @user.name %>
</span>
Jan has a pretty good documentation of this on his website. There are some gotchas/points however:

• Make sure your Show action responds to JSON. This way you won’t get stuck on “Saving…”
• Currently there is no way to show the validation errors although the save would not proceed if there are any.
• You can add a CSS for rest_in_place:hover to make sure users know the item is editable. It could also include a cursor:pointer as well.

Heroku provide Ruby on Rails hosting. They do it their way: It’s not a server but some “dyno”s and “worker”s with mythical Japanese names. There is no MySql either: They run PostgreSql instead. Nothing so bad about this so far. There is nothing wrong with their service: It works (somewhat) and does most of the stuff you expect a RoR hosting company to do. But it’s a joke:

First of all, they have taken the concept of opinionated software to a totally new level. I mean if you’re a RoR developers you’re familiar with the concept: From naming conventions to file locations, folder structures and to the almost cult-like community lead by “DHH”. But even for RoR developers, Heroku (which I never get used to typing fast), is a new level of doing things their way: No DB other than Postgres, No source control except git, No deployment method apart from their way, and no other tool except their toolset.

Now they claim this is all to give developers a full set of tools to allow them get up to speed very quickly and have something up and running in minutes. But it is all not quite so: They claim “most” RoR apps should run just fine with Postgres as long as they don’t have any DB specific code. Well this is not quite true even for some native RoR migration scripts (and Postgres is not quite helpful in returning a decent error description, so you’d have to find your way out of the hole blindly).

On top of that, I think the whole git and heroku toolset is built for people who don’t have any other git accounts. If you do, like, let’s say an account with Github, you’d have to do a lot more to get their system to work nicely with your git setup. Look around on Stackoverflow to see people asking questions about this. This is all result of sticking to their OWN way of doing things.

Then you get to the scaling: The whole Dyno and Worker business is smoke and mirrors to hide away how much horse power you can get out of this thing. After all everyone is familiar with CPU cores, GB RAM and nowadays VPUs. Heruko’s way of describing this is based on totally new made up concepts which is again unnecessary distraction.

After that, we get to the whole DB thing: There is no way for you to access your Heroku DB if you’re using their shared DB. Granted, it is supposed to be for testing and prototyping. But isn’t that a reason for you to want to have a look into your database to see what your “prototype” or “experimental code” is doing? If you want to see what is in the database, you’d need to cough up at least $200 per month for a dedicated DB instance.

On top of that, the whole pricing thing is ridiculous: You can have 1 dyno (whatever that is), no workers (I guess background delayed job workers) and a 5MB (yes MEGA BYTES) for free. Thinking about more DB space? $15 per month gets you 20GB of PostgreSql with NO access to it from outside your app whatsoever. You actually want to look into the DB? $200 please.

Anything else is going to also cost you an arm and even more legs: You want to see AND follow the logs? Buy an “add-on”. Need an hourly CRON? $3.00 per month please… and the story goes.

So, now here I am thinking: It took me 5 minutes to get my RoR 3 app running on Dreamhost. To be fair, Dreamhost is not the most reliable hosting company. But for my $200 per year, I get to host as many apps as I want, have unlimited MySQL storage, unlimited Subversion repository and a lot more.

It took me more than half day to change my app to work on Heroku (from DB change – and yes the whole code was allegedly DB agnostic – to source control issues, git SSH RSA key issues,…). After that I have an app that is only usable for testing purposes. With these prices I’m never going to run my app on Heroku for production.

So I’m asking myself this question again: Why would I go with Heroku?

Fast 0 to 60? Nope: It is quicker to deploy a RoR app to Dreamhost than Heroku.

Unlimited (or large) Db/file storage space? Nope: 5MB Postgres + Readonly file system

No lock-in? Nope.

Scalability? Well yes – allegedly, but only if you are sitting on a pile of cash to buy a lot of Dynos (I guess they make your app scalable, right?)

So why? If you are competent enough to run your own web server and have some money (not too much), then why don’t you run it on EC2 yourself?

If you don’t want to deal with web servers and load balancers (although EC2 makes it super easy), then why not go with something like EngineYard?

If you want something you can just try an idea then why don’t you do it on Dreamhost or another bog-standard apache shop.

If you want control, but don’t have the money (well in that case you should go near Heroku), why not a VPS like Linode?

However I look at it, I can’t find a good reason to go with Heroku. That’s why I think it is a joke. And not a funny one.

While the technorati of Europe were busy mingling at LeWeb 2010, students were breaking windows and throwing eggs at cars in London. I was so busy talking to people about Sentimnt, that I didn’t really have time to follow the news of student protests back home.

Now however, coming back home and having been through the experience of LeWeb, I can see a link between the two, somehow.

For the past 10 years, many have been complaining about the declining quality of British graduates and the shrinking number of the ones taking science and engineering courses. Many have complained about the increase in the number of students seeking degrees in “Media Studies” and course like that.

Proponents of the rise in the student fees, state that making students pay for part of their studies will make them think twice about taking Media Studies and will opt for a degree that will have a higher probability of paying off their debt when they graduate. This is a hypothesis that only time can prove. I tend to agree for now.

Back on the tech scene, there has been a lot of talk about how the cost of starting a technology/IT business has been falling constantly. Now that storage, processing power and even data is turning into commodities, young entrepreneurs have a very low barrier of entry when starting their new businesses. I also can’t disagree with this.

Computer industry has always been somewhat cultish. It’s the case for most new industries. The pioneers of the car industry had followers the same way Steve Jobs now has followers. Looking back at my “heros” when I was starting to learn programming on my Commodore 64, I can remember how Bill Gates, Mitch Kapor, Steve Jobs and Steven Wozniak had God like status in my mind. I grew up believing in the importance of brilliant ideas and more importantly, great engineering.

That’s why last weeks LeWeb was disappointing for me. Not because I didn’t get to meet whom I wanted to meet or didn’t get what wanted to get. But because I realised how the heros of the IT industry have changed. Heros of the new social age of computing are mostly people who build their businesses by picking pieces of technology from the vast shelf of open source library and put them together. They proudly call themselves Hackers, a name that 5 years ago meant someone who codes with a very short sight and no care (or knowledge) about maintaining a system for the long run.

Don’t get me wrong, there are plenty of great companies with exceptional engineering behind them, also the big players in the new era of social computing have built impressive technologies to deal with the “Web scale”. Cassandra and Hadoop are two examples. But there is no way anyone can convince me that you can compare the engineering talent and discipline of companies like Intel, HP, Microsoft, Sun, Oracle, Google or Apple with the new social heros on the block like Groupon, FourSquare or even Twitter and Facebook.

Where are the rioting students in all this? Well, here is my worry. I am worried that the combination of low barrier of entry and startup cost and the replacement of old tech-heavy heros with MBA like managers who can grow a social network to 500 million people is shifting our talents to businesses with less added value.

In the middle of all this, the over enthusiastic media outlets also fan the flames by focusing almost primarily on business similar to Groupon where technology is almost negligible and the whole aim is making as much money as possible. And who benefits the most from all this? Investors who love 10X exits in shorter and shorter times. No wonder outfits like Y-Combinator love to promote the “Hack” culture. After all, who cares about the long run if you can put some open source code together, give it a social twist and shift it for 10 times more than what you put in it? Knowledge? Engineering? You’re waisting your time!

To server a host-meta file in the .well-known folder from an ASP MVC application hosted in IIS, the easiest way is probably saving your host-meta file in a folder then adding a route and action for it.

So this would be your route in Global.asax

routes.MapRoute("HostMeta", ".well-known/host-meta", new {controller = "WellKnown", action = "HostMeta"});

and here is the new controller to serve it:

    public class WellKnownController : Controller
    {
        public ActionResult HostMeta()
        {
            string result = System.IO.File.ReadAllText(Request.MapPath("~/somewhere-on-the-server/host-meta"));

            return Content(result);
        }
    }

Recently I needed to convert the Json files generated by mongoexport from a MongoDb database to XML so I can analyse the data in Excel. I used Json.NET and the following code to do that.

        static XmlNode ConvertToXmlNode(string json, string nodeName)
        {
            return JsonConvert.DeserializeXmlNode(json, nodeName);
        }

        static void Main(string[] args)
        {
            if (args.Length != 2)
            {
                Console.WriteLine("Usage: convert <path> <filename>");
                Console.WriteLine("No extensions needed. Uses .json for import and .xml for export");
                return;
            }

            string path = args[0];
            string filename = Path.Combine(path, args[1] + ".json");
            string dest = Path.Combine(path, args[1] + ".xml");
            string nodeName = args[1].ToLower();

            Console.WriteLine("Loading " + filename + " and converting to " + dest);
            string[] lines = File.ReadAllLines(filename);
            Console.WriteLine("Loaded " + lines.Length + " lines");

            XmlDocument doc = new XmlDocument();
            XmlElement root = doc.CreateElement("root");
            foreach (var line in lines)
            {
                var imported = doc.ImportNode(ConvertToXmlNode(line, nodeName).FirstChild, true);
                root.AppendChild(imported);
            }

            doc.AppendChild(root);

            doc.Save(dest);

            Console.WriteLine("Done");
            Console.ReadLine();
        }

As part of developing Sentimnt (the Personal and Social Search engine) we needed to authenticate users with Gmail/Google Apps with IMAP OAuth. Here is how:
I looked for this one around the net for a long time and ended up implementing it myself so thought I should share it:

If you are going to use oAuth authentication with Gmail using IMAP in C# the you’d need to sort out two things:

1. Get Access Token and Token Secret from Google
2. Use an AUTHENTICATE method in IMAP protocol

This code uses dotNetOAuth for the oAuth part to get the access token and secret code. As for IMAP implementation I use LumiSoft IMAP and added AUTHENTICATE support to it.

Assuming you have the Consumer and TokenManager that you’ve used to get the AccessToken and TokenSecret, this will help you to build the XOAUTH parameter needed:

        public static string GetXAuth(string consumerKey, string consumerSecret, string token, string tokenSecret, string emailAddress)
        {
            Uri uri = GoogleOAuthHelper.GmailEndPoint(emailAddress).Location;
            OAuthBase oAuthBase = new OAuthBase();
            string normalizedUrl;
            string normalizedReqParams;
            string signature = oAuthBase.GenerateSignature(uri,
                                                           consumerKey,
                                                           consumerSecret,
                                                           token,
                                                           tokenSecret,
                                                           "GET",
                                                           oAuthBase.GenerateTimeStamp(),
                                                           oAuthBase.GenerateNonce(),
                                                           out normalizedUrl,
                                                           out normalizedReqParams);

            var parameters = HttpUtility.ParseQueryString(normalizedReqParams);
            parameters.Add("oauth_signature", signature);

            StringBuilder sb = new StringBuilder();
            sb.AppendFormat("GET {0} ", GoogleOAuthHelper.GmailEndPoint(emailAddress).Location.AbsoluteUri);
            IList<string> keys = new List<string>(parameters.AllKeys)
                .OrderBy(item => item)
                .ToList();
            foreach (var key in keys)
            {
                string value = parameters[(string)key];
                sb.AppendFormat("{0}=\"{1}\",", key, HttpUtility.UrlEncode(value));
            }
            string baseString = sb.ToString().TrimEnd(',');
            return Convert.ToBase64String(Encoding.UTF8.GetBytes(baseString));
        }

This is not something new. Anyone who has ever wanted to register a domain name knows that it is impossible to get the name you want and next to impossible to get anything meaningful. Hence the web startups with strange names like Spotify, misspelt like Flickr, outright weird gnolia (of ma.gnolia.com) and also the trends of adding suffixes to domain names like “fy”, “sy” and so on.

This is almost entirely fault of domain squatters who pay peanuts for domain names and sit on them forever until a buyer is desperate enough to shell out thousands of dollars for the hard work. To ease this pressure ICANN adds new top level domain names or remote islands sell the rights to their top level names to companies (.tv for example)

I personally think the only way out of this is to increase the price of .com registration – a lot. A $200 .com domain (like it used to long long ago) makes squatting very expensive and also stops people from panic buying domain names in case they might need them in the future.

Come on ICANN, go against the market pressures for once!

I needed to use a CDN in my ASP MVC app and needed it to be easy to develop against. I chose Amazon CloudFront since our servers are running on Amazon EC2 and we are also using Amazon S3 during our deployment. You can get a good introduction and step-by-step on CloudFront here.

As for the ASP MVC side, I wanted to have something like Url.Content where it converts my URL to my CDN URL while allowing for multiple development environments.

Here is how I did it:

A helper extension class on System.Web.Mvc.UrlHelper to add a new method to Url: This one is CdnContent

Here is an implementation

using System;
using System.Collections.Generic;
using System.Linq;
using System.Security.Policy;
using System.Text;
using System.Web.Mvc;
using System.Configuration;

namespace CdnHelpers
{
public static class CdnHelpers
{
private static AppEnvironment _appEnvironment = AppEnvironment.Unknown;
private static string _cdnBase = string.Empty;

private static AppEnvironment AppEnvironment
{
get
{
if (_appEnvironment == AppEnvironment.Unknown)
{
_appEnvironment =
(AppEnvironment)
Enum.Parse(typeof (AppEnvironment), ConfigurationManager.AppSettings["App.Environment"], true);
}

return _appEnvironment;
}
}

private static string CdnBase
{
get
{
if (string.IsNullOrEmpty(_cdnBase))
{
_cdnBase = ConfigurationManager.AppSettings["Cdn.Base"];
}

return _cdnBase;
}
}

public static string CdsContent(this System.Web.Mvc.UrlHelper url, string contentPath)
{
if (AppEnvironment == AppEnvironment.Local)
{
return url.Content(contentPath);
}
else
{
contentPath = contentPath.TrimStart('~');

return string.Concat(CdnBase, "/", AppEnvironment.ToString().ToLower(), contentPath);
}
}
}
}

AppEnvironment is an Enum:

    public enum AppEnvironment
    {
        Unknown,

        Local,

        Test,

        Dev,

        Staging,

        Production
    }

You need to add two keys to AppSettings of the Web.config:
App.Environment to determine which environment the code is running as and
Cdn.Base which would be something like http://cache.mydomain.com (no trailing slash)

Also add the namespace of the helper class to System.Web/Pages/Namespaces under Web.config

This should be it. Now you can replace a Url.Content with Url.CdnContent in the code.

I didn’t want to redirect all files by their extension to CDN but that’s not difficult to implement.

On deployment to Amazon:

• Amazon CDN files and folders are case sensitive.
• Create folders with the name of your environments under the root of the S3 bucket and deploy the files and folders there.

At my company, we are building a fantastic personal search engine (still in private beta. Stay tuned!). As part of our architecture, we needed to cater for scalability and redundancy from the very beginning, so we went for cloud hosted services. We wanted to be able to deploy any of the layers of the system within 15 minutes to cope with the load. Our system is running on Windows OS and is written in .NET 3.5. We tried GoGrid, some smaller Hyper-V VPS providers, Windows Azure and Amazon EC2.

I wrote some time ago about what I think about Azure at this point in time. Since then we tried standard Windows OS cloud providers and haven’t had time to play with Azure again. I’m sure it is going to improve. However, the acquisition of Atebits (the guys behind Tweetie) by Twitter made me think about Microsoft and Amazon again.

Building companies on Microsoft technologies is a risk. You are exposed to Microsoft going bust, changing its direction, etc. But how likely is that? As far as a small business is concerned, negligible. You can safely build a small/medium business and be exposed to Microsoft. Amazon is smaller than Microsoft but for me it is big enough. So I don’t worry about exposing myself to Amazon going bust. But there is one difference between building my business around Azure and building it around Amazon EC2: Lock-in.

Azure locks me in. I need to make changes to my product for it to work in Azure. Amazon EC2 on the other hand is just commoditized Windows Server horsepower. If Amazon decides to raise its prices to a level that is not affordable to me, I can switch to another dozen providers giving me same stuff. And that’s a big problem as your business gets bigger.

The other day some people from a big data integration company were giving us a presentation about their middle-tier systems. The said their system can run on Amazon EC2 and the CTO of the company wasn’t sure if he wants to build exposure against Amazon. I see it another way though: Being exposed to Amazon in this way is the same as being exposed to your ISP. They can go down and take you out of business for a week. But you CAN recover within a reasoable amount of time. Building my business around Azure isn’t the same.

Risk = Probablity x Impact

The probablity of Microsoft making breaking changes to Azure is very low but the impact is very high. The probablity of GoGrid going bust is much higher, but the impact is very low (I can switch or have a backup on another cloud provider).

As it stands, running your business on Azure is like building a business around Twitter. This can change if Azure changes to a more commoditised computation power provider or allows other companies to provide Azure as a service.