Wheel of Ezekiel

Javascript scoping problems? Is it Firefox, IE or... Prototype?

2007-11-15T14:47:00.000-08:00

I was asked to assist a friend who was experiencing some unusual behaviour in his web application. While it worked as expected in Internet Explorer, it was failing in Firefox, so he asked me to have a look.

The relevant source code:

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<script src="inc/third_party/prototype/prototype.js" type="text/javascript"></script>
<script src="inc/third_party/tooltip.js" type="text/javascript"></script>
<script src="inc/third_party/protoload.js" type="text/javascript"></script>
</head>

<body>
<div id="main">
<script type="text/javascript">
function update(orderby,sort,page) {
  $("80").update("<center><img src=\"images/loading/waiting.gif\"/></center>");
  var request = "module/80.php?orderby="+orderby+"&page="+page+"&sort="+sort+"&status_id=80";
  new Ajax.Updater("80", request, { method: "get" });
  }

  new Ajax.Updater("80", "module/80.php?status_id=80", { method: "get" });

</script>

<div id=80>
<img src="images/loading/waiting.gif"/>
</div>
</div>
</body>
</html>

The loading AJAX imported a table with headers like this:

Which rendered like this:

The expected behaviour was that clicking the header would pull in an appropriately sorted table via an Ajax request, and it worked as expected in IE7.

In Firefox however, clicking a header simply replaced the header with the orderby value being passed in the update() function:

Several minutes messing around with Firebug showed that it should work as written, although I could never get my breakpoints to fire off. (But as I haven't yet fully mastered Firebug, I put this down to my lack of experience)

As I sat there stumped, a vague sense of déjà vu crept over me, and I suggested to my friend that he rename update() to something very unique.

Which promptly resolved the issue. So, it was obviously a problem with namespace pollution... although I use the term 'namespace' liberally. Problem is, I can't replicate this using normal Javascript - that is, inline scripts will override external Javascript anytime.

So, that leaves one option - Prototype is extending HTMLElement or similar with an update() method. After a bit of digging, my suspicions are confirmed

In browsers that support adding methods to prototype of native objects such as HTMLElement all DOM extensions on the element are available by default without ever having to call Element.extend(), dollar function or anything! ...Because the prototype of the native browser object is extended, all DOM elements have Prototype extension methods built-in. This, however, isn't true for IE which doesn't let anyone touch HTMLElement.prototype.

That's one hell of a gotcha, and all to avoid using $()... I don't know if protecting HTMLElement.prototype is correct behaviour or not, but I'm now convinced that attaching your methods to it is very much incorrect.

The Prototype documentation on this 'feature' states:

This document reveals some clever hacks found in Prototype.

Clever? Very clever, yes. Hacks? Most definitely.

An unscientific comparison 3.5 - getting the cookie, Haskell

2007-06-04T19:56:00.000-07:00

Part 3.5 of N in a series. Click here for Part 3.4

Wow. This was incredibly hard.

Like, ludicrously hard. Nothing I've done before really prepared me for it. I thought I would be okay, I mean, pattern matching, no side effects? I loved that stuff in Erlang, bring it on! But then... then I met the typing system, and monads. Oh boy.

Before I get to the code, I'd like to thank Porges for his patient assistance with my struggling towards glimmers of comprehension. The file functions are his, as is the fact that my getCookie function works - he taught me a lot about monads in the process. :)


 1 module FirstCookie where
 2 
 3 import Network.HTTP
 4 import Network.Browser
 5 import Network.URI
 6 
 7 
 8 
 9 getCookie user pwd uri ua = do
10                             redir <- browse $ getCookie' user pwd uri ua
11                             browse $ getCookie'' redir
12 
13 getCookie' user pwd uri ua = do
14   setAllowRedirects False
15   let (Just parsed_uri) = parseURI uri
16   let f = Form POST parsed_uri [("id",user), ("pwd",pwd), ("dologin", "yes")]
17   (uri,resp) <- request $ formToRequest f
18   let (Just redirectLocation) = findHeader HdrLocation resp
19   let (Just parsed_redir_uri) = parseURI redirectLocation
20   return parsed_redir_uri
21 
22 getCookie'' uri = do
23                   request $ defaultGETRequest uri
24                   getCookies
25 
26 
27 --readCookie takes a file path and returns
28 -- a Cookie read from that file.
29 --
30 --read converts something from string representation
31 readCookie :: FilePath -> IO Cookie
32 readCookie f = do
33   -- read in the contents of the file (<-: extract string from IO monad)
34   contents <- readFile f
35   -- make a cookie from the contents
36   let cookie = read contents
37   -- return the cookie wrapped in the IO monad
38   return cookie
39 
40 --writeCookie takes a file path and a cookie and writes it
41 --
42 --show converts something to string representation
43 writeCookie :: FilePath -> Cookie -> IO ()
44 writeCookie f c = writeFile f (show c)

Comments

I found this unbelievably hard at first. Why? Two things, monads and typing.

Monads
According to Wikipedia, a monad is a "construction that, given an underlying type system, embeds a corresponding monadic type system into it"... which really explains it, no? There's a thousand metaphors to try and explain them.

In the course of writing these functions, I had dealings with the following monads: Maybe, BrowserAction and IO. Here's my understanding of each (which could be entirely wrong).

Maybe is a wrapper of sorts to allow polymorphic returns from functions that could fail, while retaining determinism. It seems well suited for chains of functions, where any function could fail and return Nothing. Maybe seems to remove the necessity for long manual chains of pattern matching and function calls. Maybe is also a bit like Schrodinger's Cat - until you look at it, you can't tell what the result was, hence 'extracting values' by using pattern matching to bind - that "(Just x)" stuff. Although I'm not sure quite what kind of entity Just is. Googling's not helping.

BrowserAction holds state for the Browser module - from the API documents, it looks like every stateful function returns an empty BrowserAction... not quite sure how it works, but at least I know how and where the BrowserAction is coming from.

IO is also stateful as far as I'm aware. I'm not sure how they achieve that, and would like to view the code they generate one day, like using macroexpand in Lisp, although I'm assured that it'll be a whole lot of ugly looking lambdas. But still, curiosity and all.

The biggest problem with monads? My preconceptions about monads. I've been prepped to think that monads are confusing and complicated by reading about how confusing and complicated they are. When in reality, they're merely different. The term "monad" doesn't help, sounds quite daunting and theoretical. I'm trying to substitute the term "wuzzle" in my thinking about them, to make them sound cute and fuzzy... but Maybe wuzzle sounds kind of silly.

But anyway, as is probably evident to any experienced Haskell hackers reading, I still have a very imperfect comprehension of how monads fit into the scheme of things, but they're not so scary now.

Typing
Going hand in hand with my lack of understanding about monads was troubles with types. But they tie in together, after all, a monad is simply a construction that, given an underlying type system, embeds a corresponding monadic type system into it... but yes, realising a monad is a type was one of those minor epiphany moments. My main issue seems to be understanding the type notation used. I guess it's a case of practise makes perfect. I'm also unused to using a static typing system where I'm not casting all over the show, makes me feel uneasy.

Documentation
I could've killed for some more indepth examples when I was writing this. I find API documents are great when you know what you're doing and just want to check you're passing the right arguments, but when you're trying to figure out how to do task X, API documents at best lend clues. A few examples and/or summary explaining the intended usage can be lifesavers.

Overall
Haskell feels like it's adhering to a theory (category theory?) - this manifests in behaviour that may not be pragmatic, but it feels consistent with itself - the rules that govern Haskell, whatever they are, make themselves known in the "texture" of the language - I'm afraid I can't explain it better than that, it's more of an intuitive thing. I imagine that using Haskell would reveal the nature of this theory in a manner a mathematical layman like myself can grasp... but I also feel that a grasp of the theory would make using Haskell a lot easier.

An unscientific comparison 3.4 - getting the cookie, Ruby

2007-06-04T16:38:00.000-07:00

Part 3.4 of N in a series. Click here for Part 3.3.2

Well, my first Ruby in a while, and it was certainly different to what I was expecting. I finally figured out why people like certain aspects of it, but other aspects left me cold.

But first, the code.


 1 require 'net/http'
 2 require 'uri'
 3 
 4 
 5 def getCookie(user, pwd, uri, userAgent)
 6  url = URI.parse(uri)
 7  request = Net::HTTP::Post.new(url.path)
 8 
 9  request.set_form_data({ "id" => user, "pwd" => pwd, "dologin" => "yes"})
10  request.add_field("User-Agent", userAgent)
11 
12  response = Net::HTTP.new(url.host).start {|http| http.request(request) }
13  cookieUrl = URI.parse(response.fetch("location"))
14 
15  cookieRequest = Net::HTTP.new(cookieUrl.host)
16  cookieResponse = cookieRequest.get(cookieUrl.path + "?" + cookieUrl.query,
                                     {"User-Agent" => userAgent})
17 
18  return cookieResponse.fetch("set-cookie")
19 end
20 
21 def storeCookie(cookie, filePath)
22  File.open(filePath, "w") do |f|
23   f.puts cookie
24  end
25 end
26 
27 def retrieveCookie(filePath)
28  f = File.open(filePath, "r")
29  cookie = f.gets
30  f.close
31  return cookie
32 end

Comments

Documentation
Maybe I'm looking in the wrong place, but Ruby documentation, to be frank, is terrible. The documentation that shipped with my Ruby install is out of date, and the online documentation is an API reference - but it's one of the most confusing API references I've ever used. For example, clicking on Net::HTTP::Post under the classes list is not terribly illuminating.

TMTOWDI
I have nothing against multiple ways to perform the same task - except when it leads to multiple partially implemented ways to perform the same task. For example, POSTing.

There is a method Net::HTTP.post which works almost identically to my usage of the get method above. Would've shortened the code considerably. Except, if I can quote the docs:

Posts data (must be a String) to path.

So, I can use the shorter and simpler method, but I need to convert the data into an urlencoded string myself before using it, or I can use the Request object which encodes the data for me, but has a more cumbersome method of dispatching that request.

URI class
It's normal to use a class to represent a URI, but is there a way to get my whole URI (including the query string I passed in originally) out? uri.path + "?" + uri.query feels a bit hackish.

Blocks
Blocks are awesome. Python needs to steal these asap, single line lambdas don't count. One thing I gained from this was an appreciation for blocks and what they're good for. (Also, symbols. Yay for symbols, not that I used them here.)

Overall
I actually might try and find a third party improvement on this lot. The thought of using Net::HTTP again is a tad demotivational. Documentation is terrible - but I suspect that's because for most of Ruby's life, it's been used mainly by Japanese speaking people, so hasn't developed the mature English documentation yet. To any Ruby documentation writers reading: "More examples please." I'm a sucker for examples.

Click here for Part 3.5

An unscientific comparison 3.3.2 - Getting the cookie, C# final

2007-05-25T20:57:00.000-07:00

Part 3.3.2 of N in a series - click here for Part 3.3.1

So, final version. After my embarrassing myself with my initial approach and associated tantrum, I thought I better do it in a slightly saner fashion - don't want to have eaten humble pie for nothing.


    1 using System;
    2 using System.Collections.Generic;
    3 using System.Collections.Specialized;
    4 using System.Text;
    5 using System.Net;
    6 using System.IO;
    7 using System.Runtime.Serialization;
    8 using System.Runtime.Serialization.Formatters.Binary;
    9 
   10 namespace firstCookie3
   11 {
   12     class CustomWebClient : WebClient
   13     {
   14         public event Action<WebRequest> ModifyRequest;
   15 
   16         public CustomWebClient() : base() { }
   17 
   18         protected override WebRequest GetWebRequest(Uri address)
   19         {
   20             WebRequest request = base.GetWebRequest(address);
   21             OnModifyRequest(request);
   22             return request;
   23         }
   24 
   25         private void OnModifyRequest(WebRequest request)
   26         {
   27             if (ModifyRequest != null)
   28                 ModifyRequest(request);
   29         }
   30 
   31     }
   32 
   33     class CookieFunctions
   34     {
   35         public static CookieCollection GetCookie(string user, string pwd, 
                                                   string uri, string ua)
   36         {
   37             NameValueCollection postData = new NameValueCollection();
   38             CookieContainer ckCont = new CookieContainer();
   39             CustomWebClient client = new CustomWebClient();
   40 
   41             postData["id"] = user;
   42             postData["pwd"] = pwd;
   43             postData["dologin"] = "yes";
   44             client.ModifyRequest += delegate(WebRequest req)
                                 {
                                 ((HttpWebRequest)req).CookieContainer = ckCont;
                                 };
   45             client.ModifyRequest += delegate(WebRequest req)
                                      {
                                      ((HttpWebRequest)req).UserAgent = ua;
                                      };
   46             client.UploadValues(uri, postData);
   47 
   48             return ckCont.GetCookies(new Uri(uri));
   49         }
   50 
   51         public static void StoreCookie(CookieCollection ck, string filePath)
   52         {
   53             IFormatter formatter = new BinaryFormatter();
   54             Stream outStream = new FileStream(filePath,
   55                                               FileMode.Create,
   56                                               FileAccess.Write,
   57                                               FileShare.None);
   58             formatter.Serialize(outStream, ck);
   59             outStream.Close();
   60         }
   61 
   62         public static CookieCollection RetrieveCookie(string filePath)
   63         {
   64             IFormatter formatter = new BinaryFormatter();
   65             Stream iStream = new FileStream(filePath,
   66                                             FileMode.Open,
   67                                             FileAccess.Read,
   68                                             FileShare.Read);
   69             CookieCollection ck = (CookieCollection)formatter.Deserialize(iStream);
   70             return ck;
   71         }
   72 
   73 
   74     }
   75 }

The only comment I have to make is this - I'm glad they left that little hook there for me, but what worries me is the fact that I'm dependent on them doing so. I now understand what it's like to work in a B&D language.

Click here for Part 3.4

An unscientific comparison 3.3.1 - Getting the cookie, C# - a better way

2007-05-25T07:49:00.000-07:00

Part 3.3.1 of N in a series. Click here for Part 3.3.

Well, after much moaning and complaining and cursing of Bill Gates, I've decided that I need to update the C# attempt. Turns out, there's an easy way in .NET, and there's the horribly hard way.
The easy way is to work with the library. The hard way is to to brute force it. Guess what my category my last attempt fell under.

But let me start by saying that I probably would've left it at that if not for the patient guidance of these gentlemen - linkdown, porges, and a fellow by the nick of Kanibiss. So thanks guys.

So, a dissection of where I went wrong... well basically, it was the dismissal of System.Net.WebClient as being "too high level"... ah, hubris, my eternal enemy. Now I knew that to collect cookies I'd need to add a CookieContainer to an HttpWebRequest; my error was in assuming that WebClient didn't expose this in any way that I could use.

WebClient contained all the functionality I had replicated in my HttpWebRequestWrapper - URI encoding, setting of headers, etc, butI just couldn't figure out how to set a CookieContainer and without doing so, cookies aren't retained.

I had actually seen the method WebClient.GetWebRequest in the documentation, but had dismissed it - after all, it only returned a WebRequest object, it didn't return a reference to the WebRequest object being used by WebClient. The documentation contained a code sample showing exactly what I needed to do, but my inexperience tripped me up here and I totally missed the import of it.

After linkdown kept prompting me to investigate further, I downloaded Reflector and had a closer look at WebClient. Specifically, the UploadValues method, as this was the one I would like to have used... and lo and behold, the way was revealed.


WebRequest request = null;
this.ClearWebClientState();
try
{
    byte[] buffer = this.UploadValuesInternal(data);
    this.m_Method = method;
    request = this.m_WebRequest = this.GetWebRequest(this.GetUri(address));
    this.UploadBits(request, null, buffer, null, null, null, null);
    byte[] buffer2 = this.DownloadBits(request, null, null, null);
    if (Logging.On)
    {
        Logging.Exit(Logging.Web, this, "UploadValues", address + ", " + method);
    }
    buffer3 = buffer2;
}

The relevant bit is the call to... GetWebRequest. Ah. It was then that I noted that GetWebRequest was designated as virtual - that is, as overrideable by subclasses of WebClient.

A minor epiphany occurred - the designers of this class had left me a hook, as it were, but I was too inexperienced to see it. I felt very stupid and elated at the same time.

I quickly sat down and hacked out some code to use it, and then deleted it and rewrote it to allow for generalised usage - my first attempt looked like this -


    1 using System;
    2 using System.Collections.Generic;
    3 using System.Text;
    4 using System.Net;
    5 
    6 
    7 namespace firstCookie2
    8 {
    9     public delegate void ModifyRequest(WebRequest w);
   10 
   11 
   12 
   13 
   14     class CustomWebClient : WebClient
   15     {
   16         Queue<modifyrequest> DelegateQueue;
   17 
   18         public CustomWebClient() : base()
   19         {
   20             DelegateQueue = new Queue();
   21         }
   22 
   23         protected override WebRequest GetWebRequest(Uri address)
   24         {
   25             WebRequest request = base.GetWebRequest(address);
   26             ApplyDelegates(request);
   27             return request;
   28         }
   29 
   30         private WebRequest ApplyDelegates(WebRequest request)
   31         {
   32             while (DelegateQueue.Count > 0)
   33             {
   34                 ModifyRequest modDelegate = DelegateQueue.Dequeue();
   35                 modDelegate(request);
   36             }
   37             return request;
   38         }
   39         public void AddDelegate(ModifyRequest del)
   40         {
   41             DelegateQueue.Enqueue(del);
   42         }
   43 
   44     }
   45 }

After some sage advice from this fine fellow, I modified the class to use events:


   26 class CustomWebClient : WebClient
   27 {
   28     public event Action<WebRequest> ModifyRequest;
   29 
   30     public CustomWebClient() : base() { }
   31 
   32     protected override WebRequest GetWebRequest(Uri address)
   33     {
   34         WebRequest request = base.GetWebRequest(address);
   35         OnModifyRequest(request);
   36         return request;
   37     }
   38 
   39     private void OnModifyRequest(WebRequest request)
   40     {
   41         if (ModifyRequest != null)
   42             ModifyRequest(request);
   43     }
   44 
   45 }

Which can then be used like so:


static void Main(string[] args)
{
    CustomWebClient w = new CustomWebClient();
    CookieContainer cj = new CookieContainer();
    w.ModifyRequest += delegate(WebRequest request)
    {
        ((HttpWebRequest)request).CookieContainer = cj;
    };
    w.ModifyRequest += delegate(WebRequest request)
    {
        ((HttpWebRequest)request).AllowAutoRedirect = true;
    };
    string v = w.DownloadString("http://www.google.co.nz");
}

So. Now I get to rewrite the rest of my C# piece. I'm very embarrassed at my mistake, but I learnt a lot about .NET in the process. I also have a far better vehicle for my HTTP requests now, far more flexible and far more powerful. My thanks to all who gave advice. :)

Click here for Part 3.3.2

An unscientific comparison 3.3 - Getting the cookie, C#

2007-05-20T05:58:00.000-07:00

Part 3.3 of N in a series. Click here for part 3.2.

Wow, this was unexpectedly unpleasant. After a few months of using C#, I'd grown quite fond of it and .NET - that fondness got a battering tonight.

And all I wanted to do was subclass System.Net.HttpWebRequest. But first, the code.

    1 using System;

    2 using System.Collections.Generic;

    3 using System.Text;

    4 using System.Net;

    5 using System.Web;

    6 using System.IO;

    7 using System.Runtime.Serialization;

    8 using System.Runtime.Serialization.Formatters.Binary;

    9 using StringHashTable = System.Collections.Generic.Dictionary<string, string>;

   10 using StringList = System.Collections.Generic.List<string>;

   12 namespace firstCookie

   13 {

   14     class HttpRequestWrapper

   15     {

   16         HttpWebRequest request;

   17         StringHashTable postData;

   19         public HttpRequestWrapper(string uri)

   20         {

   21             request = (HttpWebRequest)WebRequest.Create(uri);

   23         }

   25         public HttpWebResponse Post()

   26         {

   27             if (request.Method != "POST") request.Method = "POST";

   29             /*Prepare data - If I haven't actually set the data,
                  then I deserve an exception*/

   30             byte[] preppedData = prepareData(postData);

   31             request.ContentType = "application/x-www-form-urlencoded";

   32             request.ContentLength = preppedData.Length;

   34             //Now POST

   35             Stream postStream = request.GetRequestStream();

   36             postStream.Write(preppedData, 0, preppedData.Length);

   37             return (HttpWebResponse)request.GetResponse();

   38         }

   40         private byte[] prepareData(StringHashTable postData)

   41         {

   42             //Firstly, convert data to usual query string form.

   43             string formatString = "{0}={1}";

   44             StringList pairsList = new StringList();

   45             foreach (string key in postData.Keys)

   46             {

   47                 string encodedKey = HttpUtility.UrlEncode(key);

   48                 string encodedValue = HttpUtility.UrlEncode(postData[key]);

   49                 string encodedString = String.Format(formatString,
                                                         encodedKey,
                                                         encodedValue);

   50                 pairsList.Add(encodedString);

   51             }

   52             string queryString = String.Join("&", pairsList.ToArray());

   54             /*Now, convert string to byte[] - it's needed in this format,
                  can't quite say why.*/

   55             ASCIIEncoding byteEncoder = new ASCIIEncoding();

   56             byte[] encodedData = byteEncoder.GetBytes(queryString);

   57             return encodedData;

   58         }

   60         public StringHashTable formData

   61         {

   62             //Futureproof this, my Python side just wants to make postData public.

   63             get { return postData; }

   64             set { postData = value; }

   65         }

   67         public string Method

   68         {

   69             get { return request.Method; }

   70             set { request.Method = value; }

   71         }

   73         public string UserAgent

   74         {

   75             get { return request.UserAgent; }

   76             set { request.UserAgent = value; }

   77         }

   79         public CookieContainer CookieContainer

   80         {

   81             get { return request.CookieContainer; }

   82             set { request.CookieContainer = value; }

   83         }

   85         public bool AllowAutoRedirect

   86         {

   87             get { return request.AllowAutoRedirect; }

   88             set { request.AllowAutoRedirect = value; }

   89         }

   91     }

   93     class RequestFunctions

   94     {

   95        public static HttpRequestWrapper CreateRequest(string uri, string userAgent)

   96         {

   97             HttpRequestWrapper req = new HttpRequestWrapper(uri);

   98             req.UserAgent = userAgent;

   99             req.Method = "GET";

  100             return req;

  101         }

  103         public static HttpRequestWrapper CreateRequest(string uri,
                                                           StringHashTable data, 
                                                           string userAgent)

  104         {

  106             HttpRequestWrapper req = CreateRequest(uri, userAgent);

  107             req.Method = "POST";

  108             req.formData = data;

  109             return req;

  110         }

  113     }

  117     class CookieFunctions

  118     {

  119         public static CookieCollection GetCookie(string user, 
                                                     string pwd, 
                                                     string uri,
                                                     string ua)

  120         {

  121             StringHashTable postData = new StringHashTable();

  122             postData["id"] = user;

  123             postData["pwd"] = pwd;

  124             postData["dologin"] = "yes";

  126             //Create request and prepare Data

  127             HttpRequestWrapper request = RequestFunctions.CreateRequest(uri,
                                                                            postData,
                                                                            ua);

  129             //Last tweaks to the request

  130             CookieContainer ckCont = new CookieContainer();

  131             request.CookieContainer = ckCont;

  132             request.AllowAutoRedirect = true;

  133             HttpWebResponse response = request.Post();

  134             return ckCont.GetCookies(new Uri(uri));

  135         }

  137         public static void StoreCookie(CookieCollection ck, string filePath)

  138         {

  139             IFormatter formatter = new BinaryFormatter();

  140             Stream outStream = new FileStream(filePath,

  141                                               FileMode.Create,

  142                                               FileAccess.Write,

  143                                               FileShare.None);

  144             formatter.Serialize(outStream, ck);

  145             outStream.Close();

  146         }

  148         public static CookieCollection RetrieveCookie(string filePath)

  149         {

  150             IFormatter formatter = new BinaryFormatter();

  151             Stream iStream = new FileStream(filePath,

  152                                             FileMode.Open,

  153                                             FileAccess.Read,

  154                                             FileShare.Read);

  155             CookieCollection ck = (CookieCollection)formatter.Deserialize(iStream);

  156             return ck;

  157         }

  163     }

  166 }

Comments

You take the low road, and I'll take the high road...
.NET offers two ways of retrieving remote websites via HTTP. System.Net.HttpWebRequest and System.Net.WebClient. WebClient is very high level and handy for simple tasks; unfortunately, storing cookies isn't something it can do. This is where HttpWebRequest comes in.
It can perform the tasks needed, but not overly well. To be honest, I was a bit appalled at how limited HttpWebRequest was.

Url encoding query strings
HttpWebRequest can't do this. The only functionality for encoding uri strings is located in System.Web.HttpUtility.UrlEncode - System.Web is aimed at server side ASP.NET more than client side stuff like this and requires that you add a reference to the assembly. It's somewhat limited - it can only encode a string - so passing in mappings of names to values and getting an encoded string returned is not possible.

Sending the POST data
Once again, HttpWebRequest doesn't know how to do this. I'm sure there are very good reasons why not, but it's a bit frustrating. So, once I've taken my data, and uri encoded it, I then have to convert it to a series of bytes to POST to the server.

Subclassing HttpWebRequest
And this is where I decided I was going to subclass. All I wanted to add was a method to encode a generic Dictionary passed to it, and a method to handle the minor details of actually POSTing the data. And that's when I learnt about the internal keyword.
HttpWebRequest has multiple constructors. The one I needed to call was marked internal - that is, it can only be called by code residing in the same assembly. One of the constructors is accessible, but it's an obsolete one that provides no required functionality. I ended up going with the wrapper approach - which feels like an ugly hack, but will save me from replicating the same code over and over as I proceed.

Overall
I was quite disappointed with HttpWebRequest in some places - in others, I was pleased, for example, AllowAutoRedirect is a nice touch, in Python's urllib2, for example, you have no choice. Serialising objects was straightforward, and the documentation, overall, was reasonably thorough. Some more relevant examples would always be nice, but that's MSDN for you.

I'm still not happy about having to wrap a class when subclassing it would've been a lot simpler and neater - it's a bit of a horrible hack, because each time I need another property of HttpWebRequest, I'll have to add another property to HttpRequestWrapper.

Click here for Part 3.3.1

An unscientific comparison 3.2 - Getting the cookie, Scheme

2007-05-18T22:09:00.000-07:00

Part 3.2 of N in a series. Click here for part 3.1.

I must admit, writing this code took me the best part of an evening - mainly because of my inexperience in Scheme. I'm fine with prefix notation and the parentheses for Africa, it's merely that I lack the background knowledge to use Scheme properly - the background knowledge of idioms and usage patterns that one tends to acquire by osmosis. I'd like to comment that the residents of #Scheme on the Freenode IRC network are incredibly helpful, and very patient with dumb newbie question.

So, the code using Felix Winkelmann's http library for Chicken Scheme.


 1 (require-extension http-client)
 2 (require-extension srfi-13) ;string functions
 3 (require-extension srfi-8) ;receive statement
 4 (require-extension srfi-16) ;case-lambda
 5 
 6 (define create-request
 7   (case-lambda
 8     ((method uri user-agent) (http:make-request method
                          uri 
                          `(("User-Agent" . ,user-agent))))
 9     ((method uri data user-agent)
10      (let ((encoded-data (uri-encode-query data)))
11        (http:make-request method
12                           uri
13                           `(("User-Agent" . ,user-agent)
14                             ("Content-Type" . "application/x-www-form-urlencoded")
15                             ("Content-Length" . ,(string-length encoded-data)))
16                           encoded-data)))))
17 
18 
19 (define (get-cookie user pwd uri user-agent)
20   (let* ((data `(("id" . ,user) ("pwd" . ,pwd) ("dologin" . "yes")))
21         (auth-req (create-request 'POST uri data user-agent)))
22     (receive (a header-list b c) (http:send-request auth-req) ;Request A
23              (let ((cookie-req  (create-request 'GET
24                                                 (alist-ref "location"
                                                          header-list 
                                                          equal?)
25                                                  user-agent)))
26                (http:GET cookie-req) ;Request B
27                (alist-ref "Cookie" (http:request-attributes cookie-req) equal?)))))
28 
29 
30 (define (store-cookie cookie file-path)
31   (let ((f (open-output-file file-path)))
32     (write cookie f)
33     (close-output-port f)))
34 
35 
36 (define (retrieve-cookie file-path)
37   (let* ((f (open-input-file file-path))
38          (cookie (read f)))
39     (close-input-port f)
40     cookie))

Comments

Documentation
The documentation for the http library covers all the bases - my only quibble would be a lack of examples, but that's a tiny quibble that arises only from my inexperience.

Data structures

I had a bit of hassle with using alist s - the default predicate for equivalence used by the built-in function for alist item retrieval, assv, is eqv? - which returns false for "test" == "test". Chicken Scheme comes with some extra functions that I'd most likely end up writing if using R5RS Scheme, one of which is alist-ref, which allows me to specify that the predicate to be used is equal? which returns true for "test" == "test". (And also for "TEST" == "test", this could be problematic in certain situations, but is fine here) Learning how to compose an alist inline was interesting, backquoting and comma prefixing all over the show.

File IO
Reading and writing to/from a file was relatively painless compared to my experiences in Lisp.

SRFIs
The SRFIs are great, but a little obscure at first. It's become clear to me that I'll need to become familiar with their contents if I'm to make effective use of Scheme.

Overall
Using the http library ended up being a little lower level than I had anticipated, mainly because my usage fell outside of the common usages for which convenience functions existed - for instance, instead of building a request for posting and setting Content-Length headers myself, I could've used http:POST, which would've done all of that, but it only returns the body of the response, when I needed the header to follow the redirect. (Request A returns a 302 response, request B actually gets the cookie.) I did note that http:POST doesn't urlencode the datapassed to it, however, and I had to write a wee socket server in Erlang to confirm that. Not a bug, just something to note.

I learnt a fair bit about Scheme in doing this, and the support from the Scheme IRC residents on Freenode, both #Scheme and #Chicken, was fantastic. I quite enjoyed myself, and this is exactly why I chose this method - "sink or swim" seems to be the best way to learn a language and its environment quickly.

Click here for Part 3.3

An unscientific comparison 3.1 - Getting the cookie, Python

2007-05-15T18:49:00.000-07:00

Part 3.1 of N in a series. Click here for Part 3.

So, without further ado, here's some actual code. I've started with Python as it's my 'default' language.

 1 import urllib
 2 import urllib2
 3 import cookielib
 4 
 5 def getCookie(user, pwd, uri, ua):
 6     ckCont = cookielib.LWPCookieJar()
 7     opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(ckCont))
 8     opener.addheaders = [('User-Agent', ua)]
 9     data = urllib.urlencode({'id':user, 'pwd': pwd, 'dologin': 'yes'})
10     opener.open(uri, data)
11     return ckCont
12 
13 def storeCookie(ck, file_path):
14     ck.save(file_path)
15 
16 def retrieveCookie(file_path):
17     ckCont = cookielib.LWPCookieJar(file_path)
18     ckCont.load()
19     return ckCont

Comments

Duplicate libraries
This is one of the messier areas of the stdlib in my mind, and I'm hoping it gets overhauled for Python 3000. If it's not currently planned, I should probably volunteer. Both urllib and urllib2 have similar functionality with regards to the HTTP protocol, but only urllib2 has the capacity to handle cookies and proxies sanely. The only thing urllib is really necessary for (in my experience) is urlencode.

Documentation
The documentation for urllib2 is... sparse. Some more examples would've been brilliant, as would've some clarification of what needs to go with what. For instance, I'm assuming a proxy requiring login will need a ProxyHandler and a BasicProxyAuthHandler, but such (I'd assume) common usages aren't commented on. Perhaps something else I should volunteer to correct.

Edit: A commenter pointed out these two great resources: urllib2 - The Missing Manual and cookielib and ClientCookie. Seems someone else is way ahead of me in that respect.

Out-and-out broken libraries
I had to upgrade to Python 2.5.1 to make this work, as urllib2 in Python 2.4 is apparently broken when trying to use cookie handlers, and my experiences with it anecdotally confirm that. First time I've ever encountered that in Python...

Minor quibble
opener.addheaders - phrased like a method, it's actually a property, a list of headers to add. I know it's more verbose, but perhaps it should be called headersToAdd.
Edit: I did have another quibble here about CookieJars not being serialisable, but with LWPCookieJar, I don't need to serialise anything.

Overall
urllib2 is quite an indepth library, and handles moderately complex behaviour reasonably well, especially when the ability to 'chain' handlers to build a custom opener is used - as I've done with the cookie handler. It also automates some of the mucking around, which is always nice, although it does add a trace of 'magic' to the code. For example, in the above code, there are actually two requests made - if the initial POST is successful, a HTTP 302 (Object moved) response is returned, with a header pointing to a new URI which sets the authentication cookie. Urllib2 sends the GET to this new location without prompting.

While the documentation is a bit bare, it's still more fleshed out than an auto-generated API document. That said, needs work, needs work.

The code feels reasonably concise, and fairly self-documenting. The only part that could be confusing is how the cookie is getting into the CookieJar. (opener stuck it there).

Click here for Part 3.2, the Scheme version

An unscientific comparison 3 - The first cookie

2007-05-13T03:17:00.000-07:00

Part 3 of N in a series, click here for part 2.

As the web app will require I use a cookie to authenticate, the first step is to retrieve from the server and store that cookie.

Today, I want to write three functions - one that creates the cookie by logging in, one that stores it locally, and one that retrieves it. I won't be using the third one just yet, but it seems logical to get all the cookie related stuff out of the way. I'm keeping error handling simple - generally speaking, if one of these functions fails, it should let the calling context handle the failure.

So, the spec. Incidentally, cookie can either be a single cookie or some kind of container object holding a cookie, I'm easy in that regard.

getCookie(user, password, uri, userAgent) returns cookie
This POSTs the urlencoded user and password to the given uri, and sets the User-Agent header to the given userAgent if possible. The cookie from the HTTP response will be returned.

storeCookie(cookie, filePath)
This stores the cookie in the given file. How the cookie is stored isn't specified, I prefer object serialisation, but if that's not possible, I can live with a text file. Nothing needs to be returned.

retrieveCookie(filePath) returns cookie
Opens the given file, and returns the cookie. If the file doesn't exist, then any exception or error should be handled by the calling context.

I will add each implementation of this as a separate entry.

Implementations

Part 3.3.2 - C#, the right way

An unscientific comparison 2 - First steps

2007-05-10T05:35:00.000-07:00

Part 2 of N in a series. Click here for part 1.

So, first things first, let's download and install some programming languages.

For 'monolithic' languages, that is, languages with one main distribution, I'll be using that. For languages with several available implementations, I'll choose the particular implementation based on several factors:

Is it free?
Does it work on Windows XP? (My current OS)
Does the standard library support HTTP out of the box?

The HTTP requirement is a simple one - I feel that any modern programming language should support the most common protocols as a matter of course. Also, I'm not looking to reimplement any RFCs here.

How the installs compared

Python
All you really need to know to get started with Python is www.python.org - the page contains documentation, official installation packages and links to unofficial packages such as ActiveState's ActivePython. At this point in time, when people think of Python, they're thinking of CPython - although as Microsoft's DLR becomes widespread, IronPython, as the flagship/testbed dynamic language may become more well known. Jython remains relatively obscure, and I can't foresee that changing. As such there's not likely to be much problem choosing CPython over the other two implementations - that said, I'm guessing a little, I know Python too well to accurately model a newbie.

The only issue could be if you ignored the prominent links to download Python 2.5.1 on the main page and went digging through the Downloads page, you could find the link to ActivePython which could raise some questions as to what precisely to install - noting that ActivePython's page is pretty clear that it only contains some additional libraries and documentation.

Once I'd downloaded my distribution (I actually went with ActivePython because I prefer its REPL), installation was simply a case of executing the installer.

Scheme (Chicken)
Getting started with Scheme takes a bit more research. As I'd become interested in Scheme after reading SICP (and it's more easily accessible counterpart, How To Develop Programs), I began experimenting with PLT's DrScheme. However, DrScheme supports several different variations on Scheme - standard R5RS Scheme, MzScheme, a fuller featured implementation and MrEd, a graphical implementation. I wasn't quite sure which one of these I should use and to make a decision, I had to learn about how Scheme is 'managed' so to speak. I had initially assumed Scheme was monolithic like Python, but I soon found out I was wrong.

I learnt that the core Scheme language was defined in the standard R5RS, but that R5RS Scheme was incredibly minimalistic, with most additional functionality being provided by the different implementations. So I started looking at the various options.... and I found this list of Scheme implementations...

...I had no real way of assessing each implementation on it's merits, so I chose mainly on the standard library. The PLT Schemes supported sockets but not HTTP, ditto Bigloo Scheme, but Chicken's "Eggs Unlimited" included an http module. So Chicken it was.

Installing Chicken was a bit different, me being the Windows user that I am, as it was only available as source code. The first step in configuring the install was to download and install CMake, a cross platform makefile generator. I initially used this to generate a Visual Studio project, which didn't compile... so I downloaded MinGW, and created makefiles for that. This time, the compilation went flawlessly. The hardest part was figuring out which MinGW file to download, but after a bit of trial and error, I figured it out, and even wrote a page on the Chicken wiki, for any other C newbies like me.

C#
I already had Visual Studio installed for ASP.NET development, so everything I needed was included.

Ruby
Determining what I needed was straightforward. JRuby is approaching 1.0 release, I believe, not that I have much interest in the JVM, and IronRuby is, for all intents, still a twinkle in Jim Huginin's eye. So, downloading and running the Windows installer from the Ruby page is all I need.

Haskell
Haskell has multiple implementations available, and after ten minutes on Wikipedia, I'd decided that either Helium or Hugs would be best. The question is, which one? Helium offers friendlier error messages apparently, but at the expense of some functionality - for instance, no type classes. Will I need a type class in this? After some further reading on Wikipedia and on the Helium page... I still don't know. I've decided to try Hugs first, and if I have any issues figuring out what I'm doing wrong, I'll try Helium then.

Installing Hugs just requires I download and run an installer.
Edit: It has been suggested by a commenter that I stick with GHC; I shall take that advice, and have downloaded Visual Haskell, an alpha plugin for Visual Studio that ships with GHC.

Common Lisp
Common Lisp is like Scheme, in that there's a standard, and then several different implementations of that standard. After reading Cliki and ALUwiki and applying a couple of criteria (works on Windows, I don't have to buy it) I have a a narrow list of possible implementations - SBCL, LispWorks, Allegro, CLISP. After much researching of libraries etc. I've gone with LispWorks, mainly because the HTTP module I'm going to use, Drakma, was written for LispWorks and requires slightly less jiggery-pokey to get going. Also, the REPL is available within LispWorks, so no more cmd.exe REPLs or Emacs buffers! I'm quite pleased about that to be honest.

Installing Drakma actually took a little more work. Most of it revolving around ASDF. ASDF is a packaging system for Lisp. Now, there's a simpler interface to it called ASDF-INSTALL, but it's *nix-centric, and I'm not keen to go down the Cygwin path just yet. I was stumped, to be honest. LispWorks didn't ship with ASDF, so the first step was to install it. By this stage, I'd thoroughly confused myself with the ASDF-INSTALL documentation, and had given up on being able to get ASDF up and running... I was experimenting with RDNZL, a wrapper between Common Lisp and .NET, my intention being to use .NET's WebRequest and WebResponse instead.

Luckily, I'd started having a browse through the website of Edi Weitz, the author of both Drakma and RDNZL, and found his STARTER-PACK, specifically designed for Windows using newbies like me. It installed and configured ASDF for me, but also, it installed Dr Weitz's LW-ADD-ONS, which provided a simple and direct interface to control ASDF from the REPL. This gave me the breathing space I needed to learn how to use ASDF directly. For example, to load Drakma into a Lisp program, you call the following:


(asdf:oos 'asdf:load-op :drakma)

I am incredibly glad I found Dr Weitz's quality packages, as my progress would've been both painful and slow otherwise. Once I had the hang of ASDF, I could use it to install other ASDF packages I downloaded.

Erlang
Erlang is a monolithic language with only one distribution that I'm aware of, so I had no issues here. Again, just a case of downloading and executing a MSI installer.

Prolog
Prolog has several implemenations that have compatibility issues, a la Common Lisp, but the three most widely used seems to be SWI-Prolog, GNU Prolog, and Ciao Prolog. I chose SWI because it had an extensive standard library and thorough documentation - I also prefer the LGPL over the GPL for entirely irrational reasons.

Javascript & XPCOM
I'd already decided to use Firefox for this because I knew it well, I didn't need the rest of the Seamonkey package (IIRC, it's a modern version of the old Mozilla Suite), and disliked the ideological reasons for the creation of IceWeasel. The only real question was, which version to use? I went with Firefox 2.0 because it supports Javascript 1.7, which has some nifty new features, like list comprehensions, let blocks, and most importantly in my mind, destructuring bind.

Rebol
The only question was - do I download Rebol/Core the basic distribution, or Rebol/View which contains GUI libraries in addition to the core? In the end I went with Rebol/Core, because I wasn't going to need the GUI aspect of it, and hey, it's a 224KB download, which fascinates me - the documentation package is larger than the language!

IronPython
There's a couple of ways to use IronPython - it's available as a standalone download, or you can install the Visual Studio SDK which gives VS support for IronPython; I've chosen this course as it makes for easier editing.

Conclusion

For the most part, everything was straightforward, but there's some notable exceptions.

Finding the right implementation of Scheme is a complex process, there's so many of them it's not funny. Compiling Chicken took me a few days to sort, due to the VS build failing, but once I had MinGW sorted, it went fine - my Windows background feels sort of affronted when a precompiled binary isn't available, but I know that if I used *nix I'd be used to compiling from source.

Getting the Common Lisp environment set up was quite painful. If it hadn't been for Edi Weitz's starter pack, I don't think I would've been able to get off the ground - a side effect of Common Lisp's multiple implementations is that documentation is somewhat fragmented. That said, I've found the legendary newbie-unfriendliness of comp.lang.lisp's community to be a complete myth - as long as you don't try to tell them how to 'fix' Common Lisp.

Click here for Part 3.

An unscientific comparison of programming languages

2007-05-06T19:51:00.000-07:00

Part 1 of N in a series.

I've been playing with a bunch of new languages and paradigms recently, which is always fun, but while tutorials are helpful and all, I've always been a proponent of the deep end approach - pick a real life problem and try to resolve it.

Luckily, I happen to have a problem to resolve.

The Problem - Scraping a web application

The application in question consists of a series of six dynamic pages. Each page presents several options to the user as radio buttons. I need to collate both the set of options, and the context of that set - i.e., option A selected on the first page will produce a different set of options on the second page to option B, and option N on the first page, if selected, will skip to the fourth page, with a different set of options.

So, I'm mapping the possible paths through the application, and the associated data. As far as scraping goes, this is more complex than I've done before, but I've already written a (very rough) prototype in Python, which produces nearly correct output that a little hand correction is needed in.

In other words, it's a crude hack, and I need something that's a bit more robust, hence the rewrite.

The Methodology

As I said, this is an unscientific comparison from which no conclusions can be reliably drawn. The languages are being chosen simply based on my having an interest in them, and some of them will obviously be better suited for this problem domain than other.

I'm not looking for a superior language per se, I'm just going to document my journey into places unknown. Or known very poorly.

What I'm particularly interested in is:

Ease of obtaining and installing
Documentation
Breadth of any standard library
Availability of 3rd party modules

As libraries are going to play such a large part in this, I've decided to keep it as simple as possible, I'm a newbie (or near newbie) in most of these languages, so I'm going to follow this rule:

If a given module which suits my needs exists in the standard library or is bundled with the language, I will use it.

If I can't find a necessary module in the standard library, I'll use Google to find suitable 3rd party libraries. With the above said, I'm not going to go overly silly on it. While Python's standard library contains a basic html parser (very basic, check the docs for it), I'm not going to reinvent the wheel, I'll just use BeautifulSoup.

This will also help me manage the complexity... for example, there's a bunch of fantastic looking third party scraping frameworks for Ruby, but I could spend a month simply deciding which one to use! So, I'll use Ruby's standard library HTTP module, and Hpricot, which is bundled with my Ruby install.

The languages

Python
Python is the language I've been using the longest, it's like the comfy sweatpants you never throw out, because they fit you so well. It doesn't fill me with zealousness like it used to (A jihad on Perl and the Wallites!), but I still enjoy using it. As such, I can write a baseline app in it without the language getting in the way.

Scheme
Attempting to read SICP has instilled in me a desire to actually get a handle on Scheme. I chose Chicken Scheme because it has a coherent collation of standard/3rd party libraries should I need them. Also, Scheme that compiles to C sounded nifty.

C#
I've used this a bit for ASP.NET, but I've never used it in this context, and it'll provide a chance to examine bits of the .NET library that I haven't used before. Also, I needed a token "enterprisey" language. :-)

Ruby
As a Python user, I've been evangelised repeatedly to about the joys of Ruby. I've read the wry cartoon introductions, and flamed the zealots, so it's about time I gave it an indepth test run. Who knows, those fervent young fellows could be right...

Haskell
What can I say? I read Reddit too much to not try Haskell out. I'm also hoping by the end of this I'll grok just what the hell a monad is. The type system intrigues me also.

Common Lisp

SICP gave me a hankering for Scheme, Paul Graham's On Lisp did the same for Common Lisp.

Erlang
I've been wanting to give Erlang another spin around the block for a while now. Previously I'd written abour 2% of an MSN client in it, so I have a rough idea of how to use it. Erlang, along with Haskell, represents the 'pure' functional languages, as opposed to Scheme and Lisp. Erlang also brings the message passing concurrency paradigm to the party, which I'm going to try and use to speed up the scraping a little.

Prolog
Someone once told me, "I always enjoyed programming in Prolog, you never quite knew how the program was going to turn out." So, I've got to try it at least once. It'll also be interesting to learn a bit about logical programming.

Javascript + XPCOM
A friend of mine reckons it's possible to do so, so I'll give it a shot. I'll use Firefox as the hosting engine, as I'm going to need XPCOM to write data to disk. From initial pokings, this one is going to be a lot harder than you'd think.

Rebol
From what little I know, Rebol was designed for this kind of problem domain. If so, I relish the chance to learn it, as it could prove to be an invaluable tool. Rebol's implementation of certain data types like dates is pretty nifty, and would be reason enough alone to have a go.

IronPython
I've been meaning to use this for something, so here's my chance. Instead of just replicating the C# functionality with a cleaner syntax, I'm going to try to use a mixture of .NET and CPython libraries, to see how well it works.

Conclusion

I'm quite looking forward to this, it should knock a few of the cobwebs out of my brain, and hopefully I'll end up with a robust webscraper at the end of it. I reiterate that the choice of languages is purely for my own satisfaction. I'd further reiterate that no language is going to "win" - this isn't one of those closeted "My Programming Language is superior to all others!" blog posts you see from time to time, and anyone who attempts to derive such conclusions from this is misled.

Click here for Part 2

Compare in Erlang

2007-04-02T04:03:00.000-07:00


compare([], _)->
    false;

compare(_, [])->
    true;    

compare([_|X], [_|Y])->
    compare(X, Y).

Static typing eh? Not that verbose eh?

2007-04-01T20:09:00.000-07:00


using System.Collections.Generic;

...

public void OhFun()
{
   List<int> example = new List<int>();
   example.Add(1);
   ...
   example.Add(10);
   string template_string = "{0} - {1} - {2}...{9}";
   Converter<int, object> conv = new Converter<int, object>(delegate(int in_var){return (object)in_var})
   object[] template_args = Array.ConvertAll<int, object>(example.ToArray(), conv);
   string final_string = String.Format(template_string, template_args);
}

Oh my, and certain people have trouble with the statement that "...dynamic typing [can] result in a reduction in code volume of 80 to 90 percent? It seems highly improbable to me."

Strange thing is, despite the method signature specifying an object array, it's okay to pass a string array - perhaps it's being implicitly coerced into being an object array?

Oh, and the dynamic variant of this code?


def OhFun():
    example = []
    example.append(1)
    ...
    example.append(10)
    template_string = "%d - % d - %d...%d"
    final_string = template_string % tuple(example)

Not 80% in my example, but nonetheless...

Python doesn't feel like a very functional language

2007-04-01T18:34:00.000-07:00

I'm a big Python fan. It's what I learnt to program in, after all.

Even with my recent drinking of the .NET Kool Aid I've managed to bring Python along, in the form of IronPython integrated into Visual Studio - especially the REPL that I use to prototype functionality and poke new .NET objects. Sure, I could read the docs, but poking the class in question directly has always been more intuitive.

But, I have to admit, sure, it may have lambda, map and reduce (at least until Python 3000), and it may have list comprehensions, but it doesn't feel functional.

I blame Paul Graham

Everytime I start reading his book, On Lisp I start getting all excited about Lisp. It's his passion, it's contagious. But I have this problem in that I have trouble grokking Lisp. That is, I'm not at the level that I can read it and understand it.

So, my usual approach when I hit code I can't quite figure out is I try to transliterate it into Python. Once I've successfully done so, I'll type the Lisp variant into SLIME - once I know why the function was written the way it was, rewriting it in Lisp gives me the feel of it. And a lot of Lisp feels elegant as I type it. I can see why this stuff is habit forming.

So rewriting into Python is a great way of breaking down what a function does... I'm not sure what I'll do when he starts getting into macros.

But doing so makes Python feel very inelegant. I can feel that I'm doing things with Python that run counter to the grain of the language. It looks inelegant too.

For example - Graham gives a function to find if list X is longer than list Y.


(defun compare (x y)
  (and (consp x)      
   (or (null y)      
       (compare (cdr x) (cdr y)))))

The literal translation into Python is


def compare(x, y):
    if x != [] and (y == [] or compare(x[1:], y[1:])):
        return True
    return False

It can be simplified a tad to


def compare(x, y):
    if x and (not y or compare(x[1:], y[1:])):
         return True
    return False

But it's still missing that sensation of flow that gets called "Pythonicness", which is a clear sign that you're using Python wrong.

I know I've used the word "feel" repeatedly, but that's what it comes down to. The Lisp isn't immediately accessible by reading it, neither is the Python. But when you're typing them out, the Lisp correlates seamlessly with the functional thought process, the Python feels like you're cramming on shoes that don't quite fit. I've read about this sensation before - admittedly, at the time I thought the Lisp - Ducati analogy was a tad pretentious, but I see where he's coming from.

Anyway, this doesn't mean I'm becoming a Lisp addict, but it does mean that a former Python zealot is having to admit to himself that there's something his favourite language doesn't do overly well.

Conclusion

There's been a lot of grumbling about the removal of functional aspects in Py3K. The removal of map, list, filter and reduce is no biggy, as the list comprehension cheerfully stolen from Haskell (syntax and all) fulfills the same need. But it's the slated removal of the crippled lambda that's raised the most hackles. There's been many proposals for special syntax to allow multi-lined lambdas, but none have been accepted by the BDFL.

At the end of the day, Python really isn't a very capable functional language, and I don't think that multi-lined lambda would fix that. It's the syntax, it's the philosophy, and it's the feel of Python that make it a fantastic dynamic imperative language and a half-hearted functional one.

Opera, Firefox, and IE7 - they all suck/rock

2006-11-15T19:43:00.000-08:00

I feel uniquely situated to comment on the recent bout of browser releases, as I use Opera 9 at home for my daily browsing, Firefox 2 for development, and at work I use Firefox 2 for most of my browsing, and IE7 out of "lazy coder" necessity - as in I have to use certain very expensive J2EE websites that can't populate drop down boxes unless you're using the IE DOM.

I don't have a particular loyalty to any one browser. I do derive much enjoyment from the Fx fans versus Opera fans. So, my scientifically derived opinion on the recent crop of browsers?

They all suck and rock. Except for IE7.

That just sucks full stop. Let me break it down for you. My biggest bugbear tends to be resource usage. Yeah, Moore's Law etc. But when performance gradually degrades as usage continues over time, I get annoyed. (Don't try and tell me there's no memory leaks in Firefox, by the way)

Opera sucks...

Worst. Javascript. Error. Console. Ever.
Reconfiguring UI involves *.ini files in weird places.
Strange behaviour on some websites (A favourite haunt of mine will flash three times on loading. Can't figure out why.
It doesn't have an equivalent to FX extensions (Yes, I know, most of the time you don't need them, but the capability would be nice.)
Can get RAM heavy after long periods of browsing (the 100MB mark)
Can have issues with the Shockwave plugin

Opera rocks...

Render time. You do notice this when reverting to other browsers.
Response time. The UI "feels" more responsive.
Most of what Fx users install extensions for is built-in already.
Can do some mighty interesting things with those *.ini files.
More lightweight than Firefox or IE7

Firefox sucks...
Please note that (especially in regards to my complaints of memory) I have tweaked my Fx 2 install to minimise these - I even started a wiki page to catalog the tweaks

Resource hungry - my average browsing session with 15 tabs open will sit at the ~250MB mark normally.
Slow render time. You do notice this after using Opera.
UI feels a little tardy at time
Tabs can become unresponsive when another tab is loading - seems like a threading issue
about:config/Mozilla wiki - if I'm going to need your wiki to figure out how to fix FX 2 to work with my proxy, then please host it on a server that responds in milliseconds rather than minutes.
Assumptions that my system font will be the default size, leading to dialog sizing issues. Having to install custom chrome to fix this stinks. Thunderbird has the same issue.
Random crashes a bit too often.
"Missing plugin" ribbon has to be turned off via about:config
XPCOM is very, very, ugly
Firefox fanboys need to get a life.

Firefox rocks...

Good basic development support (moderately useful console, DOM inspector)
The concept of plugins
Plugins like Firebug, Colorzilla etc. that make Fx2 the dev's best friend
Some beautiful skins
No rendering issues that I've seen

Internet Explorer 7 sucks...

100% CPU usage, every single time it renders a page
Nearly as bad as Firefox 2 when it comes to RAM usage
Ugly UI
Stupidly uncustomisable UI
Security warnings that assume I'm stupid
Very slow renders
Unresponsive feel to UI
Moving the menu bar back to the normal position involves a registry hack
Turning off security warnings involves a group policy edit

Internet Explorer 7 rocks...

...um... IE6 is now obsolete?

Conclusion

Opera 9

Feels the lightest when in use and tends to have the smallest footprint on system resources. I prefer this for my usual browsing.

Firefox 2

Still a bit heavy, and feels sluggish after Opera. However, with a few plugins, is the developer's best friend for Javascript work.

Internet Explorer 7

Feels like a half-assed attempt to copy Fx2, and assumes at every level that you are an idiot. I'm not kidding.

Modifying certain security settings to make IE7 more secure (for instance, disallowing programmatic access to the Clipboard) will result in a ribbon coming up saying "Warning! Your settings aren't secure!". Being lectured by a piece of software that's wrong is just irritating It shows the IE guys obviously assumed that any change from a default setting was bad - whether by policy or laziness, I don't know.

Assuming stupidity is my guess as to why you can no longer drag and drop the UI - no doubt many a newbie managed to drag and drop their back/forward buttons and address bar out of existence, so it gets turned off for all of us.

So, Opera for browsing, Firefox 2 for development, and IE7 for lazy IE-only coders.

My Programming Language (MPL)

2006-11-02T12:27:00.000-08:00

Not quite on a par with the One True Language (OTL) syndrome but still repellent is the "My Programming Language".

Seems to occur in people who've used a particular language for a long time, and it manifests as an inability to accept criticism of the MPL. If someone's criticizing it, then they obviously don't know what they're talking about.

MPL differs from OTL in that it's not based on faith and fervour but more a sense of intellectual superiority and skill based on that language - and so insulting the MPL by proxy insults the intellect of the MPLer.

Case in point is this article, wherein the author is responding to an article which states that C/C++ is good for numeric processing - from the article, he appears to know the problem domain, but that doesn't stop the MPLers:

BTW, I've seen a number of these comparisions where an expert in one language does an implementation in several languages and, lo and behold, discovers that their favorite language wins out. It's all the same computer underneath, so if you aren't getting fast code in one language (that's sufficiently expressive) IT'S YOUR FAULT (whether because you aren't a good enough programmer in that language, don't know your compiler well enough, or aren't using the right compiler (the old Programming Language Shootout used GCC, which often gets its butt kicked by the Intel compiler on Intel machines)) not the language's.

...Except in this case, the author's favourite language (C++) lost when he compared it to other languages in this problem domain. I tend to trust the author's statement that Fortran is used for number crunching over C because someone's making and selling updated Fortran compilers. Selling.

So, if you're a MPLer, don't be. Different problem domains have different solutions; you wouldn't write a telephone exchange in Ruby, and you wouldn't write a web framework in Fortran. I hope.

Windows user meets Emacs II

2006-09-25T02:51:00.000-07:00

Well. Today was surprisingly fruitful.

My goals were:

Change Emac's default interface font if possible
Setup syntax highlighting for Erlang
Investigate indentation for Python

The first was easily done, so let's move on.

Setting up Erlang in Emacs

After some googling, I found this. Turns out the people who distributed Erlang were way ahead of me. Erlang came with two files necessary to make Emacs and Erlang play nicely.

I did, however, have to break my Windows metaphor... I created a .emacs file, and then inserted the following text in it:


(setq load-path (cons  "C:/Program Files/erl/lib/tools-/emacs"
     load-path))
(setq erlang-root-dir "C:/Program Files/erl")
(setq exec-path (cons "C:/Program Files/erl/bin" exec-path))
(require 'erlang-start)

That kind of stuff tends to worry/annoy me. I know Lisp when I'm seeing it, and I've heard all about Emacs Lisp, but yeah. Unix comes crashing in.

But, once that I was done, hey presto, I was away.

That's all for now

I didn't get a chance to investigate the Python indentation, but seeing as how Erlang can do it, I may need to steal some Lisp from Erlang and insert it into Python. I also need to bind shift + tab. I'm finding killing/yanking to be quite different to copy/pasting. Killing is quite powerful so I'll have to be careful with that. Learning the keys will take awhile, but it's been a lot less painful than I imagined.

Windows user meets Emacs

2006-09-24T04:17:00.000-07:00

My mouse's right button died. Never realise how much you use it until you lose it. I've rebound it to one of those pretty flashing-but-useless buttons on the left hand side, but it's awkward - makes coding painful.

So, in an effort to kill two birds with one stone, I'm going to try to learn how to use Emacs. I may regret it. (The other bird is write code without pain until next pay day.)

I'm like Bill Murray in Japan.

I must admit that I've only had about six months of *nix experience. Enough to learn that Gnome is more performance intensive than Windows XP, that XFCE is a nice lightweight alternative, that the way Debian manages console fonts is lacking compared to other distros and I've learnt the rudiments of the filesystem, although run levels and the associated files still confuse me.

I also learnt some basic Vim, enough to SSH into a *nix server and edit an Apache config file in it without worrying too much about breaking stuff.

But I never touched Emacs in *nix. I did use it as part of Lisp in a Box but only in the interactive Common Lisp mode, it was too confusing to try and edit with.

Foreseeable obstacles

The culture and language is going to be the largest hurdle, I think. Buffers. Meta keys. Mini-buffers... I'm okay with esoteric key sequences once I know what they're doing, but Emacs
seems like nothing else I've used.

Richard Stallman's cleverness is another. It may just be that Emacs is too difficult for me to grok. I sometimes wonder if *nix text editors were designed as some sort of initiation rite into a secret *nix brotherhood. If you could create a new file and save it, you were in. You got a swish badge and an impressive title if you could turn on code folding. We'll find it.

The parameters

I'm not going to change OSes. I can't be bothered installing another OS for the sake of a text editor. So Win32 it is.

As I'm on Windows, I'm not going to compile anything. It's a pain in the butt for someone who doesn't know much about *nix development environments to replicate - the pain I experienced when trying to compile Python extensions using MinGW was horrendous.

All I want to do is write code.

So my targets for tonight:

Install Emacs
Create a new file and save it
Open a Python file and an Erlang file and explore code highlighting

Step 1: Install Emacs

I feed Google"learning emacs on windows" - hoping to find newbie related materials. 4th result is http://www.emacswiki.org which looks hopeful - despite the trash talking I give Wikipedia, I do like wikis for this sort of stuff - I've learnt to love them through learning wxPython etc.

http:///www.emacswiki.org is good. Under Downloading and Installing, I find a page on Win32 ports in particular. After a few mouseclicks, I find myself at the homepage of EmacsW32.

Hmmm, do I want the patched or unpatched version? From the looks of it, I'll want the functionality of the patched version, with the caveat that it could be brittle.

I download the install file, a nice typical Windows installer, and execute it. I have to stop and think about the file extensions questions, and when it asks me if I want to associate the "Open" verb for XML with Emacs, I'm guessing that saying yes means Emacs will open them for editing, not try to run them.

A console window pops up while EmacsW32 edits some Lisp files, and then I'm away.

Step 2: Create a new file and save it

I could use the two huge buttons on the toolbar, but I want to learn the key sequences. Luckily, the key sequences are listed in the drop down menus besides the functions.

I hit ctrl + x, ctrl + f and enter my new file's name. I enter some text and pres ctrl + x, ctrl + w. That was reasonably straight forward. I'm quite pleased with how Emacs handles directory listings.

Step 3: Python and Erlang

Emacs recognises the Python immediately and turns on syntax highlighting immediately. I'm a bit disappointed that when I press enter in an indented block that my cursor isn't moved to the same indent - but that may just be an option I have to change.

Emacs doesn't seem to recognise the Erlang file at all, so perhaps I need to enable syntax highlighting for that, or Emacs may not have it installed by default. I can't figure out how to check the syntax settings via the menus, so it's time for the Emacs tutorial - well, I'll do that tomorrow.

Tomorrow's mission:

Change Emac's default interface font if possible
Setup syntax highlighting for Erlang
Investigate indentation for Python

Ruby and Python - there's no fight

2006-09-21T16:03:00.000-07:00

Apparently, it's a fight to the death between the two languages, based on the ecological principle of "competitive exclusion" which states that two species can't share the same ecological niche without specialization or one becoming extinct.

I have problems with this analogy because it doesn't account for the evidence, and I also have issues with the above gentleman's assertions that Ruby is more powerful.

Remember when everyone stopped using PHP?

Hah, I wish. But here's a list of languages/frameworks that share the same ecological niche in the coding world, but yet are still present and still being used.

ASP/ASP.NET/PHP/your templating language here
Javascript/VBScript
Java/C#
VB.NET/VB6
Ruby/Python/Perl
Haskell/OCAML

I'm sure there's thousands of other examples that I've missed. If the principle of competitive exclusion applied in the programming world, there'd be one templating framework, one client-side browser scripting language, one functional language, etc. etc.

Programming languages are not subject to natural selection

Natural selection is ruthless. Humans, not so. The fact that there are still COBOL programmes running will attest to this. There will always be plenty of room within a "niche" for various alternatives. Aside from dubious, highly unscientific assertions based on book sales via O'Reilly, there's no evidence that Ruby is usurping Python's userbase. (Re: those book sales - Ruby's community should take that as a sign that they need to improve their online documentation.)

Ruby users be hatas, yo... why?

Funnily enough, when I see statements like "It's a fight to the death" it's invariably from a Ruby user.

I don't see it from Python users because Python is out of the One True Language (OTL) phase, so the userbase isn't as fervent these days, we're a bit more tolerant, and more willing to filch good ideas.

I came into Python just as the OTL phase was ending (in those days, it was fight to the death between Perl and Python), so I've seen the community mature over time, and the fanatics either settle down or move on. (Guess where they moved onto...)

So, a question for any Ruby users who read this - you seem to be of the opinion that Ruby needs to kill Python. Why? Python didn't need to kill Perl.

Now, moving onto address Tomasz Węgrzanowski's points. Incidentally, anyone else find his usage of a cat picture highly ironic given the nature of the post? Unintentional, I'm sure.

Power and popularity

Quotes from taw are given in italics. He states that

right now - Ruby is a bit more powerful, Python is a bit more popular

and he defines power and popularity as:

Power (expressiveness, performance etc.)
Popularity (which results in more libraries, better tools, fewer bugs etc.)

So if you want Python to win:

Reduce the gap in power - make Python support blocks, get rid of explicit self and get better support for metaprogramming. If Python did it a few years ago, nobody would have heard of Ruby. If you don't like blocks, metaprogramming and implicit self, you don't have to use them - simply consider them a price to pay for making Python the next big thing. If it gets them today, it is still very likely to win.

I must admit that I fail to see how the usage of self means Python isn't as "powerful". It fits with the Python philosophy of explicit is better than implicit - I can't see how it impacts performance or expressiveness in any way. Blocks would be nice, but metaprogramming? If metaprogramming is why I need a language, I'm going to use a Lisp dialect.

Reverse gap in power - can Python express certain things a lot better than Ruby ? Write useful programs and libraries that would be much more difficult without this power and popularize them. Attack the enemy where it is strongest.

I don't understand this point, so I can't really answer it.

Make the most of greater popularity - work on useful things that are very useful but hard to reproduce with smaller user base - like Psyco and stable ports to JVM and .NET

Okay. Do Jython and IronPython 1.0 qualify as stable ports to JVM and .NET? IronPython even managed to retain compatibility with most of the CPython standard library.

And if you want Ruby to win:

Increase power advantage - take powerful features from Lisps, Smalltalk, Erlang, Haskell or whatever, and integrate them with Ruby - making different paradigms work together is one of the greatest strengths of Ruby, so make use of it. Make metaprogramming and DSLs easier by writing metalibraries and tutorials.

I'd add a cautionary note to this, there's an old saying about being a jack of all trades, but a master of none. Ruby should focus on being a good Ruby, and not a half-decent Lisp, a quarter decent Erlang and a two-thirds okay Smalltalk.

I'd still reiterate that if you need to metaprogramming and DSLs that you're more than likely going to be using Lisp. L# has just been released, which I find interesting, as it has access to the .NET libraries.

But anyway, before Ruby goes stealing stuff from every other language, give it threading and unicode first. Get the basics right.

Make the most of greater power - Ruby was able to get the momentum with Ruby on Rails, which would be very difficult to replicate in other languages. Ruby makes writing powerful embedded DSLs very easy, and powerful embedded DSLs are a potentially huge and pretty much unfilled niche. Take it and use it as a leverage to get more popularity.

Ruby on Rails is hard to replicate? Sorry, Django, Turbogears, Pylon, Zope, Cherrypy etc. you guys are too hard to use... erm... RoR is easy to replicate.

DSLs are a hard way to sell your language. They haven't worked for Lisp yet. Before you can sell Ruby based on DSLs, you need to tell people why they need DSLs.

Minimize effects of lesser popularity - if Python has a good library for something, simply recode it to Ruby and call it version 1.0. The languages are so similar that you should be able to do it in no time. You can get it more Ruby-like (with blocks etc.) later.
To be honest, I wouldn't recommend this. For example - Ruby's imap library is, in my opinion, better than Python's imaplib (apologies to ESR who wrote the part that annoys me the most, but an IMAP library that requires me to use syntax from the IMAP4rev2 RFC is failing), so replicating some of Python's libraries would be a mistake - at least until Python 3000's general revision, tidy up and improvement of the standard library has occurred.

There are some things in which current implementation of Python is simply better - Ruby 2 is trying to fix many of those (you cannot have Ruby Psyco without YARV). If you can - help with Ruby 2.

Yes. The priorities of the Ruby community should be - threading, unicode, speed, documentation. These are what Ruby needs right now. I'll keep an eye out.

Python and white space - biggest strawman ever

2006-09-20T15:48:00.000-07:00

Someone commented on an earlier post of mine that he wouldn't use Python because of the syntactically important whitespace.

Bah, this is the lamest reason. Tell me that you want full closures, tell me that you hate explicit self, tell me that you don't like Guido, tell me that Ruby on Rails is far superior to Django/Turbogears/Zope etc. - but don't pick on the whitespace, it's a lazy argument.

Why?

Ever used XML?

Yup, by default, that's got syntactically important whitespace too.

If you don't believe me, you need to read this.

(Anyway, how hard is it to press tab? If you indent code in a sane manner anyway, the only difference between Python and every other language is a lack of braces/keywords.)

Ruby vs. Python - why no-one should care

2006-09-19T18:18:00.000-07:00

BECAUSE THEY FILL THE SAME DAMNED NICHE.

I hate it when I see people trying to start flame wars over this. It's dumb. Ever noticed how Python users and Ruby users are, as a general rule, not really interested in moving to the other language?

It's because they fulfill the exact same need.

Object orientated? Check
High level? Check
Support functional programming? Check
Garbage collected? Check
Dynamically typed? Check
Expressive? Check
Fast to develop in? Check
Small, usable standard libraries? Check

Yeah, Ruby has closures, you can do the same stuff in multiple ways, it has Ruby on Rails, it has code blocks, and it has symbols, and Python's faster, does threading , unicode, has more web frameworks than keywords, has more third party libraries and the standard library is better documented, but honestly, this stuff only matters when you're deciding which one to invest your time in; that is, before you've learnt either.

Once you've made that time investment, there's no additional gain to be made from learning the other for the sake of it.

If you're a Java programmer, whom Ruby's community seems to be targetting, I'm sure that Ruby is pretty damned impressive. Any dynamically typed language has got to feel like heaven after Java. Ex-Java programmers seem to like Python too.

Once I've got Python or Ruby down pat, I don't need the other.

Let's run over it again

They fill the same niche
They have the same functionality
They have similar communities (Usually helpful, occasionally frosty, always passionate)
They've both been the Next Big Thing (Yeah, I'm using past tense for both, sorry Ruby dervishes)
They're converging

So, STOP EVANGELISING. I'm looking at fervid Pythonistas and rabid Ruby lovers both. It's inane, and you're wasting your time, and ours.

Footnote: I'm generalising, so please don't tell me about your conversion either which way - what I'm saying holds true for the greater majority of both communities, there will always be exceptions.

Hmmm, beta-y goodness

2006-09-19T14:22:00.000-07:00

Oh dear, Blogger seems to be inserting tags at random through my earlier post. Weird.

My firstish impressions of Erlang

2006-09-18T18:28:00.000-07:00

Having just read this blog entry about someone else's first experiences with Erlang galvanises me to write this.

I'm going to do it "good, bad, ugly" but as everyone likes the critical stuff more than the gushy stuff, I'll do it backwards.

The ugly

Strings. I bet you're surprised I started with that.

I'll start by noting that Erlang wasn't designed to twiddle strings, so you've got to keep that in mind when moaning about this - it's like complaining that Perl sucks for writing large fault tolerant concurrent telephony systems.

However, as there's a push in certain circles to spread the Erlang word into wider usage, now would be a good time for whoever maintains Erlang to consider revising string handling, because Erlang will never enter wider acceptance without the ability to handle a wider variety of tasks.

So, what's bad about strings?

Space

Each character is represented as an integer ASCII code. So, on my 32 bit system, that's 4 bytes for the character. A string is a linked list of characters, so that's another 4 bytes for each pointer, so we're looking at 8 bytes per character. Eep. (There's an upside to this, however, which I'll cover later.) - if you were handling a whole bunch of strings, you could hit memory issues, theoretically.

Speed

I don't know how broad this is, but from my tinkerings, string concatenation can be very slow if you do it wrong, because it's a linked list. I ran into this trying to write a recursive function to replicate Python's join method.

I was using an accumulator, to which various strings were being appended. Problem is, each time that something was appended to the accumulator, Erlang had to follow the pointers to find the end of it... which leads to O(n*n) execution time... the trick it would seem in Erlang is to build strings backwards. If I were going to joib the list ["The", "cat", "sat", "on", "the", "mat"] with spaces, using my approach would've looked like this

"The "
"The cat "
"The cat sat "
"The cat sat on "
"The cat sat on the "
"The cat sat on the mat "

Each iteration, Erlang would have to traverse the string to find the end. Whereas if I built it backwards, each list item would only be traversed once.

Unicode

Every programming language needs to support UTF-8 as standard. I'm not going to beat up Erlang too hard for this considering that Python and Ruby can't quite manage this either... at least until Python 3000.

I will comment that Erlang's usage of integers at least leaves the room for UTF-8 to be implemented.

String functions and regexes

Argh, no join? Also, strip functions that strip single characters at a time are not overly groovy.

As for the regex module... well, it's nice to have some functionality, (Although it was funny place to stick the split function) but in a post-Perl world, I feel like a caveman if I can't retrieve a group from a regex like "(.*?)" but instead have to use a combination of calls to regexp:match() and some adding and subtracting string indexes.

That's about it on the strings, for now. I'll cover some nice things about them later.

The bad

Syntax

I spent a lot of time wrestling with the syntax, mainly around the end of line delimiters and which to use where. Do you really need three of 'em? Now that I'm used to it and I've discovered Erlide, a plugin for Eclipse, my worries have disappeared, but the syntax is a hurdle to getting into Erlang.

Documentation

There is heaps of documentation available for Erlang... when you find it. I was bitterly amused when I looked up the built-in functions in the reference manual and saw "For a complete list of built-in functions, refer to erlang(3)."

Being told to refer to man pages on Windows is a bitter moment. Luckily, I found out about http://www.die.net/. I found the documentation for the tcp sockets easily enough (gen_tcp) but it didn't have any information about the socket options, or where to find them. The socket options are listed in the documentation for the inet module, but you do wonder why a link from gen_tcp to the relevant section of inet wasn't included.

Also, there are some useful functions that are undocumented, xmlerl:to_utf8 being one I've noticed recently.

The standard library

If you need to work with an URI, in Python you use urllib. In Ruby you use uri. Both libraries come in the standard library.

Erlang has.... Erlang has... Well, Erlang has yaws_api. Except it's not part of the standard library, it's part of Yaws, which is a webserver which you have to download separately.

Yeah, I know, telephony etc, but as I said, if you want to push a language to a wider audience, you'll need to cater to a wider range of tasks.These days, you'll inevitably end up manipulating URIs for some reason or another. Making yaws_api part of the standard library couldn't hurt at all.

Hey?! How did I end up in Visual Basic?

What's the index of a first element of a list? 0 right? Unless you use Visual Basic *snicker*... or Erlang.

In Erlang's defense, considering it has the [First_element ¦ Rest_of_list] capability a la Lisp's car and cdr, you don't need to use a numeric reference all that often, but still...

(Let me just clarify that this is a really tiny, insignificant issue, dear Reddit readers, honestly, I can live with it, see above re Erlang's variant on car meaning that you'll rarely need to use list indexes, but it does contribute ever so slightly to culture shock.)

The good

Okay, the boring bit now. ;) Despite my moans and gripes, Erlang is still pretty neat.

Pattern matching

Every language needs this! I am not kidding. This is a feature I often found myself wishing for when working in Python, without realising what it was called. For example, my (terribly inefficient) join function looked like this:


join([], Delimiter, Acc) ->
    Acc;

join([H¦T], Delimiter) ->
    join(T, Delimiter, [H]);

join([H¦T], Delimiter, Acc) ->
    join(T, Delimiter, Acc ++ Delimiter ++ H).

Basically, when I call the join function, Erlang will check to see which of the above definitions the arguments I pass matches, and then call that.

How I'd have to do this without pattern matching would be something like this -

join(TheList, Delimiter, Acc) ->
if length(TheList) == 0 ->
    Acc;
else ->
    H = lists:nth(1, TheList),
    T = lists:nthtail(1, TheList),
    if Acc == 0 ->
        join(T, Delimiter, [H]);
    else ->
        join(T, Delimiter, Acc ++ Delimiter ++ H)
    end,
end.

Note the lovely demonstration of the multiple line end delimiters ;)
But as you can see, pattern matching rocks the kasbah. You can also pattern match in case statements. This is one useful language feature.

Asynchrous message passing

I'm only just delving into this, but this and fault tolerance is where the strength of Erlang lies.

The best part from my limited dabbling is that sockets send asynchrous messages to processes.

What that means is that if the socket receives data, it sends the data in a message to the process that owns it - thus the process doesn't need to keep polling the socket and there's no messy "avoid a blocking socket" shenanigans. In other words, what Twisted Python tries to add to Python is built into Erlang from the get go, event orientated networking.

It's beautiful. Your socket handling process merely has to handle three kinds of message (one for data, one for the socket being remotely closed, and one for other errors) and that's it. An asynchrous non-blocking socket. No threading, no Queue objects, no Timeout exceptions, no blocking sockets. It's how it should be.

I'm well aware of the obvious faults of Erlang, but the obvious features of it are attractive enough to deal with the bad and the ugly. As I keep playing with it, I'll learn more and develop a rounder opinion of it. But for now, I am intrigued. Also, FP languages are always good fun - thinking recursively does hurt my brain though.

Now, to the fisking

Now for the article that inspired this one... bold and italics are the original, my comments follow.

Lack of basic Unix decency - no man page for erl or erlc (at least in the Ubuntu package), they do not accept --help, one cannot exit by ^D. These are very basic things. If they're broken, people are likely to leave.

I'm a bit confused by this. I found the man pages easy, and I'm using Windows.
I don't know how Linux distros work, but is it expected that they ship with manuals for obscure
programming languages preinstalled? Talk about bloatware...

Interpreter is terribly underpowered - in most languages you can write pretty much any code in an interpreter. In Erlang you cannot even define functions there. Of course there is no online help.

As an avid Python user myself, I completely agree. Compared to Python's interactive interpreter, Erlang's interpreter is underpowered. But then, last time I checked, interactive interpreters weren't exactly the industry standard. Try finding one for Perl. I tried a Ruby one once, but it broke when I pressed enter three times in a row. They may have fixed it now though. The inability to define functions is about as annoying as not being able to define things in DrScheme without restarting the interpreter.

Myself, I have a test.erl file that I define functions in, like DrScheme really. As I'm using Erlide, an Eclipse plugin, I have my editor and my interpreter right there. I then define the new function, and recompile. Which brings me to the chap's next point...

You actually have to compile your code, by hand, before you can use it - if you wrote a hello_world.erl, you have to compile it using erlc, then run erl, and call hello_world:whatever. If you made a mistake, you need to exit the main loop (by halt()., not ^D, to waste more of your time), correct the mistake, recompile, and start again.

Yes, you compile the code. But exiting the interpreter to do it? Funnily enough, there's a function called c() that compiles it for you......this complaint convinces me that the gentleman posting has not bothered to read the Getting Started with Erlang Guide in the HTML documentation, because it mentions it about 6 code samples in.

That's classic foot-in-mouth... reading the F manual is a good thing to do; I learnt that from harsh experience.

Don't Repeat Yourself ? Nah - In Erlang you have to repeat yourself a lot. In every file you need to write module name and list of all exported functions. And when you use them you have to prefix them by module name anyway. So even though foo:bar() simply has to be function bar in module foo, you have to explicitly list in on the export list in foo.erl anyway.

Oh come on, it's two lines. That aside, his complaint about having to explicitly export functions outside the module is weird. In the OO world, this is pretty darned commonplace, you know, all those private and public keywords that litter the place. Even in Python where the motto is "We're all consenting adults" and the language isn't designed to enforce encapsulation, there's a naming convention to indicate objects and variables that are deemed private by the developer and such, don't rely on them. So, Taw doesn't like encapsulation, a central tenet of OO? Keep that in mind for his next complaint...

It is not even marginally object-oriented. I'm not talking about having to hack object-orientation on top of it - it's just that basic object do not know what they are.

Ah, wait, no! OO is now good. But Taw has made a fatal mistake. He's mistaken a pure functional language for Ruby! If Ruby has closures, then it must be functional right? Referential transparency? What's that? When you're using a functional style in languages like Lisp that support FP and OO, you tend not to use objects, it seems to make things harder. A pure functional language does fine without objects, but Taw misses them.

Lists of numbers cannot tell themselves apart from strings and so on. Writing "Hello, world\n" out will actually output [72,101,108,108,111,44,32,119,111,114,108,100,10].

Taw has stumbled on the implementation of strings in Erlang without realising it. The one valid point he could've had, and he fumbles. See his next complaint for when the ball hits the ground.

String support is horrible - As I said, strings do not exist as first-class objects, they are emulated by strings. So no Unicode. No regular expressions. No high-level string functions. We're back to C.

His gripe about a lack of objects in a functional language and a weird muttering about emulation we can ignore.

Unicode. xmlerl:to_utf8 and xmlerl:from_utf8 - they're not documented, but they're there.
No regular expressions? Don't tell the regexp module it doesn't exist...
No high level string functions? Don't tell the string module either...
We're back to C Actually in C strings would be smaller in memory and faster, but I digress. :p

Strings in Erlang seem to suck if you've come from anything other than C, no-one denies that, but you've got to criticise real faults, not "I didn't read the manual" stuff.

Not Invented Here Syndrome - Erlang tries it very hard to be different than everything else. Syntax is different (somewhat Prolog-like), Unix conventions are disregarded, standard data types (strings, arrays and hash tables) are not provided, even io:format uses its own format strings instead of following printf. You cannot simply take a look at a few examples and start writing useful programs like with Ruby, Python or even Java. Not following conventions wastes everyone's time.

This one I'm going to break down again.

Syntax. Erlang was implemented in Prolog, hence why it looks like Prolog.
Unix conventions. I guess he means Ctrl ^ D. Trivia, but hey, I'm on Windows, so what do I know, it could be a life or death thing in *nix. Although my experiences with *nix GUIs would suggest that *nix isn't big on conventions anyway.
Standard data types. There are strings, there are arrays, and there's even a hashtable you can use if you really want. Just because Erlang calls them different things and does them differently, doesn't mean they don't exist! How do you find the lack of hashtables in Python? (Dictionary? Wassat?)
io:format doesn't copy printf. Which is true, it copies format from Lisp...
It's not Ruby, Python or Java. Well spotted! His complaint about the examples confirms he hasn't read the Getting Started Guide as I started writing my concurrent MSN client within half an hour of installing. Sure, it was terrible code, but I was writing it. I'd like to claim I'm a better coder, but I'd be talking rubbish.
Not following conventions wastes everyone's time. So does not reading documentation and expecting every language to be Ruby, hmmm?

OSS - Issues

2006-09-14T20:37:00.000-07:00

My employer recently rolled out Plone as our new CMS - you can see the results here - www.companies.govt.nz.

(Note that the transactional sections are still J2EE I think, I'm not entirely sure as to the technology being used there.)

In the short term it means that information can be updated very quickly, so if any information is erroneous or causing confusion for our clients the turnaround time to fix it is now measured in hours instead of days. This is brilliant; if information provisioned is inaccurate, it makes your entire organistion look incompetent.

Also, it just looks good. It even has rounded corners - how Web 2.0 is that?

I've been inputting data into a internal instance we're using for our knowledge store; and occasionally assisting in tweaking of the templates and some Python scripting. And wow, just wow - the capabilities. I love the versioning when editing data, and working with Python is, I think, every Python nut's dream - and it's actually quite powerful, even with the restricted execution environment.

Long term, I think our adoption of Plone is going to help establish the viability (or lack thereof) of OSS for use in the public sector in New Zealand. Let me explain that although I use a lot of OSS, I'm not a zealot. I'm a cheapskate, mainly. But I think the public sector needs OSS to be a viable option - it would save the taxpayer a lot of money.

Caution: Tautology follows

Government departments are very conservative when it comes to technology, for understandable reasons - they're highly publicly accountable for failures, especially when those failures involve large amounts of personal information about citizens that the government department has collected in its privileged role.

In this regard, I think that the culture of OSS as commonly perceived is a liability to uptake by technologically conservative organisations, and as such OSS is fighting an uphill battle to even be considered by the non-techies who make the actual decisions.

"You need to install the latest unstable SVN branch"

OSS development is pretty jungle like. Useful projects can die horrible deaths unexpectedly, the roar of competing bull developers can be heard as a project forks/schisms (Emacs/XEmacs), and sometimes you can fall into the swamp of poor documentation and , and occasionally you have to hack wearily away at the undergrowth just to get where you wanted to be.

In short "unstable branch" is a pretty good way to describe a common misconception of OSS - well, the "unstable" bit. This, however, can be dealt with - if you promote the fact that OSS projects can be stable. I think there's a tendency to emphasise bleeding edge over "doesn't crash and easy to get running".

If I'm a newbie to your new technology, something reliable is going to leave a better impression than your latest alpha release.

Your latest alpha may have scorching hot features that are revolutionary but if I have to download several patches from SVN, install a beta module for Apache and a third party hack for that module merely to get it working with my Apache install, I'm not going to be impressed. (It's what killed my interest in Turbogears - Django doesn't let you swap out templating languages like Turbogears, but it doesn't require that you do terrible things to Apache to get it to work. That's a big bonus.)

"The High One said you need to install the latest unstable SVN branch"

Okay, so a glib passing swipe at the Prophets of RMS, and the Church of Latter Day ESR (post-dotcom, natch) - but the zealotry of OSS uber alles doesn't help their cause at all. It scares people away. Advocates of open source have trouble being taken seriously. Want to know why? Write a blog entry criticising OSS Sacred Cow X (Firefox, Wikipedia, etc. etc. ad nauseum.) Make sure you only include valid criticisms. Watch as the rabid fanatics denying that their Cow is anything but divine outnumber the rational advocates 100:1. Watch as the rational advocates cringe at the hyperbole and personal attacks of the zealots.

Watch as the zealots help shape a wrong conception of the OSS community.

You see, instability and dervishes are by and far not true for the majority. But they are perceived by your average "weened on Windows" manager as representing all of OSS. And guess who gets to make the decision on what to use.

So what to do?

Stop with the fanaticism. Really. OSS won't cure cancer. All you're doing is scaring people off. Thanks.

Seriously, take Plone as an example. It's rock solid, it's a proven technology, it's used worldwide by various organisations, it has wonderfully powerful features, it's FREE, and still it is seen as a huge risk.

Sure, Microsoft got in there with their FUD and muddied the waters, but instead of calmly proving Microsoft wrong, the Church of OSS has been churning furiously, making it worse.

I'll put it this way - when you see Rev Phelps on TV, are you judging his argument based on the actual merits of it, or are you judging his argument based on the fact that he seems to be a total madman who's trying to suppress his own latent homosexuality?

Every time you type "Micro$oft" or "Winblows", you're doing a Phelps, and Microsoft is winning.

Emphasise stable releases

I'm not picking on Turbogears, honest, I quite like the concept it embodies but it is a really good example - go to their website, what's the available version available with one click on their front page? It's 1.0b1 - the last time I checked, it was an alpha version. Want to find the latest non-beta or alpha release? Click on Download in the top menu, now scroll down. About half way down, between TurboCheetah and TurboJson you'll find Turbogears 0.9.5 - I just hope you know what to do with a Python Egg.

There was some fabulous features added between Turbogears 0.9.5 and 1.0, so I can understand their excitement over 1.0b1. But, how's it going for the users? Check the Google Group - how many "Upgrade to 1.0b1 broke my install" and "1.0b1 not working" posts can you see? Okay, so it may be unfair to highlight such a new project, but there are people using Turbogears in production - I tried to run a production site on an older version, and then gave up in disgust when one upgrade broke my whole install. Upgrades should not break installs lightly.

Compare though, Python's download page - very first link is the production version, 2.4.3. Python 2.5 has some pretty damn cool features, such as pysqlite which is a Python interface to SQLite, becoming part of the standard library - having contributed occasionally to pysqlite in a docs and bugs way, I can't wait to play with Python 2.5.

But, where's the 2.5 RC2 download link? After all the links to the current production (that is, stable) version of Python.

Finis

Of course, I'm generalizing like mad. There's other issues too, like UI designed by programmers not designers, poor documentation, and occasionally hostile user communities, but they're easily fixable. But, I feel, as a pragmatic OSS user, that a combination of the "Ooh shiny whizbang" and "Raraghhhah Bill Gates wants to own your soul" is detrimental.

If it was technically savvy and experienced people who made the final decisions, we'd be okay - but it very rarely is in large organisations. OSS needs to put on a suit.

On blogging

2006-09-12T16:22:00.000-07:00

I've moved from Wordpress to here for the sole reason that I can edit the CSS, thus making the addition of basic syntax highlighted code to my posts a lot easier - I miss the pretty Wordpress templates, but I was able to hack around with the template here until I produced something satisfactory.

Incidentally, beta.blogger.com > blogger.com.