Coding like writing

As I’ve been on this journey of investigating and using high-functioning dynamic languages, I’ve been making discoveries about technology that never occurred to me before. The most rewarding discovery for me is that programming can be like writing. I am a writer at heart. I am also a problem solver. Just about any time I’m presented with a puzzle, I work incessantly on it. I don’t always solve it successfully, but my mind becomes focused and committed to solving it.

Some of my most rewarding experiences have been the articles I’ve written that have been published so that hundreds of people can read and benefit from something I’ve learned or realized. This blog has been an extension of that.

When I say that “coding can be like writing”, I’m not saying that one can program in English in Lisp or Smalltalk. You can get close to that using embedded DSLs in these languages. What I’m talking about is mixing declarative code with programming logic written in expressions that can be evaluated.

My first encounter with this was in Lisp. In the languages I used about 7 years ago, C or C++, when I wanted to create a data structure, say like the Composite Pattern, I had to create classes for the composite and leaf objects, and then I had to dynamically allocate memory for these objects, put data in them, and then traverse them. Say I had a collection of assemblies and parts for a machine. Assemblies can contain other assemblies, and parts, but parts are atomic. In Lisp if I want to instantly create a composite data structure of assemblies and parts, I can just do this:

(setf *machine*
  '(:assembly A
      ((:part "partNumber1")
       (:part "partNumber2")
       (:assembly B
         ((:assembly C
            ((:part "partNumber3")
             (:part "partNumber4")
             (:part "partNumber5")))
          (:part "partNumber6"))))))

I simultaneously create the structure and populate it with data, declaratively. In fact, this is not unlike what one does in XML, and indeed this is what’s been resorted to in the popular languages in the last several years. The data is declared in a syntactic structure, which is then parsed by an XML parser, and loaded into a DOM, where it can be traversed and modified. In Lisp much the same thing happens, though the structure you have to write is less verbose.

I can filter (based on assembly name) and traverse the structure using this code:

(defun display-assembly (assembly-name assembly)
  (cond ((null assembly) nil)
        ((and (eq (first assembly) ':assembly)
              (eq (second assembly) assembly-name))
           (progn
             (format t "~%assembly ~S" assembly-name)
             (display-parts-list (third assembly))
             (format t "~%End assembly ~S" assembly-name)))
        (t (display-assembly assembly-name
             (find-assembly-in-parts-list
                assembly-name (third assembly))))))         

(defun display-parts-list (parts-list)
  (dolist (part parts-list)
    (if (eq (first part) ':part)
      (print (second part))
      (display-assembly (second part) part))))        

(defun find-assembly-in-parts-list (assembly-name parts-list)
  (dolist (part parts-list)
    (when (eq (first part) ':assembly)
      (if (eq (second part) assembly-name)
        (return part)
        (return (find-assembly-in-parts-list
                   assembly-name (third part)))))))

Using recursion this code attempts to find the assembly I am referring to (for example (display-assembly ‘B *machine*)), and outputs its parts list. I’ve only used Lisp for a few months. I’m no expert. Maybe this could’ve been written better.

Some explanation for those unfamiliar with Lisp. You’ll notice that all of the code is in prefix form. There are no infix expressions. In Lisp there are lists, and atoms. Lists are the things in parentheses. Atoms are character or integer constants, or symbols. Every instruction and piece of data exists inside lists (the more technical name for them are S-expressions). Each instructional list starts with a function name. Each function returns a value, even if it’s just nil. A symbol or list of data elements starts with an apostrophe. A symbol can be one or more characters in length. What are strings? I suspect they’re classified as a list of characters, but I’m not sure yet.

The code here is a series of 3 functions: display-assembly, display-parts-list, and find-assembly-in-parts-list. Each function definition starts with “defun” (which is itself a function, but you can just think of it as a keyword for now).

The macro, “cond”, is like a series of if-then-else statements in the popular languages. Each term in the “cond” list starts with a conditional test. If the test is true it executes the action associated with it. If it’s false, it moves on to the next term. The last term’s conditional statement, in this case, is “t” for “true”. In effect it is the default term executed if all other terms are false. You don’t have to do this. It just worked for me here.

Macros are code generating functions. The best analogy I can think of is they’re like macros in C and C++.

The “null” function tests whether its argument is “nil” (think of this as “no value”). Every list is terminated with “nil”. So this is my bootstrap condition for display-assembly’s recursion. If “null” is true, I return “nil”. The “eq” function tests equality between symbols.

For outputting data, sometimes I use the “print” function, and other times I use “format”. It just depends on whether I want to format the output.

The “if” function is like an if-then-else statement in the popular languages. It takes 3 arguments: a conditional expression, an expression to evaluate if it’s true, and another one if it’s false. The “when” function is like an if statement without the else.

The “return” function works like you’d expect a “return” command to work in the popular languages. It breaks out of whatever context it’s in and returns its value from the enclosing function (defined with “defun”).

Another example I’ll give is from a Squeak project I’ve worked on from time to time. As an exercise in refreshing my memory of Smalltalk, and to eventually learn the Seaside framework, I decided to redo a project I had originally done in ASP.Net 1.0. As part of the project I used a screen scraping library and regular expressions to get employment data from the Bureau of Labor Statistics automatically, parse out the data, which was loaded into a database, and selectively display comparative employment data that a user wanted to see. I decided to try my hand at creating a simple DSL in Smalltalk to do the screen scraping. I wanted to query the web server in a similar fashion to a SQL query. It looks something like this (from actual code):

| query page |        

query := (WebQuery
  fromAddress: 'http://data.bls.gov/cgi-bin/srgate')
  where: #format is: 'block';
  where: #html_tables is: 'no';
  where: #catalog is: 'no';
  where: #delimiter is: 'tab';
  where: #print_line_length is: '400';
  where: #level is: '1';
  where: #series_id is: 'CES0000000001';
  where: #year is: '2001-2006';
  yourself.         

query addParametersWithNullValue: #(#lines_per_page
  #row_stub_key #date #net_change #start
  #net_change_end #percent_change_start
  #percent_change_end).         

page := query retrievePage.

Some explanation. In Smalltalk you have objects that receive messages. The object is always on the left-hand side of an expression, followed by the messages sent to it. There are three types of messages: ones with parameters (these messages have colons to delimit the parameter values), those that don’t have parameters (they have no colons), and operators which also take parameters, but don’t have colons. In the first assignment (query := WebQuery …) I use what’s called a “cascade” to send several messages to the WebQuery object all in one statement. Each message is separated by a semicolon. In a fashion similar to Lisp, the result of an expression is always the result of the last sub-expression evaluated (in simple expressions this is just the expression itself). Since I want to ensure that “query” gets a WebQuery object, I send a message called “yourself” to WebQuery to ensure that “itself” is returned to “query” in the assignment. The array that’s passed with the addParametersWithNullValue: message starts with “#(“. Actually this is the same way Lisp array literals are declared. The values inside the array are symbols (each starting with #).

The server process that serves the page expects to get 16 parameters. Since I didn’t want to fill them all in, I added some with the null value. This code will POST the query data to the BLS server and retrieve the corresponding page into the “page” variable. The data parsing comes later.

Once I tried this it was great. This is what I meant in past posts when I’ve said that I’m expressing my intent in the language.

The nature of software

I understood after working in this business for about 4 or 5 years that just about every project I worked on up to that point was “hand made”. I said this once to a fellow developer, and he joked that I was saying we were like the furniture craftsmen of old, who worked with primitive tools, making our software out of “Corinthian leather”. I didn’t quite mean it that way, but in hindsight, in a way, he was right. By “hand made” I meant that the logic of the program and a large part of the structure have to be invented each time. There’s no existing structure or documentation that adequately lays out the solution for us. We have to come up with that ourselves.

I read “The Art of Lisp & Writing”, by Richard Gabriel a week ago (h/t to José A Ortega-Ruiz), and it captures the essence of what I’ve felt for a long time about programming. It looks like it was written in the 1990s.

Gabriel makes the point in his article that mature engineering disciplines work on what’s been standardized. There’s not very much that’s invented from project to project. We know how to build bridges, and we understand how to build them well. You can lay out a set of specifications for one, and from that determine what materials, structure, shape, and land space are needed. Software is usually not like this. Have you heard of a bridge being built “iteratively” in your lifetime? Can you imagine a bridge being built in our day and age where the builders just “take a stab at it”, have inspectors review it, have the engineers significantly “revise” it, and see this process repeat multiple times before the project is done? Not on your life! This isn’t done, because it doesn’t need to be done. Further, with physical materials this can weaken structural integrity. It’s better to get it right the first time with concrete.

Yes bridges are inspected, but they are inspected for structural integrity, and approval is either given or denied based on whether the structure is up to standards. Relatively small modifications are made to improve their integrity. You don’t see them being significantly “revised” unless there was incompetence in the first place. There have been bridges with design flaws in them, even as recently as several years ago, but they are rare occurances. Design flaws and ad hoc iterative development in software are commonplace–too commonplace for software development to be considered a mature engineering discipline.

Gabriel makes the point that we humans have been building bridges for thousands of years. We’ve only been writing software for fifty. Despite the belief that software can be built in a standardized way, it doesn’t work that way. Not yet. We all know this (speaking for us developers, at least).

Programming is a creative process–a craft. Software is crafted in a highly interactive process, hopefully with the people who will eventually use it. It is molded, shaped, and refined, until it gets to a point where it’s judged to be useable. Even then, it’s not done. More gets added to it later with enhancements, or the users decide they don’t want a part to work this way, but another way.

Gabriel starts by talking about the creative process of writing:

Writing a beautiful text takes sitting before a medium that one can revise easily, and allowing a combination of flow and revision to take place as the outlines and then the details of the piece come into view. Changes are made as features of the writing emerge—first the noise, then the sense, the force, and then the meaning—and the work of the writer is to continually improve the work until it represents the best words in the best order. Great writing is never accomplished through planning followed by implementation in words, because the nature of the word choices, phrasings, sentence structures, paragraph structures, and narrative structures interact in ways that depend [on] each other, and the only possible plan that can be made with which a writer can “reason” about the piece is the piece itself.

This is true of all creative making—flow and revision are the essences. Flow is the preferred state of mind for the writer doing discovery. Flow means that there are no barriers between the movement of the mind and the placement of words on the page. When in flow, the writer feels as if he or she is in a vivid and continuous dream in which the relevant parts of the dream flow onto the page.

Revision includes reading material already written and perfecting it as presentation, but as the name suggests, it is more. Discovery is always mixed in with perfecting, because, as [Richard] Hugo says, “one does not know what will be on the page until it is there.” Moreover, revision—when performed courageously—involves determining what to leave out, just as the mapmaker determines what in the real world can remain unmentioned and unrepresented for a map made for a particular purpose.

For many, this will not sound as if I am describing programming—or as some like to put it: software design and implementation. To satisfy the biases of such folks, I will use untypical words and phrases for concepts that have acquired too strict a meaning.

So instead he uses the term “programming medium”. He goes on to describe the creative process of programming:

The difference between Lisp and Java, as Paul Graham has pointed out, is that Lisp is for working with computational ideas and expression, whereas Java is for expressing completed programs. As James [Gosling] says, Java requires you to pin down decisions early on. And once pinned down, the system which is the set of type declarations, the compiler, and the runtime system make it as hard as it can for you to change those assumptions, on the assumption that all such changes are mistakes you’re inadvertently making.

There are, of course, many situations when making change more difficult is the best thing to do: Once a program is perfected, for example, or when it is put into light-maintenance mode. But when we are exploring what to create given a trigger or other impetus—when we are in flow—we need to change things frequently, even while we want the system to be robust in the face of such changes. In fact, most design problems we face in creating software can be resolved only through experimentation with a partially running system. Engineering is and always has been fundamentally such an enterprise, no matter how much we would like it to be more like science than like art. And the reason is that the requirements for a system come not only from the outside in the form of descriptions of behavior useful for the people using it, but also from within the system as it has been constructed, from the interactions of its parts and the interactions of its parts separately with the outside world. That is, requirements emerge from the constructed system which can affect how the system is put together and also what the system does. Furthermore, once a system is working and becomes observable, it becomes a trigger for subsequent improvement. The Wright Brothers’ first flying machine likely satisfied all the requirements they placed on it, but they were unwilling to settle for such modest ambitions—and neither was the world—and even today we see the requirements for manned flight expanding as scientific, engineering, and artistic advances are made.

This is the reason the Waterfall Model, as it’s usually implemented, is such a problem. The typical implementation assumes that one can go from requirements, to design, to implementation, to testing, and to delivery, without ever needing to go back and significantly revise something. The Waterfall Model was originally conceived as having feedback loops, but most organizations don’t implement it that way.

Gabriel continues:

James Gosling in talking about Java acknowledges that at least the object-oriented notion of late or dynamic binding is important to support shipping new (though constrained) versions of libraries and to make derived classes, but he seems to forget that the malleability of the medium while programming is part of the act of discovery that goes into understanding all the requirements and forces—internal or not—that a system must be designed around.

As a system is implemented and subjected to human use, all sorts of adjustments are required. What some call the “user interface” can undergo radical reformulation when designers watch the system in use. Activities that are possible using the system sometimes require significant redesign to make them convenient or unconscious. As people learn what a system does do, they start to imagine what further it could do. This is the ongoing nature of design.

Gabriel begins to merge the concepts of creative writing and creative programming:

The acts of discovery and perfecting merge for the writer, and for this to happen it is essential that the medium in which the creation is built be the medium in which it is delivered. No writer could even begin to conceive of using one language for drafting a novel, say, and another to make its final form.

Like many types of artist, the writer is manipulating during the discovery and perfecting stages the very material that represents the work when complete. In the old days, the manuscript would be retyped or typeset, but not so much anymore. The word “manuscript” itself refers to the (hand written) copy of the work before printing and publication, which the writer constantly revises. But the manuscript—which is constantly worked over—differs from a published work only by finally pinning down decisions like fonts, page breaks, illustrations, and design elements.

What’s apparent to me is that Gabriel sees the value of having a stricter programming language for when a work product is mature, but while a product is in its early and mid-development stages, greater flexibility in a “programming medium” is the ideal.

I get the image that what he wishes more of software development was like was using a word processor. When you’re using one you don’t have to plan what you’re going to write. You can just start. Once you have some things in the document, you can shift things around. You can begin to shape it. Contrast this with the punch card, and teletype machines, or the line editors we used to use for programming (I remember them). If you made a mistake, it was a real pain to correct it. It became more productive to try to plan what you were going to write, because mistakes were costly. So it would seem this is what most of us are doing today with software development. And the thing that’s weird about it is it’s like we have something like “word processor” technology for software here with us today. It’s been there for decades. We’re just not using it, almost out of willfull ignorance.

What’s emerging in my mind as Gabriel describes his experience with developing a product is the Groovy language, which is based on the JVM, and allows both dynamic and static typing. You can use straight Java with it. You can mix static and dynamic typing together, or make your code completely dynamic. Getting away from Groovy for a moment, when I read what Gabriel is talking about, the image that emerges for me is that the developer starts out with strictly dynamic types in a late-bound system. It is written, executed, and delivered this way. At about Version 3, say, static types and restrictive scoping start to get introduced (like declaring a class “final”, and certain parts “private”). As time passes more of this gets introduced into the system as it matures, and the development team becomes certain that they don’t want certain parts to change. This is like typesetting when a manuscript is deemed ready to go to print, after many drafts have been reviewed.

Continuing, Gabriel gets more concrete about the problems with the contradictions in purposes that developers are faced with today:

The distinction some make between prototyping and software development has vexed me my whole career. I have never seen a version 1.0 of a system that was acceptable. I find that every software group messes with the software up until the very time it’s deployed or delivered. At the outset the developers want to get something running as soon as possible—they are interested in leaving things out that are not necessary to the purpose of understanding the internal forces and external reactions affecting what has been built. When prototyping was considered part of the development process, people spoke of prototyping languages or languages good for prototyping as an ideal. Though these times are long past, software developers still need to go through the same steps as prototypers did and for the same reasons, but what these developers are using are delivery languages or languages designed to describe tightly a finished program.

The idea behind a description language is to create the most complete description of a runnable program, primarily for the benefit of a compiler and computer to create an executable version of it that is as efficient as it can be, but sometimes also for readers who will perhaps be enlightened by knowing all the details of the program without having to do a lot of thinking.

I like to think of a program written in one of these languages as a “measured drawing.” A measured drawing is a set of plans made by a craftsman of a piece of furniture, for example, which enables an amateur to make an exact copy. Whereas a master craftsman will be able to take a rough drawing or an idea of such a piece and by virtue of his or her expertise make a perfect specimen, an amateur may not be able to make all the joints perfectly and in the perfect positions without having all the dimensions and distances spelled out. Often, a measured drawing requires the craftsman to build a prototype to understand the piece and then a second to make one perfect enough to provide the measurements.

The problem with the idea of using a programming language designed to describe finished programs is that programs are just like poems. Paul Valéry wrote, “a poem is never finished, only abandoned.” Developers begin work on a project and deliver it when it seems to work well enough—they abandon it, at least temporarily. And hence the series of released versions: 1.0, 1.1, 1.5, 1.6, 2.0, 2.05, etc. Trying to write a program in manuscript using a finished-program programming language would be like trying to write a poem using a Linotype machine. Perhaps using a Linotype machine and thereby producing a beautiful printed book of poetry is good for readers, critics, and therefore literature, but it doesn’t help poets and would make their writing lives intolerable with lousy poetry the result.

The screwed-up way we approach software development is because of what we have done with programming languages. With some exceptions, we have opted for optimizing the description of programs for compilers, computers, and casual human readers over providing programming media. Early on, computers were insanely slow, and so performance was the key characteristic of a system, and performance was easiest to achieve when the language required you to “pin down decisions early on.” Because of this, much work was poured into defining such languages and producing excellent compilers for them. Eventually, some of these languages won out over ones more suitable for drafting and exploration both in the commercial world and in academia, and as a result it is politically difficult if not impossible to use these so-called “very dynamic” languages for serious programming and system building.

Interestingly, the results of this optimization are well-described by Hoare’s Dictum, often mistakenly attributed to Donald Knuth, which states:

Premature optimization is the root of all evil in programming.

–C. A. R. Hoare

This is interesting to me, because it brings into relief that our practices and beliefs about software construction arise in an ad hoc fashion. We accept this, but for some reason (he attributes it to fear of size and complexity of projects) we don’t accept the ad hoc nature of software construction itself, in the current state of the art. Despite the fact that we accept that “all software has bugs”, for the most part we don’t use technologies that allow us to fix those bugs easily. Instead we use technologies that restrict what we can do in an effort to prevent bugs…but bugs appear anyway. If anything we’ve tried to make up for the inflexibility of our development technologies with Information Systems infrastructure. One of the answers to software inflexibility we as an industry came up with was “put it on the web”. My guess is politics had something to do with this. I guess the belief is if we weren’t using inflexible software development technologies, then bugs would multiply out of control. Interesting how the reaction was to bring software over to an information systems infrastructure that’s less stateful, less restrictive, and more flexible…

Gabriel’s message is we need to give up our fear, because it’s irrational. Rather than restricting developers, a “programming medium” should be used to free them to make frequent and rapid changes to software in the early stages of development. You will still have bugs, but they will be less consequential, because the technology will allow them to be fixed quickly.

Alan Kay made a similar point to what Gabriel said, at his ETech presentation on Squeak and Croquet in 2003:

The key idea here, which is still not the main idea, is what to do until software engineering gets invented. Do late binding. Virtually every piece of software done today is done as early binding, through manipulating text files, compiling. Java still has a hard time adding code to itself while it’s running, for chrissakes. So, if we go back to Peter Deutsche’s Lisp of 40 years ago it did not have that problem.

You should be able to do everything dynamically. The system should be able to safely maintain its own environment while you’re debugging its environment. You should be able to make every change in less than 1/4 of a second, and so forth. And the reason is we just don’t know how to do software yet. So basically, when you have the disaster, you finally realize, ‘Oh, I should’ve done this back here.’ If you’ve got a late-binding system you can go back and change it. If you’ve got a typical C-based–C++, Java, whatever other systems people use–you just can’t do it.

Kay was being imprecise here. What he’s getting at is that software engineering is not yet a mature engineering discipline, as is bridge building, for example. Even Kay would acknowledge that the software we build today is built using “engineering of a sort”. In the second paragraph it sounds to me like he’s talking about systems which have the “five nines” requirement, that they be up and running all the time. There’s not really time for downtime. In that case he has a point. Even if you decompose the system into shared modules (like DLLs in Windows), there’s still the possibility that trying to change one of them will force you to take the system down. The web technologies of late have managed to alleviate this issue some. For example in ASP.Net it’s possible to update an app. without shutting it down first. If you haven’t structured your modules such that multi-step user interactions are atomic ahead of time, I suspect it would be disruptive. That’s kind of the point. In a late bound system you don’t have to worry about structuring modules for application maintenance. Since there’s no distinction between what’s “live” and what isn’t in a dynamic system, you can just change it on the fly.

Gabriel, in my last quote here, gets into what makes great software–creativity: 

[Lisp] is a good vehicle for understanding how programming and software development really take place. Because programming Lisp is more like writing than like describing algorithms, it fits with how people work better than the alternatives. The problem, of course, is that writing is considered a “creative” activity, involving “self-expression,” and creativity is a gift while self-expression is hokey. Not so:

The conventional wisdom here is that while “craft” can be taught, “art” remains a magical gift bestowed only by the gods. Not so. In large measure becoming an artist consists of learning to accept yourself, which makes your work personal, and in following your own voice, which makes your work distinctive. Clearly these qualities can be nurtured by others…. Even talent is rarely distinguishable, over the long run, from perseverance and lots of hard work.

–David Bayles & Ted Orland, Art & Fear

Writing and programming are creative acts, yet we’ve tried to label programming as engineering (as if engineering weren’t creative). Instead of fearing creativity, we need to embrace it.

My emphasis in bold/italics. Couldn’t have said it better. 🙂

On Richard Gabriel’s site I also found an interview with him in 2002 titled, “The Poetry of Programming” in which he talked about his idea for creating a degree program called a “Master of Fine Arts in Software”. It would’ve focused on just what he’s been talking about. It seemed for a bit like the University of Illinois was going to take up the idea, but I can’t find references to such a degree program being implemented anywhere. I guess the idea died.

My publishing past

In case you were wondering, I mentioned earlier that I had written a few published articles in the past. My first one appeared in the Dec. ’92/Jan. ’93 issue of Current Notes magazine, called “Atari’s Future: The User Comfortability Factor”. My second was in the February and April, 1993 issues (a two-part series) in Atari Classics magazine, called “Advanced C Programming on the 8-bit”. Nothing big, but it didn’t matter. I really enjoyed seeing what I wrote in print. 🙂 My third article was published online in May, 2002 on DevX.com, called “Create Adaptable Dialog Boxes in MFC” (note: The inset graphics figures for the article are distorted for some reason. If you click on them they show up nicer).

Advertisements

13 thoughts on “Coding like writing

  1. Pingback: University Update

  2. Pingback: software engineering » Blog Archive » Episode 52: DSL Development in Ruby

  3. Pingback: prototyping » Blog Archive » Coding like writing

  4. Mark –

    You really hit a lot of the issues that have been plaguing me for a while head on. I keep wishing that I could somehow magically use a language that does away with all of the overhead of .Net or Java (in terms of programming process) to do version 1, just to see what works and what does not, and then rewirte the whole thing in a more concrete language.

    Part of the problem is the delivery process. Nearly every app that will ever be written that more than a few thousand people will ever use, has already been written and deployed in one form or another, with varying degree of quality. The only really differences are in the implementation. Word 2007 is fundamentally identical to Word Perfect 5.1, which was fundamentally identical to vi. Functionally equivalent? No. But the real productivity multiple for that genre is simply the ability to copy/paste text, and erase words without using Whiteout. That’s it. Period. The first text editor provided 80%+ of the productivity gains of Word 2007 when compared to a typewriter.

    Where am I going with this?

    Well, at this point, 95% of applications are being sold against an existing market in one way or another. As a result, no customer is willing to take version 1, use it for a while, figure out what they like and don’t like (assuming no bugs, of course), do a second version, and then go to a version 3 where the prototypes are scrapped and the whole thing is rewritten in a more concrete langugage and put into maintenance mode. No one wants to do it! No customer is going to boot their existing vendor to go through this pain, and no vendor is going to spend months or years going through the process with only one or two customers and no revenue.

    As a result, we see software developed in one of two ways: the monolithic process, where incorrect assumption prior to the first line of code being written remain in the code through EOL, and the perpetual beta that is becoming increasingly popular. The resistant to change systems tend to acheive stability more quickly since they are… well… hard to change except for fixes. The initial assumptions remain there *forever* regardless of how bad they were or how much the needs change. The perpetual beta model is simply unsustainable. No need changes so rapidly as to justify it, people eventually want stability and maturity, and after a certain point, the feature set does not need any further expansion.

    I tell you, my patience with the industry as a whole is running out. It is just impossible to meet the business needs of getting a kick butt version 1 that is perfect while working with the reality of the development process.

    J.Ja

  5. @Justin:

    Maybe you read what Richard Gabriel was talking about a little more clearly than I was. My impression was he was not talking about rewriting the app. in a stricter language, but rather talking about what he wished for, which was that programming was more like writing and publishing. He felt he had achieved most of that by programming in Lisp, but my sense of it was that he would’ve liked it if programmers could add strict typing to a program that was written using dynamic code, as it matured. This is where I brought the Groovy system into the mix. It’s based on the JVM (and is fully compatible with the Java API), but it offers a spectrum of development. You can program totally using dynamic typing, you can do a mix of static and dynamic typing, or you can do straight Java in it. The upcoming Orcas release of .Net will be similar, though I imagine it will force developers into more of a mix (if anything) of static and dynamic typing, due to the way the .Net Framework API is designed.

    I see what you mean about newer versions becoming less popular. My guess is Gabriel was not talking about the consumer market, but rather software development for large organizations, like internal IT business apps. In those cases you can have several revisions as the business customer asks for more features to be added, and bugs to be fixed. They have more of a desire to have those changes made, and they have the budget for it. I think the model would work for web apps. fairly well, too. They’re not things that have to be installed on each user’s PC, so deployment is less costly.

    Another possibility is that maybe upgrades would not be as costly if the apps. had been designed in a “programming medium” from the start. If it’s less of a pain to change, that means there’s less time involved in rewriting a portion of it, and that would help make it attractively priced for the market. The fact that the existing technologies make it so difficult to change things has to be a factor in why upgrades are expensive, no?

    I think the dynamic languages have a chance on the server side. It was difficult for dynamic languages to get a break when most apps. were written for GUIs, because the UIs were so platform-specific, and the dynamic languages did not tend to deal with that so well. Not that they couldn’t have, it’s just that the “bridge designers” who made interfaces between the dynamic runtime system and the native OS didn’t do a great job of taking advantage of all that was there.

    I agree with you that for some applications there’s no need to get beyond the basics. I haven’t bought a word processor since Word 2000. It handles what I need. I have Office 2003, but that was a freebie.

    Most experienced users don’t expect Version 1 of anything to be that great. I couldn’t tell from your comment whether you were getting pressure to produce a “kick ass Version 1”, or whether you were just lamenting that the industry in general tries to do that. From my point of view, I haven’t seen Microsoft produce something that was a “kick ass Version 1” that often. Usually it’s not, and people know it. They wait for Version 3 before they buy it en masse.

    I’d encourage you to take a look at Paul Graham’s article, called “Beating the Averages”. It elaborates on what I talked about here. He is a Lisp evangelist, entreprenuer, and venture capitalist. In the article he talks about a web start up he got going in 1994 called ViaWeb, written almost entirely in Lisp, and he talked about it being his “secret weapon”, that it was the reason he beat his competitors in his business’s space. He revised the app. frequently to keep ahead of his competition. He was able to do it faster than they were. That was his edge. Since the app. was on the web he didn’t charge his customers for upgrades. It was just a cost of doing business. He charged his customers a subscription fee for the service. In the late 90s ViaWeb was bought by Yahoo! and became Yahoo Store (though Yahoo ported it to Perl and C++ due to, they said, a lack of Lisp programmers).

    If you’re tired of what you see going on, maybe it’s time to put yourself into a different business model?

  6. I think I missed some of the point of the “programming media” idea, and simply extrapolated the idea of switching languages mid-stream onto it. That is really the only way to do what Gabriel is saying, with most current systems. It is not like mainstream systems let you build out a class using dynamic typing, and at some point say, “cement this class, and start forcing it to be statically typed” or something along those lines. I guess what he is looking for is something like sealing a class or chunk of code, but at the level of “how hard is this to change”, not in terms of inheritence.

    In terms of “version 1”, I am getting an immense amount of pressure on that front (we have a version 1 live at the moment, but it is undergoing a lot of tweaking), but I see it throughout the industry. At my last job, a single bug could cost our customers thousands, hundreds of thousands, or even millions of dollars, and possibly even bring the FDA and their lawyers into the mix. Bugs were simply not an option. But what I see now is that the existing markets are almost universally entrenched and market leaders so solid, you really need to come up with something great to justify the cost of switching. I have never met a customer who is completely happy (or even close to it) with any of their enterprise class apps. The sweet spot is lower cost of integration, equally complete feature set presented in a way that it actually gets used (think of Word 2007’s UI changes, designed to provide access to the loads of functionality no one used because they could not find it in the menu trees), general usability, and ease of deployment. Something like 2/3 of SAP seats that are bought go unused, because of these difficulties. Likewise for a lot of other software.

    I just sent in a review of Zimbra 4.5.5, at first I was upset that not every little item in Outlook was in its Web client, and then I had a blinding flash: 99.9% of Outlook users just use it for basic scheduling and email functionality anyways! In fact, most workers below the manager level barely use even the calendar features! So if the Web client does calendar & email, it is functionally equivalent to Outlook for 95%+ of users.

    I did indeed read the Paul Graham piece, a while ago. I hate to admit it, but it was the reason why I’ve been meaning to pick up F#, and why my memories of EdScheme keep getting brought up. I miss a language like that, and his article got me going on the topic. The languages in current use just seem to be pretty worthless for working at a logically high level (where the types, classes, etc. are less important than the business rules that glue them together).

    I am really not sure where I want to be in the industry any longer. I love the work I am supposed to be doing (I have taken a bit of leave from my official job duties lately, to support the app when it went live, it required a lot more support than we thought it needed, and I was one of 2 people with any understanding of it at a technical level, but our new support person starts tomorrow!). But I feel so trapped by the state of the industry. I am supposed to conjure up the technical architecture for this giant, monolithic system that we are selling as “you tell us what you want, we’ll make the changes”, plus we have already run into the wall that we need a single deployment for all customers (it is an application we are hosting). So I need a way to have customer-specific functionality and logic override the default install, and determine that at run time. In other words, I need something like XAML. But I also cannot have a system that shuts down the whole server to do aa deployment for a one-customer change (a widespread upgrade is fine), so I need to do exactly what you talk about here, dynamically load entirely new sections of code into memory without a recompilation (or a background recompile that puts users on hold while the pieces reload into memory).

    The direction that I am seriously considering is a general, overall framework of C# + XAML, with a dynamic language (Iron Python, Perl, Ruby) lurking in a database or configuration system that I can change at run time and have reloaded and immediately take effect. The catch is simple: there are only a handful of programmers out there who:

    1) Are capable of programming this vision and getting it right
    2) Won’t fight it tooth and nail, but will see the sense in it and follow along

    In a way, a system like this would be exactly the “programming media” system, or close to it. The main portions of the system would be nailed down, while the customer specific business logic (the portions most subject to bugs and the need to change) would be fairly changable, without needing to go through the pain of a full deployment to make those “draft changes”.

    J.Ja

  7. Re: “You tell us what you want, we’ll make the changes”

    I know something about what you’re talking about. At a place where I used to work they switched from a pure client/server model to an ASP (Application Service Provider) model. They still have some client/server features (like a thick client), but they switched to a web interface for supervisory functions. Another change was they decided to switch from having the customers host their own data to the vendor hosting it. Further, most of the business logic was switched to the vendor’s servers (as opposed to the thick client). The business proposition they put to customers was pretty much, “Here is our system. Take it or leave it.” Several customers found ways to adapt their business process to what the vendor offered. The vendor offered customizations to customers, but at a steep pay rate. So the vendor had some leverage in getting the customers to comply with the standard process they had set up. I’m not sure how they were able to do that, actually. I think they had competitors in their space, but this vendor was an early leader, and perhaps they had built up quite a bit of customer loyalty. I don’t know. Another aspect that might play into it is the vendor is in a vertical market (ie. highly specialized for a particular industry), so the competitors might be few. I was never clued in on that when I worked there.

    I see what you mean about trying to compete with established players. What comes to mind for me is Microsoft Word. Word started out as a DOS app. that was only a little better than DOS Edit is now. It allowed you to set margins, fonts (plus superscript and subscript, but you couldn’t pick the font size), and paragraph formatting, and search the document. That was it. WordPerfect was far better, but I’m sure it was a lot more expensive, and it had the same problem you’re talking about. Most people didn’t use most of the features. The first version for Windows, “Microsoft Word for Windows”, had the same features as Word for DOS, plus you could pick font sizes. I don’t remember exactly, but I think you could copy/paste graphics into documents. Neither of these versions had scripting capability. It was at the level of WordPad. The first major upgrade to Word was Version 6, I think, which ran in Windows 3.x, and it introduced scripting, mail merge, label printing, OLE embedding of documents, etc. I think it was basically the first version of what we have come to expect in Word these days. What I’m getting at is they started out simple and cheap, and over time grew to more sophisticated and expensive. They scaled it up over time, and eventually beat WP. Some people say the reason they beat them is they pulled the rug out from WP by switching support to Windows, instead of OS/2, and so beat WP to market on that platform. Maybe there’s validity to that. I don’t know. Personally I think it was because Microsoft started out focusing on the features people used most, made them easy to find, and for cheap. I remember WP was NOT easy. People had plastic templates they’d put on their keyboards to remember all of the key combinations you had to press to access each function. Knowing it was a genuine skill.

    From what you’re saying it sounds like you don’t think this strategy would work, because most customers already have the basic features, and they’re not going to switch to something else with the same feature set even if it’s cheaper than what they have. I use the Microsoft example because somehow they managed to pull it off. Maybe the ease of use was the draw?

    Re: Cannot have the system shut down to support specific customers

    I know what you’re talking about, but I don’t have a lot of experience in this area. ASP.Net installations were relatively painless for me, because I didn’t have to shut down the server, but I also did installations late at night when I knew no one was using the system, so there was no “transitory” stage for the app. where old transaction logic would need to be deprecated out. I could just update the whole thing at once. This included making database changes. I imagine in your situation that’s probably the one area that would keep you up at night.

    I remember from the presentations Microsoft gave they said that not having to shut down the app. or the server was a given, and that you could update modules without interfering with a transaction that may be running at the time, using the old logic. It sounded to me like what was happening is that the .Net runtime would JIT the module, maybe keep it in cache, and would only look for an update to it once it was done with a transaction that was in progress on the server with that module. I assume that once the next transaction starts with it, the runtime is going to pick up the new version. That sort of thing.

    Re: Making customer-specific modifications

    A solution that I’m sure is commonly used for this sort of thing is to structure the app. so that it’s data driven. The app. is structured as a bunch of little engines that respond to configuration data in the database, so the data specifies what data will appear in forms, specify rules of how they will be validated, what order those forms will appear in, how reports will be formatted, etc. That’s one way of doing it, but once you get into decisional logic (“if this happens, then do that”–maybe I told you about this before on your blog), then you start creeping into territory that can become unmanageable, because now you need, essentially, a debugger for your system.

    I know because I was a developer on a database-driven application framework years ago that had decisional code put into the database. Two things were a pain about it: debugging, and producing reports. The data was stored in a non-relational fashion, and so no third-party reporting engine worked with it. For a while we had our own reporting engine, but that became unsustainable. Before I came to work there they had even tried their hand at creating a kind of IDE for the framework, but I guess they didn’t end up using it. This is where having a more advanced language, like you say, becomes the better option.

    If you take the data-driven route, stick to a “configuration file” approach where you set parameters–more of a declarative model. Leave programming logic to the programming languages.

  8. This was interesting to read. I’m someone who comes from a background in writing (as a journalist, editor and failed novelist), though I always have had one foot in the world of software development as well.

    One thing I see we both share is a certain liking and perhaps occasional astonishment at Alan Kay. Kay reminds me of two rather famous British psychoanalysts, Wilfred Bion and D.W. Winnicott. (If you are interested, the best source of their works is http://www.karnacbooks.com.) Bion had an astonishing capacity for knocking over our current perceptions of the world in such a way that you just had to agree with him. Winnicott is almost solely responsible for our understanding of the importance of children’s play, that it is not about “learning” so much as developimg ways to “take in” the external world, the world of the self, the world of the family. I think Kay has similar ideas about the interactions of children and computers.

    I think when Kay writes that we have yet to develop software engineering, he actually likely means just that. I think our difficulty is that we somehow expect it to be something like other kinds of engineering. It may be that it is somewhat like writing. It’s just as likely it is not, but imagining it as being somewhat like writing is a kind of “antidote” to having to think of it only as engineering.

    (One big difference from any kind of engineering is that other engineering doesn’t have as much traffice with the issues of change. A case could be made that software is actually a modern descriptive system for dealing with issues of continuous and rapid change.)

    My guess about the long future of software engineering (though I’m not at all sure Kay would agree with this) is that the separation between what is sometimes called the “implementation model” and the “user model” is artificial. The future of software is perhaps when we can bring these two easily into synchronisation.

    If you think about Lisp, this is a step in that direction, not actually for the end users, but perhaps for the actual builders of the programs. We don’t often think of it this way (or at least I don’t), but there is also a “user” and “implementation” model at work in the actual use of programming languages. Your piece on the oddities of C is a good example of this. The programmer’s instinct — our user model — is that something like 0

  9. @Scott Lewis:

    Great comments! Yes, we share that astonishment with Kay. I am amazed by his mind. He has such a deep understanding of so many subjects. I guess to sum him up, he’s a renaissance man. His vision is very far-reaching as well. He sees things coming before a lot of people do. I’ve speculated before that even though Kay sees Smalltalk as “obsolete”, it could stand still for the next 10 or 20 years, and you could watch as the technology that is used in the world slowly catches up with it. Though the library of functionality in Smalltalk has grown, at heart Smalltalk has barely changed since 1980 when it was “released to the world”. Some fundamental changes been made to it only in the last few years, in Squeak. It’s still more powerful than most of the programming languages out there. As for convergence in the programming world, the same could be said of Lisp, though our industry stubbornly clings to C-like syntax, which is okay. Syntax in a programming language is just style. It’s the semantics that should get more powerful.

    Kay said something similar to what you were talking about, that “implementation model” and “user model” should be similar in a paper he recently released. He said something about how one of the challenges in writing software is there’s a translation step between a software spec. and actual implementation. He predicted that in the near future he and the people he’s working with should be able to develop a system whereby you can “just ship the spec.”. I don’t think this means writing software in English. I feel I always need to be careful to say that, because he’s not talking about AI. I think he’s talking about increasing the level of abstraction so that the language the spec. is written in is still using a kind of formal logic, but it’s at a high enough level that it won’t just look like a math proof.

    In one of my previous posts I quoted Kay as saying that the way we write software today is “engineering of a sort”, but it’s like what the Egyptians used for building pyramids, which produced “a trash pile covered over with some limestone”. He’s said something similar to what Gabriel said, which is that engineering has always preceded science. Engineering at heart is just a method for doing something that’s basically been found to produce a desired result. It can come about via. trial and error, and often does. There doesn’t have to be any real reasoning behind it, though that’s desirable. What science adds to the mix is it helps discover why a particular method works or doesn’t work, and it can find methods that work better, once a body of knowledge in that domain has been established. Electrical engineering and such are mature engineering disciplines because they’re backed by science.

    Re: Lisp enables a closer merger of models

    I agree, primarily because you can create DSLs in it. The same goes for Smalltalk, as my post illustrated.

    Re: C

    A language’s semantics have to be understood to be used effectively. Some of the stuff that was written about in Dyer’s article just illustrated a fundamental misunderstanding of the language. So I thought some of the criticism was undeserved. It’s like saying a cat is “defective” because it licks itself and hacks up hairballs. What I say to that is, “That’s just how it works. You’re blaming a cat for being what it is.” Better to understand it than curse it, because it’s not going to change. If you don’t like it, don’t use it. As I illustrated in my post, K&R C was rather deceptive. It had a tendency to make people assume it cared more about types than it did. At least ANSI C really DOES care.

    This is a bit tangential, but I think the reason I saw others struggling to use K&R C is they came along earlier than I did in the school curriculum. Before I came along C was not taught. Some courses used it anyway. It was just expected, since students were taught a variety of languages, that they would “get it” and take the time to learn any language they were expected to learn, on their own. The faculty wanted to inculcate this ability in us, to be flexible, to not just learn the “hot” language of the day and stick with it for the rest of our careers. I took the first course my college offered on C, so I had more things explained to me.

    Anyway, it looks like your post got cut off. Sorry about that. It’s not a length limit. WordPress reacts to less-than and more-than symbols, thinking you’re creating an HTML tag. It also doesn’t like backslashes for some reason. I think if it sees what it thinks is a malformed tag it just wipes out whatever is after it. Not too intelligent.

  10. I read Current Notes back in the day. Congrats on getting published there. One of my regrets is never getting a type-in program published in Antic. I’m a little young to have pulled it off, but I’m not that much younger than J D Casten.

    Slightly off topic, but there will never be a computer that I love as much as the Atari 8-bit. (M.U.L.E. with four joysticks. Nobody can top that!)

  11. Hi lispy! I tried getting a type-in published in Compute! Magazine when I was in high school. It was rejected though. I always thought that Mad Libs should’ve been computerized rather than in paper books. So I created 3 programs to write, edit, and “play” them on the Apple II. I called the package “Ad Libs”. Compute! never really told me why they rejected it. Maybe they thought Price/Stern/Sloan would come after them. It was a legitimate concern in my mind, but I thought I’d give it a shot. I wrote another program on the Apple, called “Week-At-A-Glance” (I think), which I intended to try and publish, but it got so large that I didn’t bother. It was my first attempt at creating a user-friendly program. That’s what made it so big.

    The Atari 130XE was the first computer I owned. I got it in 1988. I had access to other 8-bits before then. One of the first computers I used were the Atari 400 and 800. The local library had a computer lab where people could sign up for time. The only ones they had available that preteens could use were those Atari models. The first time I saw computers in schools was when I was in junior high. They got Apple II+’s and IIe’s. So I got to use both Ataris and Apples.

    I took my 130XE to college and used it there. I upgraded to an Atari Mega STe around 1991/92. I used it until 1997 when I got my first PC. I still have both of them.

    I remember playing M.U.L.E. on a friend’s C-64. I loved the intro. music. It had that funky beat. 🙂 My friend had a bunch of pirated games. This was one of them. We tried playing it a few times, and it was fun, but at some point the game would always break down. It didn’t crash I don’t think. It just wouldn’t advance to the next level or the next turn or something. Anyway, we never got all the way through it.

    Good hearing from you.

  12. Pingback: Saying goodbye to someone I never knew « Tekkie

  13. Pingback: Identifying what I’m doing | Tekkie

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s