Lisp: Surprise! You’re soaking in it

This is the best introductory, conceptual article on Lisp I’ve ever seen. It’s written by Slava Akhmechet.

Lispers have long felt that it’s very difficult to get lay programmers to understand Lisp, because there’s no common context to help people relate to its unique syntax and powerful features. Slava went against the grain and managed to pull it off. His message is “XML = Lisp with angle brackets”. Or, put another way, “You’re soaking in it!” Are you using Ant? Are you using Hibernate and/or Spring? Are you using CodeSmith to generate code? If so, then you are already halfway to understanding Lisp, because even though the syntax is different, the underlying concepts are the same. XML is your code, and the XML processors are analogous to a Lisp interpreter running your code against some pre-loaded libraries. The difference is Lisp gives you more flexibility and control (ie. more power).

If you don’t understand Lisp macros, but you know how to program in C/C++, he helps you make the transition from C/C++ macros to Lisp macros.

He sums it up with: “Lisp is executable XML with a friendlier syntax.”

Once you understand these analogies, he introduces you to Lisp code.

If Lisp has mystified you, this is a great introduction that will help you get on your way. It won’t make you totally comfortable with it right off the bat, but it will help you get over the initial comprehension barrier. To increase your comprehension, and to learn what Lisp can really do, the following books have come highly recommended to me. I must admit I haven’t read them yet: On Lisp (online book), by Paul Graham; and Practical Common Lisp (online book), by Peter Seibel.

Some others that cover Scheme, a Lisp dialect: The Little Schemer, by Daniel Friedman; and Structure and Interpretation of Computer Programs (aka. “SICP”, online book), by Harold Abelson, Gerald Sussman, and Julie Sussman.

Advertisements

21 thoughts on “Lisp: Surprise! You’re soaking in it

  1. Duh! Someone pointed me to the SICP book a week or two ago. As soon as you explained it a bit… I am 99% sure that this is the book that I used with EdScheme. It was the #1 biggest influence on me as a programmer today. PERIOD. If this book is the same one that I am thinking of, if you have not read it (or a similar text with similar exercises of building a full language from a few primitive statements) *and* worked through the problems, you are very unlikely to become a “great” programmer or reach the “architecht” level.

    I interviewed a guy today. His goal was to become an architect. He has not even read the HTTP RFC. How can you want to be a Web architect without understand how the core protocols work?

    J.Ja

  2. It was one of your blog posts, plus some recommendations from others that pointed me to SICP. So yeah I think it’s the same book. I said on your blog post on “The distinct lack of fundamental programming theory resources” (at http://blogs.techrepublic.com.com/programming-and-development/?p=379) “I don’t think I had this”. I said the one course I had that resembled what you described was a computer systems course, where we studied hardware architecture, the fact that microcode runs a CPU, and how memory is fetched and refreshed. We programmed in assembly language in this course. We built up some library routines throughout the semester, and at the end we built a couple of apps. out of them. I got exposed to the different flavors of programming languages out there, how language compilers worked: lexical scanners, parsers, finite state automata, stack machines, bootstrapping, etc. I took a course in compilers, but it was a graduate level course at the time, which meant most undergrads never took it. That, along with data structures, algorithms, operating systems, and computer graphics, was essentially my CompSci education as of 1993. What I’ve been in the process of doing is upgrading, and in a way shedding my old CS education. Believe me, I’m reorienting my mind!

    To tell you the truth, I’m only lukewarm about becoming an architect, because their traditional role is they don’t do any coding. I’d like to understand things at the architectural level while coding at the same time. To me this whole idea that you’re expected to reach a point in your career where you don’t program anymore, and you become a manager is bogus. I mean, it’s fine to become a manager, if that’s what you want. I’m talking about the expectation. All that does is perpetuate the way things have been done, where you have a few managers that think conceptually, formulate budgets, come up with project schedules, and create system designs, and you have a legion of lower level workers who continue the same old process of translating these abstract designs into computer/processor instructions. I like the approach of giving your idea to the computer using some kind of formal logic. It could even be part of the design spec., and letting the computer “figure out how to execute it”. Why continue the old process that has a propensity to not work?

  3. Thanks for the posts, and reminding me of Scheme. I’ve been doing a lot of interviews if candidates lately, and have added some basic SICP items to my interview questions… “Describe a simple function to produce the Fibonacci sequence” and “describe a simple function to do string reversal without using a loop”. Any programmer worth their salt should immediately think “recursion”. We are hiring for a “senior” developer, and I am just blown away by the number of people I have interviewed with 10, 15 years of experience who really just stuck with some simple C++ in the early 90’s, and have been using VB from 3.0 to present day. In a nutshell, these candidates are incapable of doing more than gluing pre-built libraries together with a rather shoddy general purpose OO language like Java or C# or VB.Net. Sad, really.

    Now, regarding your thoughts on being an architect… I think you’ve met the wrong architects! Seriously! The architect has to still be doing some coding, even if it is not actually on the project or even at work. Think about it… it typically takes well over 10 years experience to claim the architect mantle (how I got the job is beyond me… the interviewer asked me to architect a chicken, and I guess I impressed him…). So let’s say you are working with an architect who started transitioned to architect in ’95 or even ’98 or maybe even later… if that person is not staying current, well, I hope you like COM or C++. You *have* to stay current! Have I written many lines of code since I took this job? Nope. And it is hurting my ability to get this stuff right. I know that we need to be using something like XAML for this project. But until I sit down and work with XAML, I cannot architect a system using it, and worse, I will still see the world through non-XAML colored glasses! It is like how once you learn OOP, the next time you write procedural code, you refactor it much more differently than you used to, because you see the world through OOP colored glasses. For me, Perl was this language. To this day, my mind immediately thinks regex’s, hashes, and recursion can solve all problems.

    I really do know where you are coming from. Most architects tend to write very few (if any!) lines of code on their project. But they do need to be writing code. A smart employers lets their architects work on small projects of their own fancy, to keep up to date and prototype new techniques.

    Are architects managers? In a way. They certainly manage code. And if a project is big enough, they pretty much control the project timeline, since they define the critical path to getting enough code written to start testing, doing demos, and so on. They also provide mentoring for less experience developers. The architect should be doing code reviews to ensure that what was built is what was planned for. Sure, the architect may not be as great at debugging or line-by-line optimizations as the senior developers, but they understand the high level optimizations. For example, the senior developer knows to set a variable equal to the upper limit on a loop instead of recalculating it on every iteration. But the architect is the one who says, “since we’re passing a rectangular data set around, let’s use CSV instead of XML to make the parsing go faster”.

    Up until the last few weeks (I suddenly became Tier 1 – 3 support, ack!), my job from the outside looked an awful lot like a manager… my primary apps used were Excel, Word, Outlook, and Visio (the only even moderately specialized app in the bunch). I attend a lot of meetings (still do!). I spend a lot of time walking around with a coffee mug, trying to track people down, or talking on my cell phone in staircases. I get yanked out of one meeting to attend another meeting, and I am regularly double or triple booked for meetings. And I have direct communications to high levels of the company and I am asked for information that goes into strategic decisions. I make tactical decisions. And so on. But yet, I am not actually a manager. No one reports to me. I don’t worry about firing people, but I do have a lot of input into hiring.

    I really hope that I do not end up being the kind of architect that you describe. I really do not know which way that will land yet. But right now, I am still extremely hands-on, more of a “shop foreman” than a “manager”, even though I seem to be a de facto manager at the same time…

    J.Ja

  4. When you say “describe a function” that’s a little easier than writing actual code. I would’ve taken the recursion route with the fibonacci sequence, because that’s always how I was taught it. It’s the most efficient way to do it. It would’ve taken me some thought before I came up with recursion for reversing a string w/o a loop, since I’m used to using iteration for that sort of thing. To me, that’s the most straightforward approach.

    You’ve heard of how many experienced developers couldn’t solve the FizzBuzz problem (or took a very long time to do it), right? I couldn’t believe it. I took one look at the problem and snorted. Within a second I had a good idea of how to solve it. There were a ton of people saying, “Hey, look at me! I can solve it.” It was so elementary I didn’t feel it was even worth joining in. It felt pathetic. I looked back through my old Apple II programs, because I remembered I solved a problem similar to this when I was in computer club in high school. I found it. It was called “Buzz”. It was more difficult than FizzBuzz by a longshot. The problem we had to solve encouraged the use of recursion for part of it. The difference was we had a week to write up our solution, and an hour to type it in and test it.

    A problem that me and blogger Giles Bowkett have discussed from time to time is employers who look for “X years of Y”. To me, this is not a smart way to look for candidates. I can understand it with some broad skills, but not with a programming language, and HTML and the like. It’s a problem that’s existed in our business for a long time. I remember when one of my CS professors joked about it in the early 90s. He said it wasn’t uncommon for employers to ask for 4 years experience with a language when it’s only been out for two. It still isn’t. I went to a presentation given by a contractor last week, and he said recent studies have shown that people who have worked with more than one programming language tend to be more productive than people with X years experience with one language. It went something like that. I don’t remember exactly, but that caught my attention. I think those studies are on to something.

    I agree it’s a challenge to weed through the applicants. An old boss of mine complained years ago about something similar: people who would try to BS their way through the interview process. This was in the mid-90s. He was looking for some developers to work on PL/SQL. He asked me and another programmer to come up with some interview questions. He went and interviewed the candidates, and he came back with a rather disgusted look on his face. He asked me, “How would you rate yourself as a PL/SQL developer from 1 to 10?” Since I didn’t know the whole language, but I felt comfortable with it, I said something like, “A 7 or 8”. He said, “I asked one of the candidates how he would rate himself. He said a ’10’. I asked him one of your questions and he got it wrong.” He said with a laugh, “So if he’s a 10…you’d have to be a 12 or 13!” For the moment that comment made me feel good, but it reminds me of grade inflation. 🙂

    What you’re saying about architects needing to code to keep current makes sense. Reading your comment caused me to think back on my experience with people in this position, and it validated what you’re saying. The boss I worked for in the mid-90s did software/database architecture as one of his duties. After a certain point he did no coding. When I started there we worked in C. He insisted that we continue to do so, despite my prodding him to look at the advantages of OOP, and despite the fact that this was the direction that everyone else was moving in. Once he left his position and worked as a Sr. engineer elsewhere, he realized what I was talking about. A friend of mine is a system architect (last I checked) at this same place where I used to work, and he’s told me he’s done a little coding every now and then. They’ve been using Java for the last several years.

    You’ve heard me say this before. What I expressed earlier, as more of a wish, is that the workplace be structured differently such that the roles of architect and programmer were merged more, via. a more abstract programming language. I think it could be done with existing dynamic languages. I just haven’t tried it yet in an environment like yours.

  5. @Justin:

    Just a suggestion, but instead of looking at XAML, you might want to check out Silverlight from Microsoft. It’s the new name they’ve given WPF/E. The cool thing about it is you can program in Silverlight in C# or VB.Net, rather than XML. In the near feature they’re adding the DLR (Dynamic Language Runtime) to it, which will allow developers to use Python, and maybe Ruby code down the road, whenever they get a Ruby compiler working.

  6. I really do need to take a look at Silverlight. I need to know how it relates to Web apps, since this is what I am currently architecting. I like XAML. A lot. Especially since the application I am working on is heavily customizable during the customer integration. XAML answers our need to be able to easily rearrange workflow on a macro level while maintaining it on a micro level (in other words, put the major blocks or workl in any order, while maintaining the business logic of each work black). It also allows us to add new funcitonality for a particular customerr, while not affecting the base product or existing installations, yet simultaneously allow us to make that new functionality avaialble to other customers.

    J.Ja

  7. Great site- congrats on all the activity

    [Thanks for the compliment. I’m trying to hold down the commercial linking in the comments, unless it relates to what I posted. So I deleted your weblink. If you have a blog or personal web page that mostly contains original content, you may use that. If you have a few ads on the site, that’s fine. My comments policy is in my About page. — Mark M.]

  8. Pingback: Top Posts « WordPress.com

  9. what about, instead using a recursion, a programmer uses car, compute the next fib number, push it in the list, repeat till n, and at the end reverse the list?

    how do u rate such guy against another who uses recursion? does his method of using list-based neccessarily and sufficiently show his ignorance of recursive method? or just he cares about fast code?

  10. Xml really, really sucks for procedural code like that found in ANT. Telling me I can have the same sucky experience throughout by code base by using LISP isn’t a good idea.

  11. @Jonathan:

    The question becomes then what else are you going to use to accomplish your task? My understanding is Ant was originally created for doing project builds, to substitute for using Make, which has its own shortcomings. If Ant isn’t for you, and you’ve found something better, more power to you.

    I agree that XML as a programming language sucks. I never got the “We’ll use XML for EVERYTHING” mentality. I think it’s nuts. XML is fine as a data description language and data transport medium. That’s what it was designed for. I wouldn’t use it for anything else. The point is, though, that Lisp does what XML tries to do, but better. It was designed as a programming language. It has a less verbose syntax, and it gives you more power to do what you want to do than the typical XML processor does.

    The reason XML has been used for some things like code generators is it’s a declarative language. That particular property is well suited to the task. Lisp can be used the same way.

    The main idea I wanted to get out to people is if you’ve been curious about Lisp, and you knew XML, then you already understand enough to understand some basic things about the language that might’ve confused you before. Slava helps make the connection in that regard.

    The one thing I’d encourage people interested in Lisp to do is to get a Lisp-friendly code editor. One of the things about the less verbose syntax is it’s difficult to keep track of the parentheses. Having a good editor for that can really help. There’s a free package I’ve heard about called “LispBox”. It contains Emacs, which is an old standby for programmers. The link to the page I give here also contains a link to the online version of Practical Common Lisp. It focuses on the practical, modern day project tasks you can accomplish with the language.

  12. xxx –

    I often add the explanatory phrase, “without a loop”. What the question really tests is whetherr they learned “programming” or if they learned “computer science”. Someone with a background in “computer science” is taught sometime during their first semester to computer Fibbonacci sequences with recursion.

    J.Ja

  13. I second Jonathan Allen above.
    I simply hate the way that you have to “develop” in XML all around every damn Java framework. Saying that programming Lisp is like XML isn’t exactly selling it.

  14. @Philobuster:

    It wasn’t my intent to say that Lisp is exactly like XML. I explained that quite plainly in my post. I said if you understand XML, you already understand some things about Lisp. If that turns you off, there’s nothing I can do for you. Would you have preferred I talked about how Lisp is like parse trees? I think even fewer people would’ve understood that concept.

    I was trying to say the syntax and some of the semantics are similar. I didn’t focus group this mesage, and neither did Slava. Hopefully most people got it, but I can’t please everybody.

    I can understand a bad feeling about a technology turning you off to things that “are like” it. I’ve done the same thing. I’ve found it’s usually a mistake. I used to hate Lisp because I couldn’t grok it. For a time I swore off functional languages altogether, because trying to learn them gave me a headache. The problem was I was trying to learn them the wrong way. I needed to change my POV and assumptions first.

    What impressed me about Slava’s article is he took advantage of something that a lot of people have already learned about and used it to help people relate to Lisp. I wish someone had done this years ago. It would’ve helped me out.

  15. I learned Lisp (via Scheme) through the backdoor. No one tried showing me that it was like anything I knew. More importantly, they did not make a big deal about it. I learned from the SICP book, but since I was still so green, all I knew was that we were learning a new language, which seemed fairly different from the previous two languages we learned (BASIC and COBOL). Since I had so few pre-conceived notions, it was much, much easier.

    I have been trying to re-learn functional programming, 12, 13 years later. My mind is having a very difficult time wrapping itself around the idea. What is so frustrating here, is that *I used to know this cold!* At one point, the idea that all “variables” are really defined as functions (“x is not really equal to the number 5, x is the name of a function that always returns the number 5”), list processing, and lambda calculus in general made total sense. Years of Pascal, Java, VB.Net, HTML, JavaScript, C, MivaScript, ActionScript, XML, SQL, FoxPro, PL/SQL, VBA, T-SQL, and a dozen other languages later, and the only experience that comes close to being helpful to my working with functional programming again is Perl, and only marginally slow.

    I would almost say, from my own experience, that the longer you go without learning functional programming, the less likely you are to understand it, like it, see the need for it, or even see where it could be useful. Working in procedural (sorry OOP fans, OO code *always* needs to become procedural code at some point!) code trains you to think a certain way. It is not a bad thing, just an “is thing”.

    I just wish that this had not happened to my head. I know in my head that this is useful stuff, *since I have done it before*. I just have an extremely difficult time with it now, thanks to so many years away from it.

    J.Ja

  16. @Justin:

    When I was first introduced to Lisp, it was my sophomore year of college. I had a terrible teacher. He didn’t even teach us what a functional language was, so I had no basis of understanding. The only assigments he gave us in Lisp were a couple of hard ones, to boot. I think we only spent 2 weeks on it. A few years later I took a senior level course in programming languages, and I had a good professor. He taught us what a functional language was, and then some things about Lisp finally made sense. We didn’t use it in that class, so I didn’t get a chance to “conquer” it. We used SML instead, and for once functional programming was fun.

    Last year I was inspired by Paul Graham to give Lisp another shot. I installed GNU CLISP. It contains a rudimentary tutorial in a text file, which made a LOT more sense than the material I got in that sophomore class. I dug out my old Lisp assignments and I solved one of them before I lost interest in Lisp for Ruby and then Squeak.

    I can tell now that the professor in my sophomore class was trying to teach us about functional languages, because in the assignments he specifically forbade us from using setq. This forced us to program as if we were using a pure functional language (which Lisp has not been for many years), using recursion.

  17. Mark –

    I finally (I have only been blogging about it for about a year now) gotten around to actually using F#… wow! I just submitted an article to TechRepublic on it (a really basic “how do I install it and write a dinky program to get me started?” thing), so that should be out soon, hopefully. Although, lately, they have been taking a while to publish (used to be 1 week, I sent one in 2 weeks ago and still no word on it). Inspired, I decided to use a Fibonacci sequence function for my demo; I used the fibonacci sequence from the Microsoft documentation, and added it a few more lines to demonstrate calling the .Net Framework from F#, as well as showing off lazy evaluation (which is very cool, by the way, but confusing to debug if you are used to other languages!).

    J.Ja

  18. @Justin:

    Thanks for the heads up. I’ll look for it. As I recall you were also going to do a post on the Indian education system (the one in Asia). Still interested in that. 😉

  19. Pingback: The 150th post « Tekkie

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s