Thoughts on the 3D GUI

I got into a discussion with Giles Bowkett last week about 3D GUIs (see the comments on his post). It provoked some thought for me. Giles pointed me to an article on Jakob Nielsen’s assessment of 3D UIs. It’s almost 10 years old, though. Nielsen makes some good points, but it sounded to me like it was in the context of what was out there, not so much its potential.

When I imagined what a 3D GUI would be like, around the time this article was written, I tried to think of uses for it from a home user’s perspective. The first thing that came to mind is it extended “wallpaper”. Rather than having a background image on a desktop, you had a whole environment you could fashion to your taste–to actually decorate. That doesn’t sound very useful, and in fact sounds like a huge waste of productive time.

When I thought about it last week, I came to the conclusion that it would allow for a more flexible environment. One of the applications Nielsen talked about for 3D was modeling interior decorating. What if an interior decorating app. wasn’t its own app. like we know it today, but instead added capabilities to the existing environment? What about these 3D role playing games? They could extend the environment as well, rather than just being an app. in a window. Think of the “holodeck” as a metaphore. Everything that’s 3D could live in the environment, or in a sub-environment. Everything that’s 2D could live in a window in the 3D space, but could be found immediately. It wouldn’t have the problem in real 3D space where you can actually lose something :). A question that comes to mind is how do you switch apps like you can in 2D today, if this is the way things are done? Well, I think that would necessitate having multiple environments you could switch between quickly.

It could make apps. more extendable, because there would be a common programming metaphore for adding new elements to the environment. Say some third party company wanted to create an add-on to a game. They wouldn’t have to conform to the game developer’s API, because the original game was just a modification of the standard environment. The third party mod could do the same.

I referenced Croquet as an example of what I imagined. It’s not a pure 3D environment. There are still 2D elements that show up right in front of you no matter where you are in the space. I think that’s practical. I think 3D UIs can add more capabilities to what we’d call “thick client” computing today, but I think some shortcuts need to be added to the environment so that you don’t have to look and travel everywhere just to get something done. I agree that makes it harder than 2D today.

A question I have about it is, is it easier for people to think of “travelling around” to look at a collection of things, or to scroll a window up and down, and side to side?

4 thoughts on “Thoughts on the 3D GUI

  1. Jakob Neilsen is nearly always right, in my experience. That being said, his article refers pretty specifically to UIs where they try flying you through data, rotating things around to see different angles, etc. While his article may be older, the principles themselves remain pretty constant, which is true of nearly all usability work. What is interesting is to see that usuability guidelines developed during the green screen era apply equally well to the Web or modern desktop apps, where the elements are the same (font size, data display techniques, etc.). Indeed, the Apple interface guidelines developed over 20 years ago still hold true today, and guide smart developers.

    So while I agree with you that 3D is a very good idea for certain applications, I feel that the effort of trying to navigate a 3D space with 2D input devices just to find a file or type a document, or probably 95% of what users do with a PC on a regular basis is probably not a good idea.

    What you envision, the common APIs and such, Vista is a huge leap in that direction, with the Aero interface. It’s getting there, very quickly!


  2. So while I agree with you that 3D is a very good idea for certain applications, I feel that the effort of trying to navigate a 3D space with 2D input devices just to find a file or type a document, or probably 95% of what users do with a PC on a regular basis is probably not a good idea.

    I might be more acclimated to this than most PC users since I used to play Quake frequently. You most definitely had to navigate a 3D environment for that, and quickly! This was true for any FPS (first-person shooter) game. There were some constants that you could count on. For example there was “gravity”. So you always knew if you were looking up or down, but intending to move laterally (forward, backward, or side to side) that’s what your vitual self would do. Also things would fall to the ground if you threw them. Certain objects were always on the “floor” of whatever surface you were on, like weapons or power ups. I’m using a video game metaphore, because these are the only programs that a sizeable population have used that kind of models what I’m talking about. I think it is possible that a 3D UI could work for people, so long as it followed rules that people were used to, and they were used to the controls. A lot of 3D games I’ve played on the PC use a combination of keyboard and mouse controls. And it works quite well once you get used to it. The default keyboard layout for movement the last few years has been an upside-down T on the keyboard (W-forward, S-backward, A-left, and D-right), and using the mouse for looking side to side, and up and down, space for “jump”, and the mouse buttons do things, like “fire”, “pick up object”, “zoom in”, etc.

    When I first was learning this control model 9 years ago it took some practice, but it eventually became natural for me. I think the keyboard layout on computers would have to be modified to make it work well. It would be more intuitive to have an “arrow-pad” that people could use for lateral movements with one hand, so that they could use the mouse (or equivalent) with the other.

    As for “95% of what users do with a PC on a regular basis”, that’s what I’m talking about opening up. With the 2D interface we’ve gotten to the point where we’ve mastered the medium. With this, people write documents, send/receive e-mail, and browse the 2D web. I understand that with WPF, with its integration with Direct3D, etc. that it’s possible to create a richer experience, but it’s 3D within a 2D framework. Why constrain the 3D realm to the 2D world? I realize this is far reaching. Practical 3D is just taking its first steps in this platform. I guess what I wonder is, could we someday have a 3D web, where the internet is integrated seamlessly into the environment?

    Isn’t it possible that computing offers more possibilities than the “95%” you’re talking about, but we just don’t know it because we’ve accepted the barriers that the 2D interface imposes, and say, “This is all PCs are good for”?

  3. It’s time for computers to go beyond the simple screen and mouse interface. Wouldn’t it be awesome to have an augmented reality 3d interface?

    This is the technology of the future, not a 2d machine acting like a 3d one.

    Put on some glasses (transparent, not the heavy 80’s kind) and a blue tooth glove for a mouse.

    When 2d interaction becomes required (say, when typing a document) call up a virtual keyboard and touchpad.

    Such hardware is still in it’s infancy, and not yet combined properly. But given time, I see a combination of the two fields of research paying off.

  4. @Stan:

    I used to think that VR was going to come on the scene for a wider market as CPU speeds increased. I asked a few people about that several years ago and heard about augmented reality. I agree that sounds more viable in terms of usefulness. With the speed barrier reached and now we’re confronted with multi-core technologies, I think that’s a challenge. Same goes for my 3D interface idea in this post, but there have been some successes along these lines, like Croquet.

    Most of our programming systems have a legacy from a time when we felt the need to optimize for the hardware. Most programmers are trained with the idea of using a single core architecture, as are most programming systems. There are efforts underway now to try to make multi-core programming easy, to adapt the existing models to it, but I think ultimately it’s going to require a different model of programming altogether. That’s going to be tough.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s