Paradigmatic Change in UIs

In the first wave of computing there was virtually no user interface, comprising little more than instructions written on punch cards which were loaded into a hopper then read by the computer with the results given to the user via a printout. The second wave introduced what we now know to be a user interface, a display with characters and graphics along with a keyboard and mouse. The display evolved from simple monochromatic characters to full color graphics with ever increasing resolution over time and the keyboard and mouse technologies evolved to be smaller and integrated with trackpoints and trackpads. However, the basic elements of a small TV like display, with a keyboard and mouse beneath it, have remained constant for a remarkably long time. What appeared on that display and how a user interacts with it has remained surprisingly constant as well, especially from the time that the concept of programs running in separate windows was introduced.

While there were minor predecessors, the major shift into an entirely new form factor came with the introduction of the iPhone. We're now so used to smartphone UIs that many people forget that we hadn't ever experienced one until Apple introduced its game-changing device. The smartphone form factor existed before the iPhone, but Apple totally redefined it. Once users were used to the iPhone user interface, the adoption of the larger form factor iPad was incredibly easy because it was virtually identical. Like the smartphone, the tablet form factor also existed prior to the iPad but again Apple redefined it dramatically. Key to that redefinition was the perfected use of multi-touch. Interacting via multi-touch is so pervasive now that it isn't uncommon to see people walking up to screens in places like hotels, airports, and stores expecting to be able to interact with them with touch only to be really surprised and disappointed when they turn out not to support touch. That's when you know that we've experienced a paradigm shift as a society. Touch has been in university labs for decades but it took Apple's dedication to design excellence of the entire user experience to perfect the technology to create this paradigm shift.

Another paradigm shift in interaction modality has just started. This one involves the use of speech. Again, speech technology has been around for decades and has been used commercially successfully as well but mostly in niche markets like voice response systems and dictation systems. Apple's Siri is still in beta, a product designation Apple very rarely uses, but promises to do for speech technology what the iPhone did for touch technology - make it a pervasive and paradigmatic change in society.

There are two major insights to glean from these fairly recent advances. The first concerns how these changes took place. In each case, the basic research and foundational technologies as well as even some commercial applications existed for decades prior to the paradigm shift. It was Apple's approach to design that made the difference. The design of everything, from the industrial design of the physical elements of the device (glass, case, bezel, etc.), the visual and interaction design of the operating system and key apps, the engineering design of the internals (processors, memory, battery, GPS, etc.), the manufacturing design of the production line, the design of the website and app store, the design of the content review process, the design of the payment and app download system, the design of the stores, the design of product support, all the way to the design of the product secrecy and product announce/launch systems. Many people like to jump to simple conclusions that these paradigm changes were brought about by this or that individual element but I believe that it was Apple's focus on the total customer experience and all the elements that impact it that was critical. Designers planning a product that they hope will transform an industry need to focus on all of these aspects of design.

The second major insight to glean from these paradigm shifts is the need to rethink how all future products in any market should fit into these major paradigmatic changes in form factor, device, and interaction modality. Designers now need to understand deeply how users are using these technologies in order to design products optimally leveraging them. This is a challenge for many because, as pointed out above, form factor, device, and interaction modality hadn't changed virtually at all for decades. However, these recent changes are so profound that it really does require designers in any market to sit up and take note and consider how users in their markets may be changing.

Having explored paradigm changes we've already witnessed, let's give some thought to where these may go in the future and what other paradigms we may witness in the future. While the computer, smartphone, and tablet form factors and the touch and speech interaction modalities have mostly developed independently, the emerging trend is for them to become more consistent and a future paradigmatic change may involve them integrating deeply. We're seeing the beginnings of changes being made largely to computer operating systems like Apple's OSX to make them more similar to device operating systems like iOS. Apple is making the change gradually with each update to the OS, which is a wise approach that minimizes the magnitude of the change but still moves drives consistency. We're also seeing the very beginnings of a move to integrate form factors, devices, and interaction modalities. Responsive design is part of this trend, as is the enablement of touch and speech pervasively across devices. We're also seeing some degree of cloud based seamless access integrating content and data across devices. We're also seeing that integration spread to even larger form factors like TVs and digitally enabled physical window panes. Some call this a post-PC era, my sense is that we're witnessing a plethora of form factors which in PCs, ensuring that each of these can suit the wide characteristics and contexts of use into the future.  

We're living in exciting times that require designers to be fully aware of, intimately knowledgeable about, and be able to leverage the benefits of these incredible paradigm shifts in technology and people's use of them.

Apple's Post-PC Era

I was just setting up a new iPad for a family member who wanted to use the new device she received as a gift as a stand alone device without needing to connect to a PC. Regular readers of this blog will know that I'm usually pretty positive about Apple's designs and, of course, I'm certainly not alone in that view. Apple is incredibly good at hardware design and the integration of software with its hardware. However, the company to date has had some difficulty with the design of its cloud capabilities.

Apple has recently used the term "post-PC era" as a basis for its mobile and cloud strategies and communication. Recent updates to iTunes and iOS have provided some independence of mobile devices from computers, but only partially. You can now push updates from a PC-based iTunes to an iPod Touch, iPhone, or iPad without using a cable but you still have to use a computer in the mix.

There are other use cases that have absolutely no support on iOS devices. The most annoying and concerning one for me is the inability to subscribe to podcasts on Apple mobile devices. You can download individual episodes but there is no way to actually subscribe within mobile iTunes. With the increasing popularity and use of mobile devices (see last blog post), the inability of podcast listeners to subscribe via mobile devices is not only inconvenient for users, it is also a major problem for those creating podcasts, like me. It is really strange that Apple hasn't provided the capability to subscribe. All it would require is to include the same "subscribe" button to the right of the podcast name and artwork on mobile iTunes as is included on the computer-based version of iTunes (see visual on the right). It was pointed out to me by a friend on Facebook that there are separate apps that provide this capability with one called Downcast that is particularly good and one that he recommends. That's a temporary fix but I still think that this missing function needs to be included in the base mobile version of iTunes. I know that it is a herculean task to develop mobile device operating systems that never need to be connected to a PC and to design that really well. However, leaving off the subscribe button seems to be a rather strange oversight on Apple's part. 

To the listeners of my podcast who may be reading this, I'd suggest that you use the iTunes on your PC to subscribe to the podcast or download an app like Downcast to essentially replace the podcast part of mobile iTunes on your iOS device. I do hope that Apple addresses this problem soon in an update to iOS so that we won't see bifurcation of the podast audience on Apple devices and also no longer have a single reliable place to determine podcast popularity and feedback.

Mobile Design Innovation

I heard the buzz about a new iPad app called Flipboard a few days ago so downloaded it to see what all the talk was about. Due to its popularity, the company could only make some of the capability of the app available upon download and the remainder, Twitter and Facebook integration, after a day or two. I knew immediately when the app launched that I was experiencing something entirely different. The splash screen shows full screen photos with a slow animation cross the screen of highlights from the various content sources you've selected. This approach to splash screens is novel and immediately grabs the user's attention with directly relevant content using often stunningly good photographs. Please note we're not even really using the app yet but have been drawn into it in an amazingly engaging way.

After you swipe to the left across the screen, you see the main Flipboard content channels. There is a starter set of content sources but you can select whichever ones you'd like from a fairly long list of candidates, including Wired, The New York Times, Fast Company, The Economist, Nature, Engadget, and The Onion.

Two special sources are Facebook and Twitter. These are the ones that you need to request access to during the early days of the launch of the app. Once you get your copy of Flipboard activated you now get content directly from your Facebook and Twitter streams into Flipboard. 

So, what's so new you say, other than the cool photo splashscreen other than the fact that this app aggregates content from a number of different sources into one place?  There are other apps that do that already and I've used most if not all of them.

What's truly unique about Flipboard is its design. It seamlessly integrates and visually renders content into in a way that Steve Jobs would call magical. Take a look at the screen shot above. Looks like a magazine layout, doesn't it? A layout that a designer would have manually crafted with photos included for appropriate emphasis of certain aspects of stories, etc. What you're actually looking at in that screen shot is my Twitter stream. Twitter is usually quite stark with textual tweets, links, and the occasional image. Flipboard has designed and coded algorithms to handle the layout and rendering of content dynamically and automatically. And the results are amazing. The user also has the option to tap on a story (actually a Tweet but it feels like a story) to see more detail as well as to be able to tap on any images to see them in full screen mode. You've never seen the images in your Twitter stream, Facebook updates, or from your online news sources look better! The subtle animation effects are also very effective. 

It is often said that the spreadsheet program Lotus 1-2-3 was the "killer app" that drove the early success of the PC industry. I believe that Flipboard will do the same for the iPad and like devices. It shows was it possible with the device and its user interface patterns will likely (and should) be used widely in mobile apps like this. The app isn't without controversy, however, due to the way it pulls content from sources but I hope that this won't hamper the success of this amazing app. I should give you one caution, though. The app is so engaging to use that you'll find yourself spending much more time with it than you planned. It's like the experience you have with games except that in this case you're actually learning a ton during that extra time.       

Going Digital for Free

Many aspects of our lives have been going digital over the past few years. A typical knee-jerk response to a question is to simply Google it.  I consider Google an appliance for my brain given how often I use it in this way. The majority of people now listen to music in the form of MP3s using an iPod, iPhone, or some other personal audio device. Many also watch digital movies on their computer screens or increasingly on their TV screens too. News is increasingly consumed in digital form via websites, blogs, Twitter, Facebook, or podcasts. Books are now often read on dedicated devices such as a Kindle or directly on devices like iPhones and iPads and alternatively listened to as audio books in the way that music is listened to. 

I'm an early adopter of most things digital and have consumed content almost exclusively in digital form for the past few years. I've been paying for much of this digital content though whether through iTunes for music and movies, audible.com for audio books, and amazon.com for eBooks. I was reflecting the other day on the fact that I used to borrow books, music, and movies from the public library, the physical versions of these things. I thought it was an amazing tax-supported way of simply borrowing media for the time period you needed it. I remember checking periodically with the library to see if they were moving into the digital world and was routinely disappointed to learn that they weren't. In fact, I found it extremely frustrating that I couldn't even access the online catalog because all the computers were being used by people accessing their email!

You can imagine my delight when I recently checked with my public library website and found that they now provide digital media and the ability to download and transfer it directly to your media device (an iPhone in my case). The media stays on your device for the loan period and automatically disappears after that leaving you extra space on your device. They offer music, movies, ebooks, and audio books. Since finding this new free resource about a couple of weeks ago, I've already listened to four audio books. I'd recommend that you check out your local public library website to see if yours has digital content for download as well.  

Apple's New iOS4

I just upgraded my iPhone to the new version of the operating system. Apple renamed what used to be the iPhone OS to the more generic iOS because it is the system that powers the iPhone, iPad, and iPod Touch. The latest version, iOS4, comes with the new iPhone 4 but it is also being made available for free for previous iPhone models. I have an iPhone GS so the upgrade was a really nice surprise, especially for free!

The major enhancements that come with iOS4 include multitasking, folders, integrated and threaded mail, and camera zooming. Let's explore each of these in turn.

Multitasking is a welcomed addition to the iPhone. The method of switching between apps once you know that you have to double-press the button is pretty intuitive and convenient. What isn't so intuitive is what this control actually does. The tray of running apps is obvious enough but what is confusing is the fact that that tray just keeps filling up with apps even if you closed them down by single pressing the button. The only way to remove apps from the actively running tray appears to be to explicitly delete them, which is weird. Apparently, only a few apps are able to actually keep running whereas others simply suspend their state while not in focus. Any difference between these two types of apps in terms of their multitasking ability isn't apparent in the user interface itself.

Folders are not only a nice to have but a necessity if you have a reasonably large collection of apps. The user interface action in this case isn't intuitive but once you've done it once, it does feel natural. You have to simply drag one app icon onto another and a folder is created with the default name of the category of the apps. You can then add up to 12 apps per folder. Rather than having to scroll to the right and left through many pages of apps, the new folders allow you to simply tap on a folder which reveals all the apps within it without having to do any scrolling. I've found this a perfect way of cleaning up the apps on my iPhone and being able to access them in a much more efficient manner.

Mail has been enhanced with the addition of an integrated inbox and threaded conversations. I'm not crazy about the integrated inbox because I personally don't want my work and personal email in a single inbox. I find it jarring. However, others may think it is a great feature. As for the threaded conversations, I can't find them. I don't know if you have to somehow turn this feature on through another hidden user interface action but I don't see any threading in my inboxes. I'm assuming that the feature is essentially the approach that Google introduced into its GMail system where all replies to an initial email are nested together. I use that all the time in GMail and in Lotus Notes but I don't see it in my iPhone inboxes. When I find it, I'll report back here what I think of it.

Camera zooming is an incredibly important addition to the iPhone. I take pretty well all my pictures with the iPhone and the other day I was taking pictures of my daughter and her friends at a pre-prom party we hosted at our house. For some of the pictures, we had the group of friends all line up on the other side of the swimming pool from the parents taking pictures. The parents with actual cameras were able to zoom in to capture the group of friends across the pool but I wasn't able to zoom at all and, as a result, missed some of the detail. I now can with the updates made to the camera which allows you to move a slider on the camera view to set the right level of zoom. Pretty sweet!

While my very early experiences with the enhancements made to iOS4 weren't unqualifiedly positive, I am still very pleased with the update and especially so given the price - free. 

 

iPad - Design without Affordance

I've now had a lot more experience with the iPad. I continue to thoroughly enjoy using it. However, I'm going through a similar experience that I went through (and blogged about here) with my iPhone initially. The experience is one of thinking that certain things just don't work only to learn later that there was a hidden user interface feature to carry out some action that the user had no way of knowing about. I had concluded that certain websites were simply not enabled for the iPad and I thought it weird that they hadn't done anything about the problems given the popularity of the iPad. For example, several parts of Facebook simply didn't work such as the list of friends. The problem was scrolling the list of friends. When I tried to swipe down on the list, the way you scroll down everywhere else on the device, the page would move slightly but the pop-up layer would move with it but the list of friends would stay put. Regular users of MacBooks would have a clue as to what to try to get this type of list to scroll but everyone else is left in the dark. I asked a number of other people about this and nobody that I found knew a way of scrolling lists like the ones on Facebook and we just assumed the site was not appropriately enabled for the iPad.

It was colleague of mine that finally clued me into the solution - the two finger swipe. I just tried it tonight and it worked! While I'm glad that I now know how to do something that is pretty basic on the iPad, it got me thinking about Apple's overall strategy of designing without affordances. The overall assumption is that the various touch actions are intuitive and, as a result, it is so much more efficient to not clutter screens with affordances like scroll bars. To Apple's credit, most of the touch actions are intuitive. However, the problem is the set of touch actions that are not intuitive and how users are supposed to find out about them. I admire Apple's commitment to its design principles for virtually all other aspects of the design of this device but that purity of adherence to the principles does have its costs and this is one of them.   

iPad - A New Paradigm in Computing

I was tempted to write a post shortly after I got my iPad but thought better of it.  I wanted to use the iPad everyday for several weeks before summarizing my experience.  My overall conclusion is that the iPad truly represents a new paradigm in computing.  It's a game changer.  I've written on this blog for years that advances in operating systems that were touted by others as being significant, like Windows 7 and OSX Snow Leopard, were really tweaks on a decades old system.  The iPad is a true departure from this operating system heritage.  It's a resetting of the dial.  It's what's called "clean-sheet" design in the industry - not worrying about backward compatibility and consistency with prior systems.  

I believe that Apple had a vision for the iPad when they designed the iPod Touch and the iPhone. By the way, I don't think they planned the evolution of the non-touch iPods as carefully since the information architecture broke several times throughout that product family transition.  It was different with the touch line of products.  I think they were planned as a transition.  Introducing a phone with touch capability was revolutionary but it also set the stage for the later introduction of the iPad so that everyone was already comfortable interacting with a touch device.  You can use the iPad the moment you turn it on partly because you've experienced an iPhone or iPod Touch before (or any of the other manufacturer's products that have copied the interaction design).  Interestingly, though, the iPad feels quite natural to those who haven't had experience with a touch device previously.  The actions are generally so natural and obvious that it leads truly to a walk up and use experience.

Here are the things that I think are key to the new computing paradigm that the iPad is introducing.

 

  1. Instant On - while other computers need to boot up even if they're coming off a sleep state, the iPad comes on as fast as you can swipe your finger to unlock the screen.  It's amazingly fast - essentially instant. This is a game changer because the iPad becomes the device of choice when you want to quickly check your e-mail, an app, or a website.  
  2. Convenient Portability - while notebook computers and netbooks are portable in that you can carry them around, you still have to use them like a computer; similarly, smartphones are portable too but are too small to do any number of activities that require a larger screen and keyboard. The iPad has the ideal form factor to conveniently access and interact with information.
  3. Simplicity - while other operating systems make a user figure out a file system, the concept of an application that works with files in the file system, left and right mouse clicks, click and drag, and such esoteric things as drivers, settings, and the need to download, install, and configure applications, the iPad simply has apps which you can get by pressing a button called "Install App" and an interface that mostly works like you would expect it would if you were using an analogue version of the thing (flip the page in a book by placing your finger on the page and pushing it.)

The iPad at my house is used by anyone who needs quick access to content, for watching movies personally, and for using doing school projects.  I should point out that since I got the iPad in Canada before it was available in Canada, my access to the App Store has been problematic.  Initially, the App Store wouldn't load at all and it would after I created a US account but then I could only download free apps because I needed a US credit card in order to buy paid apps.  I now know about the latest fix to even be able to buy apps in the US store but haven't gone through the laborious steps to do that.  It has been frustrating working with the iPad given these conditions and it has also meant that I haven't been able to use higher-end productivity apps to further test the iPad experience.

There are some limitations that I've experienced.  The glare of the screen is problematic in bright light conditions but I use mine indoors mostly so this isn't a problem very often.  The lack of printing support may become a problem once I use higher-end apps.  The lack of a camera is a problem in trying to use the iPad for Skype calls but a recent Camera for the iPad app appears to have partially solved the camera problem.  I find the lack of multitasking a problem in only certain situations and I suspect the 4.0 version of the OS will address this issue. The major problem I've experienced thus far has actually been the inconsistency in the user interface design of the apps on the iPad.  Apple appears to have approved some of these apps too quickly in order to fill out the App Store for the launch and I hope they'll review apps more carefully in the future.

I'll provide another review in time when I get direct access to my country's App Store and then use more sophisticated apps.  All in all, my impression from a few weeks of using the iPad is that it represents the first in a new paradigm of computing.  

 

The Rush to HTML5 Video

Anyone who ever questioned the market power of Apple and specifically, Steve Jobs, just has to have a look at large websites right now.  Most web-based video and audio used to be delivered with the use of Adobe's Flash player or some other Flash-based control.  However, Steve Jobs has never allowed Flash on the iPhone or the iPod Touch and, of course, it also doesn't appear on the iPad either.  In fact, Jobs recently confirmed that it will never appear on these devices.  What effect has this had?  The realization that they may be excluding this large and increasing market with their sites, has led most sites to implement alternatives.  The alternative is, in fact, to use the draft html5 standard.  What's interesting about this trend is that some sites that aren't quite there yet with their html5 implementation are essentially apologizing for it.  I just wanted to watch something on Ustream on my iPhone and was greeted with the message that "Currently this video is only viewable in Flash :(  We're working hard to make all content viewable on both the iPad and iPhone, please check back later!"  Now, that's market power.

Are iPad Apps too Expensive?

We used to pay hundreds of dollars for games and other consumer software. Apple changed all that with the introduction of apps in iTunes for the iPhone, iPod Touch, and now the iPad. The typical price of an app was in fact free and paid apps had prices of $0.99 and apps going for $4.99 and $6.99 were considered expensive. I hear complaints now that iPad apps are way too expensive at $10 and this from people who make good money and think nothing of buying a coffee or a drink for that amount. Is this a case of Apple having been too successful at resetting our perception of price? With the announcement of iPhone OS 4.0 with its iAd platform, is Apple trying to reset the app price back to free and have app developers and Apple itself make money solely from advertising? Is everyone thinking that advertising is the only monetization strategy based on Google's success with it? Twitter even thinks so with their recent announcement, right? I prefer to buy my apps and think that I'm getting really good value if I get a really good game for $10.  Remember that games on any other platform cost several fold that amount at least.  What's your experience with this?  I'd appreciate any thoughts you may have on this.

The iPad that isn't - for me

Technology media is centered in the US. As a result, anyone reading the tech press or even the popular press who take the lead from the tech press would be of the impression that Apple released a new and exciting product on April 3rd and that users everywhere love it. There's one small detail missing from this picture and that is that the product, the iPad, has only been released in the US. When I tweeted about and discussed the iPad on Facebook, several of my American friends were surprised to learn that they were among the select few (if you can call 500 thousand few) in the world who were able to buy an iPad. I was at my local Apple store on April 3rd to buy replacement earbuds but was surprised to see many more than usual customers in the store. When I asked a staff member, he said that there had been lines out the door earlier of people who thought they could buy the much talked about iPad in Canada.

So, what's going on here? I fully understood that the iPhone had to be delayed worldwide because it involved special arrangements with cell/mobile phone companies in each country. However, that isn't the case with the iPad, at least not the WiFi version that is the only version that is available now in the US. While the user experience is amazingly superior to that of a computer, the basic componentry makes it no different than a computer for the purposes of worldwide release. I don't get it. And, I can't get it, an iPad that is, for another few weeks. There is no firm date nor price yet. Very frustrating for customers wanting to spend their money with a company that is so confident that they will that the company can disregard their worldwide market.

Mobile Work

I've observed colleagues for years walking around with their Blackberries staying connected with their work. While I've stayed connected with work via cellphone and texting, I haven't until very recently used a smartphone to connect to work e-mail, intranet, and other work applications. I've just installed IBM Lotus Notes Traveler and I absolutely love it. I now have my work e-mail, calendar, contacts, to do lists all directly integrated seamlessly into the relevant apps and services on my iPhone. Through VPN on the iPhone, I also now have access to the company intranet and services like chat messaging.

I find it amazingly convenient to be able to quickly check my work e-mail while in line at the coffee shop and to look up where my first meeting of the day is when I'm running late and don't have time to take out and open up my notebook computer. It's an amazing time saver and uses the form factor I so love.

When I think about the convenience of having work information available like this on the device that's always in my podcast, I can imagine the utility of having it available on the somewhat larger but equally enjoyable iPad. I'm looking forward to that. There is one feature that I haven't heard anyone mention with regard to these devices that I think is critical for accessing work systems and that's fingerprint recognition. That would be so useful and Apple cool.

As always, I'd appreciate any thoughts you may have on my thoughts. By the way, I also wrote this post on my iPhone.