5. Smart Desk and a Learning System

5.1 Introduction

Smart Desk, a web-based interactive learning environment, was initially designed for a patient who was cognitively disabled from early brain damaging seizures. Smart Desk (SD for short) is a generic interface system, which, to some degree, can be personalized for individual users with different disabilities and can also provide user-tracking functions for future or existing applications. SD is not only an application environment but may be used as an alternative hardware interface. That is, rather than a traditional keyboard and a mouse interface set, SD can be fitted with accessing hardware specifically tailored to the sensorimotor capacities of particular users (Figure 5-1 and 5-2).

By working within the Smart Desk environment, users are able to improve their cognitive performance over time. Moreover, commercial applications such as educational software, personality assessment tests, neuropsychological tests/batteries, and user-defined applications will be able to communicate and interact with the Smart Desk environment to the extent that they are web browser compatible. In the context of a medical studentís education this includes, for example, parameters such as specific data to be recorded, representation of result data, or type of user data to be collected including history of disease, interviews, tests and personal information can be the user definable.

Figure 5-1. Hand Mouse (part of Smart Desk Chair built by Mike Konieczny)

The chair provides an alternative interface for the keyboard and mouse. On the board of the chair, the hand mouse from the picture is equipped with several other square buttons to provide easy access and generate analog or digital signals. This particular hand mouse can generate 10 signals from each finger and by pushing both hands, it can generate two additional signals each. The chair was built by Tim Lauring and exhibited at Siggraph 98 as a part of SD environment.

Figure 5-2 Smart Desk Hardware Interface by Matt Carbone

This interface provides additional assistance for learning subjects using the SD system; letters and objects can be mapped to games or learning languages or shapes. At the bottom of the device is a graphic tablet and by pointing with a pen, a user selects object at the screen in SD system. It works similar to a touch screen, which we also tried in SD.

Creating a generic environment provides a basis for psychiatrists and economists who look to evaluate computerized systems, educational experts who look to verify their educational system and children who require "home schooling" because of motor deficiency or geographical reasons.

5.2 Other Learning Environment

The first programmed learning environment dates back to 1926, then called the "teaching machine" (Adams 1976). It was originally invented to be an automatic test administrator. Later, Pressy added another mode for learning, after he found that the machine could teach as well if the subject was allowed to continue answering until he/she got the correct answer .

In the context of instrumental psychology, as Skinner contended , learning through a programmed learning environment or a teaching machine can be more effective in teaching than present-day instructional methods with classroom educational systems. Also the computerized learning environment can play a role as an auxiliary tool for the psychologist or pedagogue in the cognitive assessment process of physically disabled people where motor skills are critical factors in the testing process . Within Rodriguesí system, for each individual, a unique environment can be made which is socially and culturally tailored. Such a tailored environment is supported by the research using Piagetian showing that true cognitive ability manifests itself better when the user is familiar with the environment. This is another example of personalized and computerized educational environments within a computer science paradigm.

Interactive computer-simulated learning environments are found more often in college education. The SIMPLE learning environment is an authoring tool to make such an environment . It is supported by relational databases, to maintain consistency of data, and has built-in tools such as replay function, refinement of crude recorded data, and answer-checking. More and more colleges now offer virtual classes which are interactive web-based classes. As another adaptable web-based courseware, Calvi (Calvi and Bra 1998) explored adaptable hypertext links that adjust themselves while the userís learning session progresses.

Currently, web-based learning environments are being created in different fields. Childrenís television station such as Nickelodeon , training software, and even employee review softwares like Computer Based Testing/Training (CBT) or Web Based Testing/Training (WBT) are getting on the Web and are providing interactive learning environments.

5.3 Smart Desk Features

SD is an interactive learning environment architecture which runs in a web browser. It provides a thorough structure for interactive learning. By expanding its subjects, just like inserting a cartridge for an electronic dictionary, the environment grows (Figure 5-3).

Given the webís interactive nature, SD exploits the current web technologies to create interactive learning tools along with performance analysis capabilities. SD can be personalized for a userís specific needs. This reduces the size of an environment dramatically. At the same time, through customization processes, one environment can generate as many different instances as a user will want to have.

SD contains several user-tracking mechanisms. With any user session performed in a Java application, mouse movements, clicking, dragging, and keyboard events are recorded via scripts or built-in functionalities. In the case of web software which does not support communication with other languages such as Java or Javascript to report events information, background processes can be used in the local machine to record events. More detailed ideas on these will be explained in the chapter 5.4.3. What is more, some of the results will be reflected back to the system (learning process) to improve their sessions. For example, for a six-year old learning how to add, we can start from very simple pictorial examples, then his session would be watched and adjustments of difficulty can be made later.








Figure 5-3 Interface to Add a New Application

5.4 Implementation Issues

5.4.1 Personalized Environment

When we donít have much information about the users of a learning environment, creating such an environment to fit their needs is hard. Following is the approach we have taken on this issue. A new user who wants to create his/her environment will walk through an index of learning components such as tools, programs, games, or performance tests which are all classified by subjects and levels. By choosing items from the index, a user can create an environment. Later a user can update or change the environment as performance level increases or decreases. Using the same principle, the same user can have more than one environment (Figure 5-4).

Figure 5-4. Personalizing Process

A learning environment of a user is composed of components at the learning system. Squares stands for components and other shapes mean different environments.

A personalized environment refers to the capacity for tailoring the learning systemís interface and content features to the cognitive and sensorimotor needs and capacities of particular users. Further, the ability to update gives a user challenges and stimuli by providing new components or changing levels of learning .

We provide a generic and easily updatable environment such that a user can add more tools or tests to the index so that other users can share them too, to the extent they are web-based (Figure 5-3). An interface to manipulate data is written in Java servlets, HTML, and Javascript, and is operated in a web browser. User session information, or registration, is stored at a database (Oracle) which supports web interfacing through Java DataBase Connectivity (JDBC) between servlets and the database. Also, information on learning components is itemized and stored at the database (Figure 5-5).

5.4.2 Adaptiveness Issues

Just like a 1st grader goes to the 2nd grade after one year of school, a learning environment needs to be changed as a user successfully progresses. This environment can be adjusted to upper or lower levels according to a user's performance. The adjustment may be done to each learning component without affecting other components. After a given period of time and number of learning sessions, by examining user performance, it is updated or left unchanged if necessary.





Figure 5-5. Communication Between Front end and Server

The basic structure of the communication process is three tiered . Compared to a two tier architecture of client-server model, a three tier architecture has the third layer to isolate data processing at server side and maximize objects reuse.



The most important issue is how efficiently the learning environment is designed to gather user data so a teacher or user can know the quantitative and qualitative performance of the work accomplished. Automatically tracked information such as mouse movement and keyboard inputs is available along with results from each learning session. Tracked or traced information can reveal the performance of users. Classification of this information and measuring levels are challenging issues ; however, they are beyond the scope of a computer science analysis.



Figure 5-6. User Tracking

This figure shows how tracked data from each user session is transferred to the database at the server side.



Figure 5-7 (a) an example of measuring and analyzing mouse inputs

Results from a concentration game with an option of "color-to-words" mapping. Each number below the cards signifies the number of times a user clicked the mouse during the learning session.







Figure 5-7 (b) "word learning" game

another example of tracked user data at the client side.










Figure 5-7 (c) an example of tracking user events along with time axis

Gap between each vertical bar (events) represents elapsed time between each event. The number above each bar means "which" card was selected. It specifies which objects were chosen.





Figure 5-7 (D) an example of mouse trajectory

X and Y trajectories were recorded separately. First part of the above graph shows that there were extensive movements along with the X axis, i.e., a user moved the mouse mainly left and right. The second part of the graph shows up and down movements of the mouse.

Figure 5-6 (E) an example of right-wrong answer measurement

Figure 5-7 (Eí) Proportion of right and wrong answer

(E) is to fill the circle with mouse or mouse-equivalent device. (Eí) is the resulting graph. Lighter part represents the percentage that a user filled in the object. In this case, it is about 80%.

5.4.3 User Tracking (Figure 5-5)

User tracking is a quantitative approach for adapting a learning environment. For measuring user performance, not only are the results or answers of assessment applications important, but also how a user interacts at each session. For example, mouse movement can tell us about a user's hand coordination at certain times in responding to certain types of information.
Generally, user tracking schemes in SD can be viewed in three categories. If learning components are written in Java and have event data tracked inside, then the tracked user session data is sent to a database residing at a server side (Figure 5-7 A Ė Eí). A web browser also provides information on mouse movement and navigation behavior. Perl scripts and Javascripts are used for this purpose. Since Javascript version 1.2, event objects can detect "clicks", "double clicks", "down", "up", or "over" for mouse event, and "down", "press", or "up" for keyboard event, over the entire web browser window or document, for example. A set of Javascript functions to implement such event detection along with time information by date objects was created; Perl scripts were written to generate the javascript code automatically at the header parts of HTML files. Other applications which are not written in Java (or any other language that can record user data and send it back to a server side by communicating with a web browser and web server) can be tracked using a background process at the userís local machine. For instance, a software successful at tracking user interactions is NeatTools, implemented by Yuh-Jye Chang (Appendix C). It can, among many other things, record mouse movement, keyboard event and time information associated with such events. In this case, after the user session, one can upload the resulting data file to a server through the web.

Smart Desk is designed as a web-based application. Specifically, it requires no client side support to run Smart Desk other than a web agent, or browser, and a background process program if required. Centralized information enables users to share the most up-to-date information, eliminating the need to worry about stale data. This gives relief to developers as well.
For classrooms, or users without Internet access, a CD-ROM version may be made available. The database components will need to be edited; and because the connection to the database is JDBC, with a few changes, local databases such as MS Access can be used instead of the centralized database at a server side. Even without a database, by using the file system of the local computer to store data, SD can be run.

5.5 SD Information Architecture and Events

Registration and Customization processes are conceptually handled by AS (Application Server, see chapter 3). At SD, Servlets play the role as AS in both processes between user requests and a content server (Figure 5-8). It is essentially an administrative service. Figure 5-9 and 5-10 show interfaces of both processes.

Figure 5-8 Servlets at AS

Figure 5-9 User Registration



Figure 5-10 Customization

The event model of SD is depicted at Figure 5-11. Figure 5-12 shows more detailed client-server event model. In detecting user events, SD uses various methods. For a client-side event tracking or detection, Javascript and background process lurking at userís local machine are chosen. Server-side events detection is to watch events among components at server side and user logs. To represent event data, XML (Extensible Markup Language) may be used. XML provides a noble way to describe data. It is based on extended BN (Backus-Naur) notation. Both client and server sides event data are specified at Figure 5-13 in a terse XML notation.





Figure 5-11 SD Event Model

Figure 5-12 Data and Event Flow at SD




<?XML version="1.0"?>



















Figure 5-13 Events at SD in XML

5.6 Conclusion

Smart Desk provides a web-based interactive learning environment and enables users to personalize those environments. Updating or adapting their environments are possible through analyzing their prior performances in user sessions. It solves many problems in computerized education or performance tests. It can be used as a valuable tool to simulate user behaviors and observe user-environment interaction. SD is a testbed for measuring interactivity, as will be discussed in a later chapter.